Parsing and the AST: turning text into a tree (lazily)

The previous lesson laid out V8's pipeline as parse, then bytecode, then maybe JIT. It's easy to skim past "parse" as a formality — of course the engine reads your code before running it. But parsing is not free, and V8 treats it as a cost to be minimized rather than a box to be checked. This lesson takes the first stage apart: what parsing actually produces, and the trick — lazy parsing — that lets V8 skip most of it entirely.

Lexing: characters become tokens

Source code arrives as a flat string of characters. The first pass, lexing (or tokenizing / scanning), walks that string once and groups characters into tokens — the smallest meaningful units of the language: keywords (function, const, return), identifiers (add, x), literals (2, "hi"), punctuation ((, ), {, }, ,), and operators (+, =).

Given this source:

function add(a, b) {
  return a + b;
}

the lexer produces a flat stream of tokens: function, add, (, a, ,, b, ), {, return, a, +, b, ;, }. Notice what's already gone: whitespace, and the distinction between where one token ends and the next begins in the original text. The lexer has thrown away everything that doesn't carry meaning and kept a clean list of "words" for the next stage to work with. It doesn't know yet that a and b are parameters, or that this is a function at all — it's purely a character-level pass.

Parsing: tokens become a tree

The parser takes that flat token stream and assembles it into an Abstract Syntax Tree (AST) — a tree of nodes representing the program's grammatical structure. The word "abstract" is doing real work: an AST discards the surface syntax (parentheses used only for grouping, semicolons) and keeps only the structure that matters — what is a function, what are its parameters, what statements make up its body, what is an expression versus a statement.

For the add function above, the AST looks conceptually like this:

// FunctionDeclaration
//   name: "add"
//   params: [Identifier "a", Identifier "b"]
//   body: BlockStatement
//     [0] ReturnStatement
//           argument: BinaryExpression "+"
//             left: Identifier "a"
//             right: Identifier "b"

Every node knows its kind (FunctionDeclaration, BinaryExpression, Identifier) and its children. The flat token stream function, add, (, a, ,, b, ), {, return, a, +, b, ;, } has become a nested structure that says, unambiguously: this is a function named add, taking two parameters, whose single statement returns the sum of them.

Why a tree, and not just re-reading the text

You might ask why the engine bothers building a tree at all instead of compiling straight from tokens, or re-scanning the source text whenever it needs to know what a piece of code does. The answer is that a tree is a structured representation the next stage can walk mechanically. Bytecode generation (lesson 3) needs to know, for a return a + b statement, exactly which sub-expression is the left operand and which is the right, and it needs that answer as a data structure it can traverse — not as a string it has to re-parse. Building the AST once, up front, means every later stage gets a clean, unambiguous shape to consume instead of re-deriving structure from text every time it's needed.

Lazy parsing: the key optimization

Here is the fact that shapes everything else in this lesson: most functions defined on a typical page never run — event handlers that never fire, branches for browsers you don't have, utility functions covering cases this page doesn't hit. If V8 fully parsed every function's body into a complete AST before running anything, it would spend real time and memory building trees for code that contributes nothing to this page load.

So V8 doesn't. When it first encounters a function, it does a pre-parse: a cheap pass that scans just far enough to find the function's boundaries (where it starts and ends) and confirm there's no syntax error inside it, without building a full AST for the body. It records enough to reconstruct the function later — its name, its parameter list, where its source text starts and ends — and moves on. The function's body is not turned into a tree yet.

Only when that function is actually called does V8 go back and do the full ("eager") parse — building the real AST for the body, right before handing it off to generate bytecode. A function that's defined but never invoked pays only the cheap pre-parse cost, forever.

This is why "lazy parsing" is the right name: parsing work is deferred until the last possible moment — the point of actual invocation — rather than done eagerly for every function the moment the file loads. On a page shipping megabytes of JavaScript where only a fraction of functions run during a session, this avoids a large fraction of parsing work outright.

The double-parse hazard

Lazy parsing isn't free of tradeoffs. A function that gets pre-parsed and then called still has to pay for a full parse at call time — so in total it was scanned twice: once cheaply during pre-parse, once fully when invoked. For a function that runs exactly once, shortly after the page loads, that's strictly more total parsing work than if V8 had just eager-parsed it the first time. This is exactly why the IIFE hint matters: for code you know is about to run, signaling "parse this eagerly, skip the pre-parse-then-parse dance" can be cheaper than the default lazy path. Lazy parsing is a bet that most functions won't run; for the ones you know will, betting the other way pays off.

Startup cost, and why smaller bundles parse faster

Parsing and compiling are not free background work — they compete for time on the same main thread that has to become responsive before a user can interact with the page (the event loop lesson covers why that thread is so precious). On real sites, parsing and compiling JavaScript is a measurable, sometimes significant, fraction of total page startup time, especially on lower-end devices where scanning characters and building trees is comparatively slow. Two consequences follow directly from everything above:

• Lazy parsing is a genuine win at scale, because it turns "parse every function" into "parse only the functions that run," which on most real pages is a small fraction of the total code shipped.
• Smaller bundles parse faster in an absolute sense — minification strips whitespace and shortens names, which shrinks the character stream the lexer has to scan and the token stream the parser has to walk, even before lazy parsing enters the picture. Less text in, less tree to build, less time spent before anything can run.

Where this goes next

The AST is the structured form that makes the next stage possible: it's what gets walked to generate bytecode. Bytecode and Ignition picks up exactly there — how V8 compiles this tree into a compact instruction set for its own virtual machine, and how its interpreter, Ignition, executes that bytecode as the baseline tier every function runs in first.