The GPU pipeline: how a triangle becomes pixels

Three.js gives you Mesh, Material, Camera, Scene — comfortable objects that make 3D feel like arranging furniture. That comfort is exactly the problem when something goes wrong or runs slow, because none of those objects are what the machine actually does. Underneath every Three.js program is a much smaller, stranger thing: a pipeline the GPU runs to convert a list of points into a grid of colored pixels, the same way for every frame of every 3D application ever shipped. Learn that pipeline once and Three.js stops being magic — every class in it becomes "oh, that's the thing that fills in this stage."

The single framing question for this whole module: you have a triangle described by three points in 3D space, and you have a screen that is a 2D grid of pixels. How does one become the other? The answer is a fixed sequence of stages, and your job (or Three.js's job on your behalf) is to supply data and two small programs at the right points in that sequence.

Everything is triangles, and a triangle is three vertices

First, the input. However complicated a 3D model looks — a character, a car, a terrain — it is, to the GPU, a big list of triangles. Triangles are the only surface primitive that matters, because three points always define a flat plane, which makes the math for "which pixels does this cover" tractable and uniform. A sphere in Three.js isn't a sphere to the GPU; it's a few hundred triangles approximating one.

Each triangle is three vertices, and a vertex is not just a position — it's a bundle of attributes. Position (x, y, z) is the one you can't skip, but a vertex typically also carries a normal (which way the surface faces at that point, needed for lighting), texture coordinates (where to sample an image), maybe a color. Hold onto that idea: a vertex is a little record of attributes, and a mesh is a big array of those records sitting in GPU memory. That array is the raw material the pipeline consumes.

The pipeline, stage by stage

Here is the sequence. It runs in this order, every frame, for every triangle you draw. Some stages are fixed-function (the hardware does them a fixed way; you configure but don't program them) and two are programmable (you supply a small program that runs at that stage). That distinction is the most important thing on this page.

Vertex buffers. The starting point: your mesh's vertex attributes, uploaded once into GPU memory as raw arrays. The GPU reads from here; it does not reach back into JavaScript. (Getting data into these buffers is its own lesson — geometry and buffers.)

Vertex shader — the first program you write. The GPU runs this small program once for every vertex. Its one required job is to decide where that vertex lands in clip space — a normalized coordinate system the rest of the pipeline expects, which is how 3D positions eventually become 2D screen positions. This is where the model/view/projection matrices get applied (all of lesson 4). The vertex shader can also pass data down the pipeline for each vertex — a normal, a texture coordinate — to be used later.

Primitive assembly. Fixed-function bookkeeping: the stream of transformed vertices gets grouped back into triangles (every three vertices, typically), so the next stage knows the actual shapes.

Rasterization — the stage that turns geometry into pixels. This is the conceptual heart of the whole pipeline and it's fixed-function, done by dedicated hardware. Given a triangle (now in screen terms), rasterization works out which pixels of the screen grid fall inside it. Each such pixel becomes a fragment — a candidate pixel that still needs a color. Critically, rasterization also interpolates the vertex attributes across the triangle's face: if one corner's normal points one way and another corner's points another, every fragment in between gets a smoothly blended normal. That interpolation is why a coarse triangle mesh can look smooth, and it's the quiet mechanism a lot of shading tricks rely on.

Fragment shader — the second program you write. The GPU runs this once for every fragment the rasterizer produced. Its job is to output a single thing: the color of that pixel. This is where lighting is computed, textures are sampled, and material logic lives (the subject of shaders and materials). A modern scene can produce millions of fragments per frame, so this little program runs an enormous number of times.

Per-fragment tests, then the framebuffer. Before a fragment's color is written, fixed-function tests decide if it survives — most importantly the depth test: if something closer to the camera already painted this pixel, the farther fragment is discarded, which is how the GPU sorts out what's in front of what without you sorting triangles yourself. Survivors are written (possibly blended for transparency) into the framebuffer, the block of memory that is the finished image, which then goes to the screen.

The one idea that explains why it's a GPU

Look at the two programmable stages again: the vertex shader runs once per vertex, the fragment shader once per fragment. A real scene might have a hundred thousand vertices and several million fragments per frame, at sixty frames a second. If you ran those programs one after another on a CPU, you would never come close.

The trick is that every invocation is independent and identical in form. Vertex #40,000 is transformed by exactly the same program as vertex #1, just with different input data. Fragment over here is colored by the same program as the fragment over there. That is the perfect shape for parallelism: a GPU is essentially thousands of small cores running the same program on different data at the same time (the model called SIMD — single instruction, multiple data). The pipeline is designed the way it is specifically so that the expensive stages decompose into millions of independent, identical little jobs a GPU can chew through in parallel.

What the two programs actually look like

To make "you write two small programs" concrete rather than abstract, here is about the simplest possible pair, written in GLSL (the C-like shading language WebGL uses). Don't worry about the exact syntax — the shaders lesson covers it — just see the shape: one program positions a vertex, the other colors a pixel.

// Vertex shader: runs once per vertex.
// It receives this vertex's position as an attribute, and a combined
// transform matrix as a uniform (same for every vertex this draw).
attribute vec3 position;
uniform mat4 modelViewProjection;

void main() {
  // Output where this vertex lands in clip space.
  gl_Position = modelViewProjection * vec4(position, 1.0);
}

// Fragment shader: runs once per covered pixel.
// This trivial one paints every fragment flat orange.
precision mediump float;

void main() {
  // Output this pixel's color: R, G, B, A.
  gl_FragColor = vec4(1.0, 0.5, 0.0, 1.0);
}

Two tiny programs, plus an array of vertex positions, and you have a solid orange triangle on screen — that is genuinely the whole minimum. Everything Three.js does is (a) fill those vertex buffers from your Geometry, (b) generate far more elaborate versions of those two shaders from your Material and Lights, and (c) issue the command that runs the pipeline. When you understand that the fancy objects reduce to "buffers plus two generated programs plus a draw command," the library becomes legible.

Where Three.js fits (and why we still start here)

Three.js is a convenience layer over WebGL, the browser API that actually drives this pipeline. You will spend most of your time in Three.js's comfortable objects — but every one of them exists to fill in a specific part of the pipeline above:

• a BufferGeometry populates the vertex buffers;
• a Camera supplies the matrices your vertex shader uses to place vertices;
• a Material (plus lights) is compiled into the fragment shader that colors pixels;
• calling renderer.render(scene, camera) is what finally issues the draw that runs the whole sequence.

We started with the raw pipeline instead of the friendly objects for the same reason the other deep tracks start at the bottom: when your scene is black, or slow, or the lighting is wrong, the abstraction can't tell you why — the pipeline can. Every remaining lesson in this module fills in one stage of the diagram above.

Where this goes next

The very next question is the one we skipped: Three.js is a wrapper, but a wrapper over what, exactly? WebGL: the state machine under Three.js takes apart the actual API — and it's a genuinely unusual one, not a set of "draw" functions but a big configurable state machine you set up and then trigger. Understanding its shape explains why Three.js exists at all, and why raw WebGL code looks the way it does.