I had been staring at OpenGL's tessellation shaders for hours, trying to make sense of patches, layouts, and all those mysterious gl_TessLevel variables. Here is a summary of it all.
What's a "patch"?
A patch functions as the fundamental input unit to the tessellation pipeline, comparable to how vertices feed the vertex shader or fragments feed the fragment shader. It represents a group of vertices that defines a piece of a surface to be subdivided.
To specify patch vertex count:
glPatchParameteri(GL_PATCH_VERTICES, 4);When drawing patches:
glDrawArrays(GL_PATCHES, 0, 4);The Tessellation Control Shader (TCS) executes once per vertex in the input patch.
What happens in the Tessellation Control Shader?
The TCS accesses patch vertices through the built-in array gl_in[]. With layout(vertices = 4) out; declared, four control points exist per patch. Each TCS invocation is identified by gl_InvocationID and writes to gl_out[gl_InvocationID].
Basic forwarding:
gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;This passes control point positions to the Tessellation Evaluation Shader (TES) — a crucial step for downstream processing.
What about the tessellation levels?
Within the TCS, tessellation granularity is specified:
gl_TessLevelOuter[0..3] = ...
gl_TessLevelInner[0..1] = ...Level counts depend on domain type:
- triangles: outer = 3, inner = 1
- quads: outer = 4, inner = 2
- isolines: outer = 2, inner = 0
Outer levels govern edge subdivision; inner levels control interior density.
How does the Tessellation Evaluation Shader fit in?
The TES interprets patch structure through layout declarations:
layout(quads) in;This specifies quad domain interpretation using (u, v) coordinates between 0 and 1 for vertex interpolation across the surface. Alignment requirements: TCS vertex count must match TES domain:
layout(vertices = 4)+layout(quads)= compatiblelayout(vertices = 3)+layout(triangles)= compatible
Mismatched counts result in OpenGL ignoring excess data.
What are Tessellation Coordinates (gl_TessCoord)?
The tessellator generates new vertices with built-in variable gl_TessCoord, indicating position within the tessellated patch.
| Domain | gl_TessCoord contents | Description |
|---|---|---|
| quads | vec3(x, y, z) where z = 0 | (x, y) in [0,1] across the quad |
| triangles | vec3(u, v, w) with u+v+w=1 | Barycentric coordinates |
| isolines | vec3(u, v, w) where u,v∈[0,1] | u = position along line, v = line index |
float u = gl_TessCoord.x;
float v = gl_TessCoord.y;For quads: u = 0 is left, u = 1 is right, v = 0 is bottom, v = 1 is top. Bilinear interpolation between four control points:
vec3 P = mix(mix(P0, P1, u), mix(P3, P2, u), v);Why is the MVP multiplied in the TES?
Applying the Model-View-Projection transformation in the vertex shader before tessellation produces incorrect results. Projection via the MVP is not linear, and tessellation interpolates linearly. Tessellating in clip space distorts edges and produces curved artifacts.
Correct approach: maintain object or world space coordinates during tessellation, then apply MVP in TES:
gl_Position = uMVPMatrix * vec4(P, 1.0);This ensures all generated vertices receive proper clip-space transformation.
How does everything connect?
| Stage | Operates on | Space | What it does |
|---|---|---|---|
| Vertex Shader | 1 vertex | Object | Passes raw vertex data |
| Tessellation Control | Patch (N vertices) | Object | Sets tessellation levels |
| Tessellator | — | Object | Subdivides patch |
| Tessellation Evaluation | Generated vertices | Object → Clip | Interpolates + applies MVP |
| Fragment Shader | Fragments | Screen | Final color |
Quick Recap
- Four vertices = one patch via
glPatchParameteri(GL_PATCH_VERTICES, 4) - TCS executes per patch with
layout(vertices = 4) out; - TES interprets patch structure through
layout(quads) in; - Tessellation level counts vary by domain type (triangles / quads / isolines)
- Apply MVP transformation in TES, not vertex shader
- The
gl_out[]array bridges TCS and TES data flow