Anti-Aliasing

The core problem of rasterization is that the real world is continuous and pixels are discrete. Every edge rendered on screen is a lie — a staircase pretending to be a line. Anti-aliasing is the collection of tricks we use to make that lie less obvious, and the history of these tricks tracks the evolution of GPU hardware itself.

The Sampling Problem

When a GPU rasterizes a triangle, it evaluates the pixel shader once at each pixel center. If the triangle edge cuts through a pixel, the whole pixel gets the triangle's color or nothing — there's no in-between. This is point sampling, and it produces the characteristic jagged edges (aliasing) that have plagued computer graphics since the beginning.

The mathematically correct solution is to compute the exact area of the triangle that covers each pixel and weight the color proportionally. Offline renderers do this and it looks gorgeous. But it's absurdly expensive: for a pixel that might overlap dozens of triangles, you'd need to compute polygon-polygon intersections for every one. Jonathan Olson's work on exact polygonal filtering shows that this is actually tractable for 2D vector graphics using Green's theorem — you can convert the area integral over a polygon into a sum of line integrals over edges, and for polynomial filters (box, bilinear, bicubic), the result has a closed-form solution.¹ The Shoelace formula for polygon area is a special case. But for 3D scenes with millions of triangles, this remains impractical in real time.

Supersampling and MSAA

The brute force approach is supersampling: render at higher resolution and downsample. Sample 4 or 8 or 16 times per pixel and average. This works perfectly but multiplies your rendering cost by the sample count.

Multi-Sample Anti-Aliasing (MSAA) is the hardware optimization of this idea. Instead of running the pixel shader multiple times, the GPU runs it once per pixel but tests triangle coverage at multiple sub-pixel sample points (typically 2x, 4x, or 8x). The shader output is replicated to all covered samples, which are then resolved to a single color. This is much cheaper than full supersampling because the expensive part — the shader — runs once, while the cheap part — coverage testing — runs multiple times.²

MSAA works well for geometric edges (triangle silhouettes) but does nothing for aliasing inside the triangle — texture shimmer, specular highlights, alpha-tested foliage edges. It's also expensive in terms of memory bandwidth, because every render target needs to store N samples per pixel, and it interacts badly with deferred rendering where you'd need to store N copies of every G-buffer channel.

Post-Process Anti-Aliasing

The insight that changed the game was: instead of computing better samples during rendering, just look at the final image and try to fix the edges after the fact. This is post-process anti-aliasing (PPAA), and it works by detecting color discontinuities (likely edges) in the rendered image and blending across them.

FXAA (Fast Approximate Anti-Aliasing) by Timothy Lottes at NVIDIA is the simplest and fastest. It looks for contrast edges, estimates the edge direction, and blends neighboring pixels along that direction. It's incredibly cheap — a single full-screen pass — but it blurs everything indiscriminately. Text, UI elements, high-frequency textures all get softened because the edge detector can't distinguish intentional color boundaries from aliasing artifacts.²

MLAA (Morphological Anti-Aliasing) by Alexander Reshetov took a more structured approach: detect edges, classify them into simple geometric shapes (L-shapes, Z-shapes), and blend based on the reconstructed sub-pixel geometry. This produces much better results for long diagonal edges.

SMAA (Enhanced Subpixel Morphological Anti-Aliasing) extended MLAA with better pattern recognition, local contrast adaptation, and optional temporal super-sampling. It's the quality leader among image-based methods but costs 2-3x as much as FXAA.²

CMAA (Conservative Morphological Anti-Aliasing) by Filip Strugar at Intel targeted a different niche: running efficiently on integrated GPUs while being "minimally invasive" — designed to avoid blurring text and small patterns that FXAA destroys and SMAA distorts.² CMAA's key innovation is the "conservative" philosophy: it only anti-aliases symmetrical edge patterns (Z-shapes), deliberately leaving asymmetric cases alone. This means it smooths fewer edges than SMAA but also does less damage to things that shouldn't be smoothed. The locally dominant edge detection step — where each edge's contrast is compared to its 12 neighbors — aggressively prunes edges that aren't the most prominent feature locally, which is surprisingly effective at preserving text.

The Mipmap Problem

Anti-aliasing isn't just about edge jaggies. Texture aliasing — the flickering and moire patterns that appear when a texture is displayed smaller than its native resolution — is often worse. Mipmapping, invented in the early 1980s, precomputes successively half-resolution versions of each texture. The GPU selects the mipmap level where the texel-to-pixel ratio is closest to 1:1.³

The problem is that mipmaps are isotropic — they scale uniformly in both dimensions. A floor texture viewed at a glancing angle has wildly different scaling on the horizontal and vertical axes. The GPU has to pick based on the worst case (the axis with more minification), which makes the entire texture blurry. Anisotropic filtering helps by sampling the mipmap multiple times along the anisotropic direction, and 8x or 16x anisotropic filtering is now essentially free on modern hardware.

But even with anisotropic filtering, mipmapping overblurs camera-facing surfaces. Ben Golus's work on sharper mipmapping for VR games reveals a practical truth: the mipmap transition points are conservative by design — the GPU switches to the next mip level when the texture is only about 60% of the way to the half-resolution point, not at the exact boundary. This means textures are always somewhat blurrier than they need to be.³

The fixes are pragmatic: mip biasing (tell the GPU to use a sharper mipmap level via tex2Dbias), and shader-based supersampling (take multiple texture samples at different UV offsets within the pixel). Both reintroduce some aliasing but the tradeoff is often worth it, especially in VR where display resolution is precious. Temporal anti-aliasing can clean up the aliasing these tricks introduce, which is why both Unreal Engine and Unity's HDRP enable negative mip bias by default when TAA is active.³

2D: Still Harder Than 3D

Here's the counterintuitive thing: anti-aliasing 2D vector graphics is in many ways harder than anti-aliasing 3D scenes. As Jasper St. Pierre argues, this is because 2D graphics is fundamentally about shapes — accurate letterforms, Bezier curves, precise geometry — while 3D is about materials and lighting.⁴ GPUs evolved to make materials look good at the expense of geometric precision, because that's what 3D games need. For a first-person shooter, 4x MSAA on triangle silhouettes is fine because the player's attention is on textures and lighting, not pixel-perfect edges.

But font rendering, SVG display, and UI elements demand subpixel accuracy. A font glyph rendered with 4 fixed MSAA sample points looks rough; you really want to compute the exact area under the curve for each pixel, which is the problem that Olson's exact polygonal filtering solves for the polygon case.¹ The PostScript imaging model, which underpins everything from PDF to SVG to HTML Canvas, is built on Bezier curves, and rendering those curves at pixel resolution with correct anti-aliasing remains an active research problem. Projects like Pathfinder, Blend2D, and Raph Levien's piet-gpu / Vello are all pushing different approaches.⁴

The deeper irony is that rotation and panning in a 2D document — scrolling, zooming, dragging — is perceptually unforgiving in a way that 3D camera movement isn't. In 3D, fast viewpoint changes trigger something like saccadic masking; you re-orient to the new view and don't notice frame-to-frame artifacts. In 2D, you're staring at the same text as it scrolls by one pixel per frame, and any temporal instability — the boiling artifacts of poorly tuned TAA — is immediately obvious.