Watch the companion video for this post: Learning from TeX’s testing philosophy
Donald Knuth’s TeX has an almost mythical reputation as one of the most stable, bug-free large codebases ever created. When Knuth officially frozen the language, he famously instituted a bug-bounty check program where rewards doubled with every bug discovered (reaching $327.68 per hexadecimal dollar). Bugs are so rare that finding one today is practically a historical event.
When working on a custom engine like NemaTeX, one reaches an intimidating milestone: the engine must face Knuth’s official conformance test suite, the TRIP test. Passing the TRIP test is not merely a benchmark, it is part of the definition Knuth established in the tripman.tex document for any program to call itself “TeX”.
The Anatomy of a Diabolical Test
Modern software engineering heavily emphasizes unit testing: writing small, isolated tests for individual functions and classes. Knuth took the opposite approach. Rather than testing components in isolation, he created trip.tex: a single 450-line file designed to torture almost the entire engine simultaneously.
Calling trip.tex an adversarial test is an understatement. It does not contain a single paragraph of normal, readable text. Instead, it is a dense, handwritten minefield of edge cases (sometimes I like to think of it as a hand-written fuzz test). There is deliberately malformed syntax designed to trigger nested error-recovery branches, recursive macros crafted to push the input stack to within a single slot of overflowing, mathematical spacing edge cases, extreme glue stretch and shrink values that force arithmetic routines right to the boundary of integer overflow, dynamic allocations structured to trigger memory reclamation and free-list recycling, and so on.
The few times I broke trip-test compliance during development and had to try to read and understand the trip.tex file… these were probably the single most difficult moments in this project.
Anyway, running trip.tex through the engine a few times produces terminal output, large log files, a .dvi output – all of which must precisely match a gold standard (with a handful of explicitly allowed deviations).
The 80% Coverage Surprise
As a fun little experiment, I tried running the trip test while using source-based coverage tools to generate a code coverage report. Code coverage is obviously not the be-all-end-all, but in any event… it was a surprise to discover that across a roughly 40000-line codebase, the tiny little 450-line trip.tex “test suite” managed to exercise over 80% of the entire codebase.
Most of the “missing” parts of that code coverage are either (1) in parts of the codebase I wrote (related to pdf generation, etc), (2) related to dvi output, or (3) related to fatal errors. Amusing, because the trip.tex test didn’t exercise the happy path, I’ve found that it’s possible to pass the trip test but have incorrect dvi output on much more normal documents!
Meditations on Testing
I don’t know what I want people to take away from this, other than that I think there is a lot to learn about testing philosophy from Knuth’s approach. Unit testing is essential for verifying modular components during development, but nothing builds confidence quite like a golden snapshot test that exercises an entire system under adversarial conditions. Knuth’s 40-year-old torture test reminds us that long before automated fuzzers existed, disciplined manual adversarial testing could produce software that endures for generations. Something to think about.