Watch the companion video for this post: How hardware constraints shaped TeX’s literate programming

When first opening the source code of TeX (whether in Donald Knuth’s original tex.web or via translations like arusson’s tex-c) the reaction of most modern programmers is a mix of awe and disorientation. Global variables are mutated indiscriminately across thousands of lines, macro spaghetti and gotos send control flow careening across the code base… it was simultaneously radically performant for its time and also very hard to reason about.

While it’s tempting to dismiss this kind of “macro spaghetti,” I think doing so misses the entire point of how software engineering evolves. Knuth’s tex.web is a masterpiece of literate programming, designed under computational constraints that are almost unimaginable today.

Before refactoring a 40-year-old codebase, I wanted to explore a little bit more about why it was built that way.

The Computing Environment of 1982

TeX82 was designed on a DEC PDP-10 mainframe running the WAITS operating system. In that environment, computing resources looked very different:

• Core memory was measured in kilobytes or tens of thousands of 36-bit words (which, you’ll notice, is not divisible into 8-bit bytyes!).
• Memory was so scarce that even holding the full code for initialization and runtime in RAM simultaneously was impossible. In fact, iniTeX was originally built as a completely separate executable because including format-initialization routines inside the main engine would leave insufficient RAM to typeset a page.
• Compilers did not perform aggressive dead-code elimination, function inlining, or register allocation.
• Calling a procedure incurred a measurable runtime penalty for stack frame manipulation.

Under these conditions, the design choices of TeX look like a brilliant set of engineering trade-offs. The WEB system allowed Knuth to write essays explaining the algorithmic logic in human terms while the preprocessor (“tangle”) expanded macros inline and reused global memory locations, delivering the performance of hand-tuned assembly while maintaining literate documentation.

Deterministic Fixed-Point Math

One of the most consequential decisions in TeX’s architecture was the complete rejection of hardware floating-point arithmetic. As a scientific programmer today, it is hard to appreciate that in 1982 IEEE standardization had not happened: floating-point implementations varied wildly across hardware architectures. Furthermore, things like “should I round down, or round closer to zero” might differ from one compiler to another.

This led to Knuth basing (almost) all calculations on fixed-point arithmetic using scaled point integers (sp). For instance, 1 pt = 65536 sp. But even this was often tricky. In 1982, avoiding 32-bit arithmetic overflow during multiplications and divisions without floating-point support required intricate manual arithmetic routines. But in modern C++ (where language standardization has happened, and where 64-bit integer representations exist as a language construct), many things becomes much easier. For instance, we can replicate Knuth’s exact division semantics and overflow avoidance cleanly by just casting to a wider type:

struct DivisionResult {
    scaled quotient;
    scaled remainder;
    bool overflow;
};

inline DivisionResult x_divided_by_n(scaled x, int n) {
    if (n == 0)
        return {0, x, true};

    int64_t wide_x = static_cast<int64_t>(x);
    int64_t wide_n = static_cast<int64_t>(n);

    int64_t q = wide_x / wide_n;
    int64_t r = wide_x % wide_n;

    return {static_cast<int>(q), static_cast<int>(r), false};
}

You can take a look at the tex.web source to see, I think, just how much simpler this fixed-point integer code has become compared to the original!

More language features that make things more transparent

The video talks about other features where the constraints of the time led to more difficult code. For instance, PASCAL did not have easy support for multidimensional arrays, so the legendary rules for spacing in the typesetting of mathematics had to be encoded in a cryptic 1D string. That… well, that’s a solved problem. In modern C++, we can replace cryptic numeric lookups with strongly typed enumerations and constexpr lookup tables. The mathematical spacing behavior remains identical to Knuth’s original specification, but the code becomes self-documenting.

Perhaps the crown jewel of TeX’s engine is the Knuth-Plass paragraph line-breaking algorithm. Unlike simple greedy line-breaking algorithms (which fill a line until it overflows and then break), Knuth-Plass evaluates the paragraph as an interconnected whole using dynamic programming. And yet the version of Knuth-Plass that exists in TeX82 – entangled as it is with the internals of TeX’s memory representation of different nodes, and error reporting, etc – makes it hard to see the beauty of the dynamic programming approach.

Understanding these 1982 constraints helped clarify for me the path forward for modern engine development. The genius of TeX lies in its algorithms: the Knuth-Plass line breaker, the fixed-point layout determinism, and the mathematical spacing rules.

Ideally, we can retire a lot of the the manual memory management, the global state mutation, and the preprocessor trickery that was necessary to fit those algorithms onto 1980s mainframes. Instead, we can atleast try to encapsulate those algorithms inside clean abstractions, pure functions, and structured memory models without sacrificing a single point of Knuth’s original typographic brilliance, or the performance of the engine.