Watch the companion video for this post: Expressive error messages in a custom TeX engine

Perhaps you are a TeX wizard, but if you’re anything like me you’ve experienced the typical wall of error messages. You mistype a macro name or forget a closing brace, and the terminal explodes with a barrage of terminal text – cryptic line numbers, obscure internal macro expansion traces, and line breaks awkwardly splitting the offending token in two. In the 1980s, when TeX was developed for interactive teletypes and terminals with severe memory limitations, this compact reporting was an efficient compromise.

It reminds me of the kind of old errors I used to get from the gcc compiler, but over the last decades Clang and Rustc have helped completely redefine developer expectations for diagnostics. Part of this is aesthetic: modern compilers pinpoint errors with colored source code snippets, caret underlines, context notes, and automatic typo suggestions. But another part of it is philosophical: what if we think of these diagnostics as “help messages” rather than “error messages”?

When building a custom TeX engine, modernizing error reporting seemed like an obvious goal. However, doing so immediately runs into two deep challenges: first, automated conformance test suites (such as Knuth’s canonical TRIP test) demand byte-for-byte exactness with legacy log output; second, TeX’s macro-expansion pipeline makes tracking the true physical origin of an error surprisingly tricky.

Abstracting the Reporter Interface

The idea in solving this is not so complicated. I decoupled the generation of an error event from how that event is formatted and displayed. In the legacy tex.web codebase, error strings, help texts, and terminal output are tangled directly into the parsing and evaluation routines, and if you want to change how an error looks you risk altering the internal control flow of the engine.

Instead, we can introduce an abstract Reporter interface. When the parser encounters an error condition (say, an undefined control sequence or an overfull box) it packages the relevant diagnostic data into a typed event and hands it to the active reporter. Very schematically, something like

struct UndefinedControlSequenceEvent {
    std::string_view tokenName;
    SourceLocation location;
    std::optional<std::string_view> suggestion; // Levenshtein nearest-match
};

// A base class for handling polymorphic diagnostic events
class Reporter {
public:
    virtual ~Reporter() = default;
    virtual void emit(const UndefinedControlSequenceEvent& ev) = 0;
    virtual void emit(const OverfullBoxEvent& ev) = 0;
};

This kind of abstraction allows the engine to easily switch between two distinct reporting backends. A KnuthReporter can emits the strict, traditional TeX82 logging format (preserving the terminal output expected for the test suite), and an ExpressiveReportercan try to format diagnostics for the rest of us (with modern terminal styling, ANSI colors, file previews, and suggested corrections).

Tracking Token Provenance

While abstracting the reporter is straightforward, gathering the data needed for expressive diagnostics is where the real engineering challenge lies.

In a traditional compiler, an AST node usually corresponds directly to a span of bytes in a source file. In TeX, code travels through what Knuth described as the “eyes, mouth, and stomach”:

• The eyes read characters from disk into line buffers.
• The mouth tokenizes characters and performs dynamic macro expansion.
• The stomach ingests tokens to construct horizontal and vertical lists of layout nodes.

Because macros can generate new tokens, splice parameter tokens (#1, #2), and repeatedly re-tokenize strings via \scantokens or \csname, tokens frequently exist in memory without an obvious 1-to-1 link to a physical file location.

To provide Clang-style underlines, the engine must attach lightweight source provenance metadata to tokens as they are scanned. Each token carries a file ID, line number, and column offset. When a macro expands, the resulting tokens retain a breadcrumb trail pointing back to the call site. If an error occurs deep inside an expanded macro, the expressive reporter can display both the definition site and the exact line in your document where that macro was invoked.

Expressive Diagnostic Features

With provenance tracking in place, we can easily implement useful diagnostic improvements. For instance, when an undefined control sequence is encountered (for example, typing \tectbf instead of \textbf), the engine computes the Levenshtein distance between the unknown token and the identifiers currently registered in the hash table. If a close match is found within a small edit distance, the reporter suggests the correction inline:

error: undefined control sequence '\tectbf'
  --> paper.tex:42:5
   |
42 |     \tectbf{Introduction}
   |     ^~~~~~~ did you mean '\textbf'?

In the youtube video, you can see similar enhancements to things like the visualization of overfull boxes, or adding more contextual information when TeX encounters missing delimiters (such as improving on the classic missing $ inserted error).

Tradeoffs: Provenance vs. Memory

Of course, nothing comes for free. Attaching source coordinates to tokens increases their memory footprint and requires careful management during high-frequency macro expansions. If every intermediate token allocated heap memory for diagnostic tracking, compilation speed would crater.

So far I’ve balanced this by keeping token identifiers compact and storing extended source spans in auxiliary side-tables that only get indexed when diagnostic events are actually emitted. In the common case where documents compile cleanly this maintains overall performance, but continuing to work on this system (especially to handle runaway style arguments) is still something I need to think hard about.