Watch the companion video for this post: A standard library for TeX, part 1: format files
When building NemaTeX, I’ve been particularly interested in the boundary between features that sit at the engine level vs the macro level. Some of these were foundational issues (I think the engine itself should know about modern fonts, and different output file formats, and pdf tagging, and…), and those were the real original motivations for the project.
As the project developed, I’ve been increasingly thinking about the concept of a “standard library” for the TeX engine – are there some macro packages that are so common that their functionality should be baked into the engine itself? Lots of potential features might spring to mind (something like BibTeX for citations, or a package for graphics management, or…), but the things that particularly draws my attention are packages that feel like they are somehow fighting against how TeX works rather than working with it.
Format files
The first kind of thing that came to mind were packages like mylatex and mylatexformat. These packages try to address the fact that (a) TeX is fundamentally built to load in configurations of macros via its .fmt file dumping and loading mechanisms, (b) a very standard workflow involves a “preamble” of a document that sets up LaTeX packages and document formatting parameters and which is edited very rarely and a “body” which is edited and compiled repeatedly, but © it is surprisingly hard to set up a smooth workflow to generate a custom .fmt file based on the preamble you are actually using.
These packages are very cool, but there is a lot of friction in using them. They require multi-pass workflows (compile the preamble, separately compile the document, remember to recompile the preamble every time that preamble occasionally changes), and there are plenty of sharp edges.
The root cause of this friction is, I think, that these packages are fighting with the engine. Knuth’s original \dump primitive has very strict, hardcoded constraints. For instance, it can only be called once during a special initialization run, it intentionally discards certain types of state, and it immediately terminates the engine process after writing the .fmt file.
Building NemaTeX from scratch, we’re not bound by the same constraints. It is relatively straightforward to serialize and deserialize arbitrary amounts of engine state at arbitrary points during document compilation.
To test this idea, I added a new primitive: \checkpointformat.
An ergonomic, automatic workflow
With this primitive – and combining it with a new CLI flag – we can build a much more natural workflow for automatic format-file generation. The idea is to automatically detect not only if a precompiled format file exists (and, hence, should supersede something like the latex.fmt format), but also can correctly reason about whether that format file needs to be regenerated.
Heavy packages like TikZ involve scanning thousands upon thousands of lines of tex code when something like \usepackage{tikz} is encountered, and skipping all of that work by loading the state of the engine after such a command can substantially improve compile times. In the video, I demonstrate editing the body of a (particularly simple) TikZ image and recompiling in about 20 milliseconds.
A hint at the next standard library feature
At the end of the video, I also ran a much heavier TikZ workload: calculating a mathematically complex wireframe mesh of a quartic surface. All that heavy coordinate math, path routing, and solving takes pdfTeX about 2.8 seconds to compile. Right now NemaTeX handles exactly the same TikZ code in roughly 200 milliseconds (something I am confident could, in fact, be even faster with more work on my end). While bypassing the preamble certainly helps, that order-of-magnitude speedup is probably an easy hint about what the next entry in this “standard library” series will be.