Byte for byte: how we test a template engine upgrade
Every Accent theme is a set of MiniJinja templates, and every page you serve or build passes through that engine. MiniJinja 3.0 shipped on 8 October 2026. It is a breaking release that brings the engine closer to Python’s Jinja2, and it changes what some templates render.
For an upgrade like this, “it compiles and the tests pass” is not good enough. The engine decides what your pages say. So before deciding whether to move, we asked one question at seven levels, from the cheapest to the most expensive: what actually changes?
Rule one: measure the noise first
Every comparison ran against itself before it ran against the other version.
- Building the same site twice with the same binary: 0 files differed.
- Fetching the same page twice from the same server: only the CSP nonce differed, which is new on every response by design.
- Taking the same screenshot twice: 2 pixels differed on one page.
- Running the same benchmark twice: results drifted by up to 2.8 percent.
Without that floor, a difference between two versions is a rumour. With it, it is a finding. You will see below why the floor also has to include a process restart.
1. Let the compiler talk
We changed one line, the MiniJinja version, on a throwaway branch and compiled. 67 errors came back from 47 call sites, in two patterns: a removed conversion function, and a macro that now takes ownership of its values. The fix was about 100 changed lines, all mechanical.
The compiler is the cheapest reviewer, and it was the least informative. Everything it found was easy. The interesting changes were the ones it could not see.
2. Probe what compiles but changes
A changelog is a list of hypotheses. For each entry that could touch Accent, we wrote a few lines that ran the same input on both versions. Three of them changed behaviour without a compile error:
- Building a map from key-value pairs. On 2.23 it produced a map; on 3.0
it produces a list of pairs. Accent did this when it handed nested
frontmatter to your templates, so
{{ page.custom.hero.title }}would have rendered as an empty string, with no error anywhere. - Keyword arguments to filters that do not take them.
{{ site.current_language | default(value=site.language) }}is how Tera, and therefore Zola, writes a fallback. MiniJinja 2.23 accepted it quietly, and printed the literal{"value": "en"}whenever the fallback actually fired. 3.0 rejects it. Two of our own development themes contained exactly this line, and 42 of that site’s 44 pages failed to render. - Key order. Objects that Accent converts for templates now list their fields in declaration order instead of alphabetically, which matters only to a template that loops over an object’s keys.
Probes tell you what could break. The next levels tell you whether it does.
3. Contract tests: tests that are supposed to fail
Accent keeps 100 tests that pin the exact output of MiniJinja’s built-in filters, tests, and functions. They exist for this moment. On 3.0, 11 of them failed, all for the same reason: printed lists and maps now use Python’s quoting, and pairs are tuples.
| Template | 2.23 | 3.0 |
|---|---|---|
{{ "a,b,c"|split(",")|list }} | ["a", "b", "c"] | ['a', 'b', 'c'] |
{{ {"a": 1}|items|list }} | [["a", 1]] | [('a', 1)] |
{{ dict(a=1, b=2) }} | {"a": 1, "b": 2} | {'a': 1, 'b': 2} |
That is the contract tests doing their job: they turn a one-line release note into an exact list of what your templates would print differently.
4. The full test suite
The whole suite, 6,562 tests, failed 30 on the first run: 18 from the map change the probes predicted, the 11 contract tests, and 1 test that pins the alphabetical key order. With the map fix in, 12 remained, and every one of them is either an expected new output or a decision we still have to make.
5. Render everything and diff the bytes
Tests check what someone thought to test. A build diff checks everything. We
built the documentation site, 729 pages and 3,174 files, with both versions
and compared the output byte for byte: no file differed. The starter site
was identical too, and so was the development site once its
default(value=...) line was fixed.
6. What a build cannot see
The admin UI is rendered only by accent serve, never by a build. It is also
where most of the upgrade’s code changes landed. So we ran both versions side
by side, signed in to each, and captured 15 pages: every admin screen and a
sample of public pages, as raw HTML and as full-page screenshots.
The HTML was identical once the per-response CSP nonce was set aside. Of the 16 screenshots, 29.8 million pixels in all, 14 were identical, and one page differed by 2 pixels, the same 2 it differs by from itself.
Then one page differed for real.
The schema browser listed its two content models in a different order on the two servers: 2,366 pixels of difference. It would be easy to blame the new engine. Instead, we restarted each version ten times and looked again. Both versions listed the models in both orders. The cause was ours: the models live in a hash map whose order changes with every process, and the list was never sorted.
That is a small, unrelated bug, and exactly the kind of difference a single before-and-after comparison pins on the upgrade. The noise floor has to include a restart, not only a repeated request.
7. Speed, with a drift check
A benchmark run once on each version shows a difference, not its cause. We ran Accent’s template benchmark three times in sequence: 2.23, then 3.0, then 2.23 again. The rerun tells you how much the machine drifts on its own.
| Case | 2.23 | 2.23 rerun | 3.0 | Change |
|---|---|---|---|---|
| Leaf page | 6.55 µs | 6.56 µs | 4.90 µs | -24.7% |
| Listing, 10 pages | 8.40 µs | 8.34 µs | 6.42 µs | -23.5% |
| Listing, 100 pages | 32.6 µs | 32.2 µs | 28.0 µs | -14.2% |
| Listing, 1,000 pages | 277 µs | 270 µs | 246 µs | -11.3% |
| Docs sidebar, 210 nodes | 153 µs | 153 µs | 141 µs | -7.5% |
3.0 renders every case 7.5 to 24.7 percent faster, and the rerun moved by at most 2.8 percent, so the gain is real. The smallest pages gain the most, which suggests the cheaper part is turning Accent’s page data into template values rather than rendering itself.
What this means for your themes
Accent’s own sites and admin UI render the same on both versions. Your theme might not, because Accent cannot test templates it has never seen. Check yours for these:
| If your template uses | On 2.23 | On 3.0 |
|---|---|---|
"don't stop"|title | Don'T Stop | Don't Stop |
2.5|round | 3.0 | 2.0 (rounds half to even) |
7 // -2 and 7 % -2 | -3 and 1 | -4 and -1 |
1 / 0 | inf | An error |
true is number | False | True |
A printed list or map, {{ items }} | Double quotes | Single quotes, tuples for pairs |
default(value=x) | Works by accident | An error; write default(x) |
-page.weight | An error | Works |
Most themes use none of these. If yours does, the fix is usually one line,
and the default(value=...) one is worth making today: it is wrong on the
current version too.
The upgrade is in review. When it ships, the release notes will carry this
list, and part of the work is better error messages in development: instead
of undefined value, a template error will name what was missing, as in
undefined value: `page.hero` is undefined (from line 2).
Use the method on your own upgrades
None of this is specific to MiniJinja. For any dependency whose output you ship:
- Measure the noise floor first, including a process restart.
- Read the changelog as hypotheses and probe each one with the same input on both versions.
- Keep contract tests that pin the dependency’s output, so a release note becomes an exact list of changes.
- Diff the built output byte for byte, not a sample.
- Diff what the build cannot see: server-rendered pages and screenshots.
- Benchmark A/B/A, so you know the drift before you believe the delta.
The compiler found 67 problems, and every one was easy to fix. The probes, the diffs, and a restart found the ones that would have reached your pages.