holdout questions frozen 2026-09-29T16:34:31 (ET)
sha256(json.dumps(holdout, sort_keys=True)) = c8262d15fe5205119b3d0aae213e3ff00ba1e0ad78723626fd4c5c091142f74f
n = 40
Written before engine.js existed. Same author as the engine: a regression check, not an independent benchmark.
The holdout was seen once by the engine author after v1 was built; see FAILURE_ANALYSIS.md for v1 numbers before any holdout-informed changes.
Caveat: the engine author wrote the holdout questions and then wrote the engine with those questions in mind. The hash proves the questions were not edited afterwards, not that they were unseen.