Failure analysis: rule-based talk-to-data engine (SYNTHETIC data)
Engine: assets/engine.js, deterministic rules, no LLM. Questions: 124 (eval/questions.json): 54 dev, 40 holdout, 30 stress. Gold answers = hand-written SQL run in SQLite (eval/score.py), independent of the JS engine. Regenerate with node eval/run_eval.mjs, then the scoring script (eval/score.py).
How to read the numbers
| Split | Written | Used for | v1 | v2 |
|---|---|---|---|---|
| dev (54) | alongside the engine | tuning | 54/54 | 54/54 |
| holdout (40) | before the engine, hashed (HOLDOUT_FROZEN.txt) |
generalisation check, scored once on v1 | 40/40 | 40/40 |
| stress (30) | after v1, adversarially | finding failures | 15/30 | 21/30 (partly tuned) |
- Dev 100% only means the engine does what its author meant. On the first dev run it scored 52/54 (96.3%): the "by brand" phrase was split wrongly by a regex. That one bug was fixed, then dev was 54/54.
- The holdout was written before the engine but by the engine's author, who then had those questions in mind. The hash shows they weren't edited, not that they were unseen. 40/40 is a regression check, not evidence of accuracy.
- The honest untuned number is v1 on stress: 15/30 (50%). v2 was changed after seeing those failures, so 21/30 is partly tuned and shows a trade-off, not generalisation.
- 124 questions from one author: differences of a few points are noise. This is not a benchmark against any other system.
- Baselines on the same questions: always-refuse 4%, always-return-total-net-sales 0%.
Failure taxonomy
Each failure gets one label. Severity order: missed_refusal/leak on access questions (safety) > wrong_answer (silent misinformation) > false_clarify/false_refusal (unhelpful but safe).
v1 (15 failures, all on stress)
| Label | n | Questions | Root cause |
|---|---|---|---|
| wrong_answer | 9 | s03 share of total, s04 average monthly, s07 two quarters, s11 threshold, s12 "latest quarter", s13 "second quarter of 2025", s14 "January through March 2026", s16 "which 3 regions…", s17 top-per-group | Three different causes: (a) unsupported constructs silently ignored (s03, s04, s07, s11, s17): the parser found a metric and a year and dropped the rest; (b) date phrasings not covered (s12, s13, s14); (c) "which 3" limit not parsed (s16). |
| false_clarify | 4 | s01, s02 typos; s18 Spanish; s30 "top-line" | Fixed vocabulary; asks "which metric?" (safe, unhelpful). |
| missed_refusal | 1 | s08 "Why did net sales drop…" | Causal question answered as a plain aggregate: the engine describes, it cannot explain. |
| wrong_outcome | 1 | s19 "profit for Ridgeline" as a Sales analyst | "profit" alone is not a defined term, so the role got a clarification instead of a refusal. No data leaked, but the reason was wrong. |
v2 (9 failures, all on stress; 0 wrong answers, 0 safety-critical)
| Label | n | Questions | Cause / decision |
|---|---|---|---|
| false_refusal | 5 | s03, s04, s07, s11, s17 | The v2 guard refuses constructs it can't compute (share, average, threshold, several periods, top-per-group). Deliberate trade-off: a refusal is cheaper than a confident wrong number. Proper fix = new metric definitions or a query planner, not more regexes. |
| false_clarify | 4 | s01, s02, s18, s30 | Unchanged: typos, another language, jargon. Needs a normaliser in front of the semantic layer. |
What changed in v2, and why each change is limited
- Unsupported-construct guard (keyword-based): fixes the silent wrong answers by turning them into refusals. It only catches the constructs I listed.
- Fail-closed profit/cost wording under restricted roles. Found by probing after the stress set: "net sales and what it cost us to make it" answered the net-sales half and silently dropped the cost half for a role that may not see cost. Now refused, with regression tests g31–g35. I can't prove no other paraphrase slips through; the control that matters is the metric-level deny on the resolved query, this is defence in depth.
- "why" / causal → out of scope.
- Three date phrasings ("second quarter of 2025", "January through March 2026", "latest quarter") and an N-parse fix for "which 3 …".
Known remaining weaknesses (not measured by the 124 questions)
- Aggregates can be differenced by a user with a row-scoped role (total minus visible = hidden). No minimum group size or query budget.
- Keyword guards fail closed on tested phrasings only.
- No conversation memory ("and for the West?" → clarification).
- The question set is one person's guess at user phrasing. A real eval needs sampled real questions with human-labelled gold and a second annotator.
- Everything runs client-side, so policy is demonstrable, not enforceable, in this demo.
Path to an LLM-backed engine (not built)
Keep this harness and the ask(question, role) contract. Let the model propose a structured query (metric, dimensions, filters, period) validated against the semantic model; keep the policy layer outside the model. Score it on the same 124 questions plus a larger paraphrase set, report accuracy, false-refusal, missed-refusal (the number that gates release), cost and latency, and compare against this rule baseline.