Ten questions about a 10,000-household vehicle-ownership panel, each answered three ways by the same model with the same settings. The only thing that changes is what the model is given. Every number in every answer is checked against ground truth. Eight runs across five labs say the same thing.
Every household is generated from one timeline. A promotion at 38 moves the income bracket, which raises the price band of the next purchase; a second child triggers the minivan; retirement sells the second car. Each transaction carries the event that caused it, so questions like "what happens to spending after a new child" have a real answer to check against.
Synthetic, seeded, regenerated from code. Nothing here is a real person. Schema and generator in the repo.
Same model, same settings, same question. The page replays Claude Sonnet 5; the same ten questions were also run on Gemini 3.1 Pro, GPT-5.4 with and without reasoning, Grok 4.6 and Qwen 3.8, and Sonnet three times over. Ungrounded gets the schema only. Naive-grounded gets the raw records for the segment, pre-filtered by the pipeline, about two hundred households, and is asked to do the math. Pipeline-grounded gets the aggregated statistics the pipeline computed and is told to narrate nothing else. Then every number in each answer is checked against the pipeline's output.
Want to run it on your own model or key? MODELS.md in the repo is the runner: one environment variable per model, same questions, same check.
A question becomes a structured query. The pipeline filters and aggregates deterministically and returns statistics with explicit denominators, or an explicit "not answerable" with the reason. The model sees only that. The check then takes every number the model wrote and looks for it in the pipeline output.
Two servers: a data pipeline that never calls a model, and an orchestrator that never touches raw data. The pattern comes from analytics work I have done on real customer panels; this is an independent build from scratch, on synthetic data, with the argument made measurable.
How a number is checked: integer counts must be exact; percentages and dollar figures within 1.5% or half a unit at the stated precision. Every number the model wrote is looked up in the statistics the pipeline produced; if it is not there, and is not a simple difference or share of figures that are, it is marked as no match. A number restated within one answer counts once. Calibrated to 52/52 on a template narrator that prints pipeline values only, then hand-checked on every unverified claim across all eight runs — that sweep found five places where the check was wrong, not the model. The figures here are the ones after the fix. Cross-model table in AGGREGATE.md.
Every individual household the naive-grounded model quoted was exact: four "$94,844, HH-9032, 2015" citations, all correct. Every aggregate it computed was wrong. On the new-child question it chose a stricter window than the pipeline, a defensible reading, and then miscounted its own cohort: 57 households where there were 73, a before-mean of $9,456 where the truth was $9,733. On the EV question it counted 43 of 173 households; the answer was 44. On three questions it thought for over two minutes across 285k tokens, hit the output limit, and produced nothing. Qwen 3.8 did that on nine questions out of ten.
I ran Sonnet 5 three times to see how much of this was noise. Its naive-grounded rate moved between 62.9% and 79.2% — a spread wider than the one between five different labs, which sat between 48.7% and 62.2%. Pipeline-grounded did not move: 100% on every model except Sonnet, which lost the same two claims in all three runs. Swapping the lab changes the number less than running the same model again does. Changing what the model is given changes the answer.
GPT-5.4 at its provider default spends zero reasoning tokens. It answered a 225-record aggregation in ten seconds and verified at 48.7%. Told to reason at medium effort, it took a hundred seconds and verified at 62.2%. Both were 100% pipeline-grounded. The gap between those two numbers is smaller than Sonnet's gap with itself, and neither is close to the mode that does not ask the model to compute at all.
"I don't have access to the records, so I can't report an exact figure. For context, nationally EVs have represented roughly 7–9% of new sales, per Cox Automotive and Kelley Blue Book." Three external sources and three fabricated figures on a question that has no external answer. Asked how many households moved twice and changed commute, it estimated 3,000–4,500 citing "life-course research"; the answer was 6,125. Across eight runs the ungrounded mode invented 22 figures on the two questions that have no answer. The pipeline-grounded mode invented none, and refused all sixteen.
One question asked why empty nesters spent more in their 40s than their 50s. They didn't; mean spend was lower. The pipeline-grounded answer said so with the numbers. The ungrounded answer accepted the premise and explained it: "that stage typically generates more vehicle transactions." That is the question an analyst actually asks, and the failure that actually ships.
A verifier that scores the architecture I built is not evidence until someone reads its mistakes, so I read all of them — every unverified claim across all eight runs. Five were the check's fault, not the model's: household ids parsed as quantities, "$57,705 mean" read as 57.7 billion, an abbreviation cutting a citation off from its household. Fixed, with regression tests, and every run rescored. Pipeline-grounded moved from 99.8% to 99.7% and naive-grounded moved up five points on the headline run. The numbers on this page are the ones after the fix.
Three of 1,003 pipeline-grounded numbers had no match, all in Sonnet's runs, none of them invented. Twice on Q07 the narrator reported the complement — 10,000 households less the 6,670 with a qualifying move count, given as 33.3% — and once on Q03 it wrote that one cohort was 8.8× the other, which is 6,251 ÷ 714. Every input to all three was verified; only the arithmetic laid on top had no field to match against. The narrator is allowed one step, and these are two. Both personas stayed inside the rules, but the strategist voice, prompted to find opportunities, pushed closer to the edge of what its statistics support. The check is what makes that visible.
Ten curated questions. Eight runs, six model configurations, one persona for the cross-model comparison. The check is strict and was calibrated on real answers, then audited claim by claim; every pass is in the repo. A synthetic panel has cleaner causal structure than a real one, which makes the grounded mode's job easier and the naive mode's harder in equal measure; the comparison is between modes, not between this data and yours.
The system prompts differ between modes because the tasks differ, and two counts inherit that. Pipeline and naive are both told not to use outside knowledge; ungrounded is not, so the external-knowledge counts are not a clean three-way comparison. The same holds for refusal, where the meaningful figure is 10 of 16 against 16 of 16, both instructed. And nine of the ten exposes notes are unchanged since the commit that predates every run: Q05 is the exception, its question and its note rewritten in the same commit as the first run, so the false-premise framing there is a finding rather than a prior intent.