The Answer Key Is the Contract: Grading Natural-Language SQL Before It Touches a UK Production Warehouse

The most persuasive natural-language SQL demonstration is often the one that answers a question nobody thought to ask. Fluency is not correctness. For a UK data leader, the relevant question is narrower: can the tool answer questions whose answers the business has already agreed? The known-answer test is a half-day exercise that replaces demo theatre with evidence. It is not a benchmark of general model capability. It is a governance gate.

The case for such a gate is strengthening in 2026. Natural-language query interfaces are being connected to warehouses, semantic layers and reporting marts while procurement teams are still using feature lists as proof. A vendor can show a polished answer to a plausible question. What it cannot easily show is whether that answer matches the organisation's settled definition of net revenue, active customer or financial-year-to-date margin. Those are the questions that matter, and they already have answers held by finance, operations and commercial teams.

The answer key is a control, not a demo script

The test begins with a simple inversion. Instead of asking a model to generate questions and judging whether the answers look reasonable, the business supplies fifty questions and the values it knows to be correct. The tool is then run against a frozen copy of warehouse data, with prompts issued exactly as a user would type them and no coaching from the vendor. The output is graded on exact match, near miss and confident nonsense.

This distinction is important. A near miss can be discussed, corrected and retested. Confident nonsense is the failure mode that damages trust, because it arrives with the same polish as a correct answer. A tool that returns a plausible but wrong figure for statutory reporting, board packs or regulatory returns is not a useful assistant. It is a new source of uncontrolled variance.

Building fifty questions the business can defend

The questions should be drawn from decision-critical reports rather than invented for the occasion. A workable set covers five families, with ten questions in each. The first family covers core financial metrics: gross and net revenue, margin, credits, refunds and the treatment of cancelled orders. The second covers customer and entity definitions: active customer, churn, onboarding, consent status and account hierarchy. The third concentrates on date logic, including the UK financial year beginning 6 April, month-end cut-offs, rolling periods and time-zone handling. The fourth tests joins across customer, product, store and invoice structures. The fifth tests filtering: nulls, soft deletes, test accounts, returns and status codes.

Each question needs a written expected answer, the grain at which it is measured and the report or process where that answer is already used. This is not bureaucracy. It is what separates a controlled test from a quiz. If the business cannot agree the answer, the question should be removed until it can. Contested definitions belong in a metric custody process, not in a vendor evaluation.

Running the half-day without coaching

The exercise works best as a single half-day with three roles: a facilitator, a scribe and a business reviewer. The facilitator issues the prompts verbatim and does not help the tool. The scribe records the prompt, the generated SQL, the returned value, the execution time and any error message. The business reviewer compares the result with the answer key. A read-only replica or sandbox should be used, not production tables, and the dataset should be frozen for the duration so that results are comparable.

It is useful to run the same fifty questions twice: once with the tool's default settings and once with the semantic layer or glossary enabled. That comparison often reveals more than a day of slides. It shows whether definitions are being applied consistently and whether the tool is relying on table names, column names or a curated business vocabulary.

A scoring model that penalises confidence

A simple four-part score is sufficient. The point is not statistical sophistication. The point is to make risk visible to a sponsor.

Outcome	Definition	Treatment
Exact match	Value and grain match the agreed answer	Counts toward pass
Near miss	Recognisably close but wrong on filter, date, grain or rounding	Requires explanation and retest
Confident nonsense	Plausible answer with no basis in the agreed definition	Fails the test
Refusal or error	Tool declines, times out or returns an error	Recorded separately

Confident nonsense should carry the heaviest penalty, especially on the twenty questions designated as critical. A tool that says it cannot answer is safer than one that invents a number. Sponsors should see the critical-question score separately from the overall score, because a high average can hide a dangerous failure in a statutory or board-level metric.

The failure taxonomy to record

Every failure should be classified before the half-day ends. Join failures include fan-out from one-to-many relationships, missing bridge tables and duplicate dimension rows. Date failures include inclusive versus exclusive boundaries, calendar versus fiscal periods, partial-period comparisons and null dates. Filtering failures include incorrect null semantics, ignored soft deletes, misread status codes and test accounts left in the population. Metric failures include gross versus net treatment, revenue recognition rules, customer activity definitions and aggregation at the wrong grain.

The taxonomy matters because it turns a score into a remediation plan. A vendor can fix a date-boundary bug. It is harder to fix a tool that consistently ignores the organisation's definition of an active customer. The pattern of failures is the finding.

Gating criteria before production access

A UK data leader can put five conditions in front of an internal sponsor or vendor. First, the tool must achieve exact match on all twenty critical questions, with zero confident nonsense across the full fifty. Second, it must achieve at least 80 per cent exact match on the remaining thirty, with near misses documented and accepted by the business owner. Third, every prompt, SQL statement and returned value must be logged and available for audit. Fourth, access must be read-only and restricted to a replica or sandbox until sign-off. Fifth, the vendor must accept retesting after any material schema change, model update or warehouse migration.

If a vendor resists those terms, that is itself a finding. The answer key is not an obstacle to adoption. It is the contract that makes adoption defensible.

Repeatability is the real deliverable

The fifty questions should become a governed asset, versioned alongside the semantic layer and reviewed when metrics change. Re-run the test quarterly and after any significant change to the warehouse, the model or the business glossary. Keep the prompts and expected values in the same repository as the definitions they test. That way the organisation is not buying a demonstration. It is buying evidence that the tool can be trusted with the questions the business already knows how to answer.