The Fallback File: A Two-Hour Rehearsal for the Day the Warehouse Goes Dark

Few data platforms fail dramatically. They fail administratively: a certificate left to lapse, a service principal rotated without a handover note, a warehouse credit ceiling reached mid-morning, a single sign-on change that locks out the BI layer while the underlying tables remain perfectly intact. The data is fine. The route to it is not. And in that gap, the question that matters is not whether the pipeline can be restored but whether the business can still decide anything before lunch.

Most UK data teams have a continuity document. Fewer have rehearsed it. The distinction is not pedantic. A document tests the plumbing; a rehearsal tests the people, the permissions and the judgement calls that sit between an outage and a decision. What follows is a short, repeatable exercise — two hours, no theatre, no disaster-movie framing — designed to surface where an organisation is quietly fragile.

Why a rehearsal beats a runbook

Runbooks are written by the people who understand the system, for the people who understand the system. They are excellent at describing how to fail over a replica. They are much weaker at answering a different question: which decisions stop when the dashboard stops, and who is authorised to make them on worse information?

That question belongs to the business, not to the platform team. A tabletop exercise puts both groups in one room and gives them a clock. It costs perhaps half a day of five people's time — modest against the cost of discovering the answer during a real incident, when the finance director is asking for the month-end position and the only answer available is a shrug.

Set the terms before the lights go out

Fix the scope in advance. Choose one scenario — a full BI layer outage is the most useful default, because it is plausible, common and does not require anyone to speculate about fire or flood. Choose a working day, not a quiet one, and choose a period that includes a real deadline: a board pack, a regulatory return, a pay run, a trading update.

The room needs a facilitator who is not the data lead, a scribe who is not the facilitator, and business owners who are empowered to make the decisions being tested. Invite finance, operations, customer service and commercial. Ask each to bring the two or three questions they would genuinely need answered that week. Ask the data team to bring nothing except the truth about what exists.

One ground rule matters above all: nobody fixes the outage. The exercise is not an incident, and the platform team must resist the urge to solve. Their role is to answer 'can we get that number another way, and how long would it take?' honestly, including when the answer is 'not today'.

The midnight list: fifteen reports, three tiers

Before the session, assemble a shortlist. Not the full catalogue — the reports someone would genuinely want at two in the morning. Cap it at fifteen and assign each to a tier.

• Tier one: the organisation cannot trade, pay or report without it. Expect three to five items.
• Tier two: decisions degrade, but continue with delay or reduced confidence.
• Tier three: inconvenient to lose, comfortable to defer a week.

The discipline is in the cap. A list of ninety reports tells you nothing except that the catalogue exists; a list of fifteen forces a ranking that someone will dispute, and the dispute is itself a finding. If two departments rank the same report as tier one and tier three respectively, that disagreement has been quietly costing money for months.

The two-hour clock

Run the drill against a timer and narrate the scenario as it unfolds.

0–15 minutes: declare and orient. Announce the scenario, confirm who is in the room and restate the ground rules. The facilitator reads a short scripted statement — what is unavailable, what is believed to be the cause, what the supplier has said — and stops there. No speculation about recovery times beyond a deliberately vague initial estimate.

15–45 minutes: decide. Each business owner takes one live decision off their list and attempts it using whatever fallbacks exist: yesterday's extract, a finance system report, a supplier portal, a PDF already in someone's inbox. Record what was used, how long it took and how confident the owner felt.

45–75 minutes: escalate. Test the route upward and outward. Does the escalation contact list still contain the person who left in February? Does the supplier's contractual response time match the tier the business assumed? Does communications know what to tell customers, and who approves the wording?

75–105 minutes: examine the workarounds. Ask each participant to describe, without penalty, what they would actually have done in a real outage. This is where the exercise earns its keep.

105–120 minutes: debrief. Capture findings while memories are fresh, agree owners and dates, and set the date for the next run.

The grey estate: where the workarounds hide

Almost every organisation maintains an informal second data estate: the weekly spreadsheet emailed between two managers, the local copy of a customer list, the analyst who keeps a personal database because a request queue once took three weeks. These artefacts are usually treated as a governance failure to be stamped out. In an outage drill they look different — they are often the only reason the business keeps moving.

That should not be read as an endorsement. Undocumented copies carry version confusion, stale records and personal data that nobody has mapped. But the finding is twofold: the workaround exists because a legitimate need was never met through official channels, and it cannot be relied upon because it has never been tested. Record the grey estate, then decide deliberately which parts to formalise and which to retire.

What to record — and what counts as evidence

Keep a decision log rather than a narrative. For each attempted decision, capture seven fields: the decision itself, its deadline, the data source actually used, the time taken to obtain it, the owner's stated confidence, the cost of being wrong, and whether the decision would have changed had live data been available.

That final field is the one that reframes the conversation. Many dashboards turn out to be consulted rather than depended upon: the number is read, noted, and the decision is made the same way it was made last month. Others reveal a genuine dependency that nobody had articulated. The difference between the two is where investment should go, and it is rarely where the refresh schedule suggests.

Testing the human wiring

Technical resilience attracts attention because it is legible: replication, recovery objectives, failover tests. Human resilience is quieter. Does the on-call rota hold a number for the supplier account manager, or only a generic support queue? Is there more than one person who knows how to run the month-end extract? If the data lead is on leave, who signs off a restatement?

UK organisations already carry obligations in this territory — operational resilience expectations in financial services, NCSC guidance on incident response, ISO 22301 for those pursuing certification. A two-hour drill does not discharge any of them. It does, however, generate the kind of evidence auditors and boards increasingly ask for: not a statement that plans exist, but a record that they were exercised and found wanting in specific, correctable ways.

Make it a habit, not an event

Run the exercise quarterly and rotate the report shortlist so that no single area becomes over-rehearsed. Change the facilitator. Publish a one-page note afterwards — what held, what did not, what has been fixed — and keep it dry. The point is not to demonstrate preparedness but to keep discovering where it is absent while the cost of discovery remains low.

Resilience in data work is not a document that sits in a shared drive awaiting an auditor. It is a rehearsed habit, tested in daylight, by people who would rather find the gap on a Tuesday morning in November than at 8:40am on the day the warehouse actually goes dark.