After the Number Breaks: A Blameless 48-Hour Review for UK Data Teams
Photo: NASA/SAO, Public domain, via Wikimedia Commons
Missing numbers are rarely just a pipeline problem. By the time a finance pack shows a blank tile or a regulator asks why a figure moved, the incident has already travelled through people, processes and promises. UK data teams are generally good at restoring service. They are less consistent at extracting the organisational learning that stops the same failure returning under a different filename.
A 48-hour review is not a witch-hunt. It is a bounded, blameless exercise that starts once service is stable and ends with a one-page note senior leaders can actually use. The aim is not to identify a culprit. It is to reconstruct what happened, understand who was affected, quantify the decision risk and leave behind changes that survive the next busy quarter.
Start with containment, not confession
The first mistake in many incident reviews is timing. Teams either convene a meeting while engineers are still firefighting, or they wait until the following week when logs have rotated and memories have softened. A better pattern is to separate containment from review. The incident commander restores service. A nominated review lead, ideally someone close enough to understand the system but far enough from the immediate fix, begins preserving evidence.
That evidence should include pipeline run histories, dashboard version changes, access logs, alert timestamps, Slack or Teams threads and the names of the reports affected. It should also include the informal context: who first noticed, what they were told and which meetings used the suspect figure. This is the material that turns a root-cause statement into an organisational lesson.
Hours 0 to 4: stabilise and freeze the facts
Once the immediate issue is contained, the review lead should create a simple incident record. Record the time the issue is believed to have started, the time it was detected, the time service was restored and the time affected users were informed. Those four timestamps alone often reveal more about monitoring and communication than any post-mortem narrative.
At this stage, resist the urge to explain. The goal is a fact base, not a theory. Ask system owners to export relevant logs before retention windows close. Ask report owners to save copies of affected outputs. Ask the person who raised the alarm to write a short note on what they saw. Keep the language neutral and the scope tight.
Hours 4 to 12: reconstruct the timeline
A useful timeline distinguishes between technical events and human events. Technical events include failed jobs, late arrivals, schema changes, permission changes and deployment activity. Human events include the first report of an anomaly, the first escalation, the first communication to a business owner and the first decision taken on incomplete information.
The review lead should aim for a single chronological table with three columns: time, event and evidence source. This avoids the common failure mode in which the technical timeline is meticulous while the human timeline is reconstructed from memory. For UK teams operating across multiple time zones or supplier boundaries, the table should also note who was responsible for each hand-off. Many incidents are less about a broken job and more about a broken hand-off.
Hours 12 to 24: map the blast radius
Blast radius is usually defined too narrowly. A broken pipeline may affect one dashboard, but that dashboard may feed a board pack, a regulatory return, a pricing decision or a customer-facing service. The review should map affected data products to their consumers, then to the decisions those consumers support.
A simple three-column register is enough: data product, consumer and decision at risk. Include informal consumers as well as formal ones. A spreadsheet downloaded by a regional manager and emailed to a finance business partner is still part of the blast radius. This step often exposes the difference between the systems teams think they support and the reports the organisation actually relies on.
Hours 24 to 36: connect the incident to decisions
Once the blast radius is visible, the review should ask what decisions were taken while the data was wrong, missing or late. This is the step most reviews skip. A pipeline can be fixed in hours, but a pricing decision, a stock allocation or a regulatory submission may carry the consequences for weeks.
The review lead should speak to the owners of affected decisions and record three things: what they decided, what information they used and what they would have done differently with earlier warning. This is not about blame. It is about designing better signals. If a business owner says they had no way of knowing the figure was stale, that is a governance finding. If they say they knew but had no alternative, that is a decision-intelligence finding.
Hours 36 to 48: write the one-page board note
The final artefact should fit on one page. A board or executive committee does not need a forensic account of every job failure. It needs a clear statement of what happened, what was affected, what the organisation did, what remains uncertain and what change is being sponsored.
A practical structure is: incident summary in three sentences; affected decisions and reports; customer, regulatory or financial exposure; immediate actions completed; two or three systemic fixes with owners and dates; and one explicit ask of leadership. The ask might be funding for observability, a decision to retire a fragile report or a mandate for a data owner to resolve an ownership gap.
The learning that survives the week
Most incident reviews produce actions that are sensible and quickly forgotten. To avoid that, each action should be assigned to an existing governance forum rather than a temporary working group. Data ownership gaps belong in the data governance backlog. Monitoring gaps belong in platform investment. Decision-impact gaps belong in the design of management information.
The test of a 48-hour review is not whether the root cause was found. It is whether the organisation can point to a change in behaviour, a retired report, a clearer escalation path or a better early warning six weeks later. In that sense, the review is a governance product in its own right. It should be designed, owned and improved like any other.
