The DR Plan That Survives the Day: Who Decides, Who Communicates, When Systems Fail

When the main system goes down at 3 a.m., the first question in the group chat is usually not which tool to recover with. It is who has the authority to decide, and who to call right now.

Most Disaster Recovery (DR) Plans answer the technical questions well. They say where to restore data from, which standby to shift load to, and in what order to bring the system back. What often goes missing is the human side: who declares that this is a serious incident, who orders the failover, and who tells users what is happening.

This article is written from the perspective of the team that has to recover the system for real, not the one that finishes the document and files it away. We will walk through the decisions people face in the first hour of an outage, the window where the plan on paper and what actually happens are furthest apart.

A good plan on paper does not guarantee a smooth night.

One figure explains why we focus on people over tooling. In Uptime Institute's Annual Outage Analysis 2025, which surveys data center operators worldwide, around 40% of organizations had suffered a major outage caused by human error in the past three years, and 85% of those stemmed from staff not following procedures, or from flaws in the procedures themselves (source: Uptime Institute, 2025). The signal is clear: most DR failures fail at the level of process and human decisions, before technology enters the picture.

Who has the authority to declare a disaster

A DR Plan is never triggered if no one is willing to say the situation is serious enough. In real incidents, what happens often is that everyone sees the system behaving abnormally, but each person waits for someone else to make the call, and half an hour passes with no one starting the plan. That lost time is part of the downtime that was entirely preventable.

The fix is to define the declaration criteria in advance, before the incident, not in the middle of the panic. Those criteria should be tied to things you can measure: the core service unavailable beyond a set number of minutes, the request error rate above a ceiling, or data starting to diverge between two copies. Once any one threshold is crossed, treat it as incident mode immediately, without debating whether it is really serious.

Once an incident is declared, someone must hold the role of lead decision-maker for it. This role is usually called the incident commander. Their job is to hold the overall picture, decide, and keep the technical team from being interrupted, not to fix the system themselves. The right person is not necessarily the most senior; they need the authority to give orders during the incident, and there should always be a designated backup, because incidents tend to happen outside working hours when the primary person is unreachable.

A common objection is that the team already knows the system well, so why formalize roles at all. The question is fair, and the answer is not that the team is incapable. The people who run the system every day understand it better than anyone. What defining roles adds is not knowledge but speed of decision when everyone is stressed and information is still incomplete. In normal times a team decides well together; during an incident, waiting for consensus is lost time.

The international business continuity standard ISO 22301 frames this at the organizational level: roles and decision authority must be defined before an incident (reference: ISO 22301:2019). On the security incident side, NIST SP 800-61 makes the same point, assign responsibilities and escalation paths ahead of the day it matters (reference: NIST SP 800-61 Rev.2). Neither framework mandates a fixed shape, but both agree that decisions need a clear owner.

Decision criteria should be numbers, not feelings

The heart of both declaring an incident and ordering a failover is criteria written as numbers in advance. At 3 a.m., no one wants to interpret what serious or too long means. Good criteria reference two values the team should already know for each system: RTO, the maximum time the service is allowed to be down, and RPO, the amount of recent data you can afford to lose.

With these two values, the decision becomes a matter of comparing numbers. If the primary is estimated to return later than the RTO window, proceed with the failover. If the failover would lose more data than the RPO allows, you need a procedure to reconcile the gap, such as a later back-reconciliation. These criteria belong in each system's runbook, not in one person's head.

Setting sensible RTO and RPO values is design-time work, not incident-time work. NIST SP 800-34 describes how to derive them through a business impact analysis, which ranks how critical each system is and how long it can be down (reference: NIST SP 800-34 Rev.1). It is done once and used for a long time, and it is the basis for every decision made on the night of an incident.

Failover or wait is the most expensive decision of the night

Once an incident is declared, the next decision is usually a fork where both paths carry risk. One is to order a failover to standby immediately, fast, but at the risk that the last slice of data is not complete within the RPO. The other is to wait for the primary to recover, safer for data, but it stretches the downtime, and there is a chance the primary does not come back when expected.

What makes the decision fast is a time threshold agreed in advance, tied to that system's RTO. Once the set line is crossed, order the failover without walking it up an approval chain step by step. This kind of threshold turns the decision from guessing what an executive wants at 3 a.m. into following what the whole team agreed while heads were cool.

In practice the choice is not always two poles. Some systems support a partial failover, for example moving reads to standby first, then moving writes once you are confident. Having a middle option like this has to be designed and tested in advance, because no one invents a new architecture while the system is down.

To be clear: not every system needs a decision structure like this. If your system is internal, has a few dozen users, holds no personal data, and can be down for half a day without affecting anyone, an incident commander and failover thresholds are overkill. What is worth more in that case is an up-to-date contact list and one recovery procedure that has actually been tested. Complexity no one uses does not disappear; it becomes a cost the team maintains every month.

The matter of testing restores for real, and writing that condition into the TOR at procurement time, we covered separately in how to write a TOR that yields a maintainable system. This article stays focused on the incident itself.

Incident communication must be agreed in advance

In a real incident, the communication gap does as much damage as the downed system itself. Users who hear nothing for two hours start calling in at once, the recovery team that is heads-down has to stop and answer, and executives who do not know the status start giving orders that cut across the people on the ground.

NIST SP 800-61 recommends defining communication guidelines in advance: who talks to whom, what level of information can be shared, and contact channels ready for both working and after hours, covering the CIO, the security team, and whoever owns business continuity (reference: NIST SP 800-61 Rev.2).

Two things work in practice. First, a single source of status truth that everyone refers to, an internal status page or one chat room, so there are never two versions of the state. Second, one person handling outbound communication, separate from the people fixing the system, so the technical team can keep working without interruption.

An update pattern that cuts inbound calls is a status note on a steady cadence, even when there is nothing new in that round. A short sentence that work is ongoing and when the next update will come is worth more to users than silence.

Another point that is easily forgotten is external communication with legal weight. If the incident involves personal data, the organization may have a duty to notify under PDPA, the details and timelines of which are set by the regulator, not the technical team. The plan should state in advance who decides this and when to consult legal or the DPO.

Handing over on-call while the incident is still open

Incidents that run past a few hours hit a problem most plans do not write about: the people recovering the system start to tire and have to hand off to the next shift while the incident is still open. If the handover is verbal, context is lost. The new shift spends time reconstructing what has been done, and risks repeating what the previous shift already tried without success.

Uptime Institute notes that staff fatigue is one of the factors that turns human error into a cause of outages (source: Uptime Institute, 2024). Letting someone who has worked many hours keep making big decisions is a risk that stays invisible until a mistake happens.

What helps is making every handover produce something tangible. At minimum it should have a short timeline of what happened and when, what has been tried and the result, the assumptions the team is currently holding, and the next step they intend to take. This document does not need to be pretty; it just needs the new shift to read it and take over immediately without asking everything again.

After the incident, run a blameless postmortem

Once the service is back, the work is not done, because the same incident will return if no one learns from it systematically. What works is a postmortem focused on what and which process led to the incident, rather than on who was wrong.

The blameless postmortem practice described by Google's SRE team rests on the principle that when you stop blaming individuals, people are willing to report what actually happened, and the organization gets enough information to fix the system (reference: Google SRE, Postmortem Culture). One document should have a timeline, the impact on users, the true root cause, and a list of things to fix, each with an owner and a due date.

The list of fixes is the heart of it. If it is not tracked to actual closure, the postmortem is just a document written to complete a ritual. Assigning an owner and a due date to each item, and revisiting them in the next round, is what turns a lesson into real change.

This loops back to the figure at the start. When most human-error incidents come from not following procedures or from flawed procedures, the fix that works is adjusting the runbook and the decision criteria so the next person can do it right more easily, consistent with the roughly four in five Uptime survey respondents who believe better management and processes would have prevented their most recent outage (source: Uptime Institute, 2023 survey).

Summary

What decides whether a Disaster Recovery Plan survives the actual day usually lies on the human side, more than in the recovery technology you chose. The questions to answer before that night are: who declares the incident, who orders the failover, and who communicates. If you take one thing from this article, write those three roles into the plan, with criteria you can measure as numbers.

And rehearse it at least once, before you have to use it for real.

If your organization already has a DR Plan but has never rehearsed it from the angle of people and decisions, and you would like someone to read it over first, we are happy to talk. No fixed deadline, and no rush to decide.

To discuss a project in detail, reach us at 088-983-9386 or email [email protected]. Our office is in Bang Kapi, Bangkok. For organizations still reviewing their incident response plan, we are glad to set up a call to understand the problem first, and if it turns out the plan you have is already enough, we will say so plainly.