RPO and RTO Explained: Restore Tests That Show What Your System Can Really Recover

The backup dashboard had shown a green tick every night for two years. Then the main database disk failed, and the executive in the room asked one question: when will the system be back? Nobody could answer, because nobody had restored that backup since the day the system was accepted.

The numbers everyone reached for that day were the RPO and RTO written into the TOR. RPO is how much recent data you can afford to lose, measured as time back from the moment of failure. RTO is how long the system can stay down before the impact becomes unacceptable. Both are targets set in advance. The only way to know what your system can actually achieve is to restore it and time the result.

This article is for the people who look after systems that are already live. We explain why a restore test done at acceptance goes stale, what each level of restore test proves, where restores tend to break even when every backup job reports success, and how to turn measured numbers into a decision management can make.

What RPO and RTO mean, and how they differ

NIST SP 800-34, the US guide to contingency planning for information systems, defines both terms. RTO is the maximum time a system resource can remain unavailable before there is an unacceptable impact on the business processes it supports. RPO is the point in time before the disruption to which data can be recovered, given the most recent backup copy (source: NIST SP 800-34 Rev.1, section 3.2.1).

Picture a system that backs up its database once a day at 2am. If the database fails at 4pm, everything written between 2am and 4pm is gone. Those fourteen hours are the RPO the system actually delivered that day. The time from the failure until users can work again is the RTO it actually delivered.

The two numbers answer different questions. RPO asks how much data you can lose. RTO asks how long you can wait. One system may tolerate half a day of downtime but no lost transactions at all. Another may need to be back within the hour but can live without its last hour of data. Setting both values is a conversation with the owner of the business process. The technical team cannot settle it alone.

NIST adds a third value worth knowing, MTD, the maximum tolerable downtime once every impact is taken into account. RTO must be shorter than MTD, because after the system comes back there is still backlog to process. RPO sits outside MTD. It is about how much data loss the process can tolerate during recovery.

How to use these numbers to decide on a failover during an outage is covered in our article The DR Plan That Survives the Day. This article looks at the other side: how you know the numbers you will be deciding with that night can be trusted.

The numbers in the TOR are targets, not measurements

The RPO a system actually achieves depends on three things: how often backups run, how long each backup job takes to finish, and any replication lag to a second location. If the TOR says four hours but backups run once a day, the document and the system have disagreed since day one.

The RTO a system actually achieves is a chain of steps added together. First the team has to notice the failure. Then someone decides to restore, the target server or database is prepared, the data is restored, and the result is checked before users are let back in. Most documents estimate only the restore step and leave the rest out of the sum.

Our article on writing a TOR for a system you can maintain (in Thai) recommends at least one real restore test before acceptance. That condition matters, but its result starts ageing the day the system goes live.

Data grows every month, and restore time grows with it.

Schemas change, systems move region, new services are added, and the person who ran the first test may have moved on. A test from two years ago tells you the system could be recovered back then. Nobody has yet shown that the system running today can be.

Three levels of restore test, each proving something different

The first level is restoring individual files or objects. It proves the copies are readable, uncorrupted, and retrievable by the people who hold the access rights. It is cheap and can run often, but it tells you very little about the RTO of the whole system.

The second level is restoring a whole database. Where the platform supports it, also try a point-in-time restore to a moment you choose. This level produces the two numbers executives ask about most: how long a restore of today's database takes, and what time the last recovered transaction is from.

The third level is bringing the entire application back up in an isolated environment and actually using it. This catches problems the first two levels cannot see, such as an application that cannot connect to the restored database because its configuration still points elsewhere, or a dependency that is not ready yet.

Google Cloud makes the same point in its Well-Architected Framework. It says restore tests should be judged on data integrity, RTO and RPO, should exercise the entire application stack and the critical infrastructure services on the restored data, and should run in a non-production environment (source: Google Cloud, Perform testing for recovery from data loss, updated 30 December 2024).

How often to test

NIST SP 800-34 scales testing to the impact level of the system. For low-impact systems a tabletop exercise is enough. Moderate-impact systems should run a functional exercise that includes recovering the system from backup media. High-impact systems should run a full-scale exercise, including recovering a server or database from backup and restoring the system to a known state. NIST leaves the frequency of each to the organization (source: NIST SP 800-34 Rev.1, section 3.5.1).

The line in the same document about testing the plan annually is policy for US federal agencies. A Thai organization can use it as a starting point. If your business is regulated, for example by the Bank of Thailand or Thailand's Securities and Exchange Commission, check the minimum frequency in that regulator's own notices. We could not find a notice that sets a clear frequency for restore testing, so we will not put a number in the regulator's mouth.

Beyond the calendar, our view is that you should test again whenever something changes how recovery works: a major schema change, moving the database to a new region or provider, switching backup tools, or a sudden jump in data volume after a new service launches. Any of these makes the previous result unreliable straight away, without waiting for the next scheduled round.

Alongside scheduled tests, watch the backup jobs every day. Google Cloud recommends automated consistency checks after each backup, and alerts when the interval between backups exceeds the defined RPO. How to set alerts so they only wake people for things worth waking up for is the subject of our article Monitoring and Alerts That Wake You Only for What Matters.

Where restores break even when the backup reports success

A successful backup job tells you the copy finished without an error. It does not tell you that what was copied is enough to put the system back together. The failures restore tests find tend to fall into a handful of patterns.

The data was backed up, but not what makes it usable. The database comes back complete, but configuration files, application secrets, or the encryption keys needed to read the data were never in the same backup. If the keys live in an account you can no longer reach, the restored data cannot be read. Who holds an organization's accounts and keys is covered in our article Who Actually Holds the Keys to Your Cloud Account?

The backup sits in the same place as what it protects. If the account that can delete production can also delete the backups, an attacker who takes that account can destroy both. CISA's #StopRansomware Guide notes that many ransomware variants try to find and then delete or encrypt accessible backups. It recommends offline, encrypted backups, and regularly testing their availability and integrity in a disaster recovery scenario (source: CISA #StopRansomware Guide, September 2023). On cloud platforms the nearest equivalent is an immutable backup. Microsoft Azure, for example, offers a mode that cannot be reversed once locked, so test its effect before you lock it.

There are numbers behind this risk. A Sophos report surveyed 2,974 IT and security professionals at organizations of 100 to 5,000 employees in 14 countries, all hit by ransomware in the previous year, with fieldwork in January and February 2024. Among those organizations, 94% said attackers tried to compromise their backups, and 57% of those attempts succeeded. The median overall recovery cost was about US$3 million where backups were compromised, against about US$375,000 where they were not (source: Sophos, The Impact of Compromised Backups on Ransomware Outcomes, March 2024). The survey was commissioned by a security vendor, and Thailand is not among the 14 countries. Treat it as a global picture, not a description of the Thai market.

Replication is counted as a backup. A continuously replicated standby database helps a great deal when the primary machine fails. But if someone drops the wrong table or the data is encrypted by an attacker, that damage is replicated straight away too. A system with replicas and no backups at multiple points in time cannot recover from human error or an attack.

Restore time grows with the data. Restoring a first-year database may finish in an acceptable time. Three years later the data is several times larger, and the restore takes longer with it. If nobody measures again, the team learns the new number on the night it matters.

The things the system depends on do not come back with it. The data is complete, but DNS still points to the old location, the certificate on the new machine has expired, or the application's service account has no permission on the newly created database. These only show up when you test at the third level.

Measure and record each round so rounds can be compared

A test without a record leaves only a feeling that recovery would probably work. A useful record captures the main points below in the same format every round, so you can see the trend across rounds.

First, the time for each step. Note when preparing the target, the restore itself, checking the data, and opening the system for a trial each started and ended. The sum is your measured RTO, and the breakdown shows where the time went.

Second, the data lost. Look at the timestamp of the last transaction in the restored data and compare it with the moment you treated as the failure. That gap is your measured RPO, and it should be set directly against the target.

Third, whether the restored data is correct: compare row counts on key tables, compare checksums, and have a regular user run two or three of the tasks they do most. Detailed methods for checking completeness and correctness are in our article Complete Is Not Correct, and the same approach works for checking a restore.

Finally, what got in the way. Every time someone had to open a document, ask a colleague, or change a step mid-test, write it down. That list is what needs fixing in the runbook before the next round, and it is often worth more than the timings.

Report the gap so management can decide

Test results only matter once they become a decision. A format that works is a one-page table with one row per system: target RPO and RTO, the values measured in the latest round, the test date, and the gap between the two.

When the measured values fall short of the target, put two options in front of management at the same time. One is to invest so the system meets the target, for example more frequent backups, a different backup method, or a target environment prepared in advance. The other is to accept that the original target was higher than the business needs, and lower it formally. Either is acceptable. What is not acceptable is letting the target on paper and the real capability drift further apart without anyone signing off on it.

NIST takes a similar line. When the RTO cannot be met right away and the MTD cannot flex, it calls for a plan of action and milestones to document the situation and how it will be fixed. Give each item in that plan an owner and a date, and bring it back at the next test.

The limits, stated plainly

Third-level testing has a real cost. You need an isolated environment large enough for the full data set, several hours of the team's time per round, and on cloud platforms the cost of resources left running during the test. If your system is an internal tool with a few dozen users, can stop for half a day without affecting anyone, and holds data that can mostly be recreated, a full-scale test may not be worth it. For a system like that, we suggest starting with a scheduled database restore and simply recording the time.

Also take care with the data itself. What you restore into a test environment is real production data. If it includes personal data, that environment needs the same access control and post-test deletion as production. Do not open it to people who would not normally see that data, and talk to your organization's DPO before you set the procedure.

Lastly, a restore test tells you how long recovery takes. It does not tell you who starts the recovery on the night, or who tells the users. That belongs in the DR Plan, alongside this work.

Where to start

Pick your single most important system. Restore its latest database into an isolated environment, time each step, check the timestamp of the last transaction that came back, and put those two numbers next to the targets in the TOR. A first test like this does not take long, and it gives you something most organizations do not have: the real RPO and RTO of the system running today.

If the result meets the target, you have evidence to show management. If it does not, you found out before the day you needed it.

Summary

A green backup status every night is evidence that copying works. It does not prove the system can be recovered. If you take one thing from this article, schedule restore tests of your main system's database, measure the RPO and RTO you actually achieve each round, and put them next to the targets where management can see them.

If your organization has systems that have been live for years and have not had a restore test since acceptance, and you would like help planning the first round, we are happy to talk. No fixed deadline, and no rush to decide.

To discuss a project in detail, reach us at 088-983-9386 or email [email protected]. Our office is in Bang Kapi, Bangkok. For organizations reviewing their backup and recovery plans, we are glad to set up a call to understand the problem first, and if it turns out what you have is already enough, we will say so plainly.