How to think about system monitoring and alerting in an organization: measure from the user's side, separate alerts worth waking someone for from those that can wait, tie alerts to an SLO, and avoid the traps Thai organizations hit, from the shutdown of LINE Notify to deciding who picks up the alert at 3am.
Monday morning, the customer service team posts in the group chat: people have been calling since seven to say they cannot log in. The IT team opens the dashboard. Every server is green, CPU looks normal, there is disk to spare, and not a single alert fired all night.
This does not mean the organization has no monitoring. Most have plenty. The problem is that it measures what is easy to measure, not what users experience. The other side of the same problem is the team that receives hundreds of alerts a day, stops reading them, and misses the one that mattered.
This article is for the people who decide what an organization's systems measure, when they alert, and who they alert. We explain how to separate the alerts worth waking someone at 3am from the ones that can wait until morning, and where organizations in Thailand tend to stumble when getting an alert into the hands of a real person.
It covers the period before an incident is declared. For the decisions that come after, see our article The DR Plan That Survives the Day: Who Decides, Who Communicates, When Systems Fail.
Measure from the user's side first, then the machine's
Most monitoring starts with what the tooling collects automatically: CPU, memory, disk space. Those numbers help when you are hunting for a cause, but they say very little about what users are going through. A server with low CPU may be returning an error on every request because it cannot reach its database. A server with high CPU may be doing exactly what it should.
Google's Site Reliability Engineering book frames this as two separate questions: what is broken (the symptom) and why (the cause). Its example is a service serving HTTP 500s, which is the symptom, while the database refusing connections is the cause. The book recommends that alerts which page a human be tied mainly to symptoms, with internal data kept for diagnosis (source: Google SRE Book, Monitoring Distributed Systems).
The same book suggests four user-facing signals to start with, which it calls the four golden signals.
Latency: how long the system takes to answer a request, with successful and failed requests measured separately, because fast errors pull the average down and make things look better than they are.
Traffic: how much demand is coming in, such as requests per second.
Errors: the rate of failed requests, both the ones the system reports and the ones you have to infer, such as a success response with the wrong content.
Saturation: how full the system is, measured on whichever resource is most constrained.
The layer organizations most often lack is measurement from outside, known as black-box monitoring: a tool that calls your website or API from outside your network at regular intervals, the way a user would. If outbound networking or DNS breaks, a tool sitting inside the same network will not see it at all. The cloud providers we work with offer this out of the box, for example uptime checks in Google Cloud Monitoring and Health Checks in Cloudflare.
Alerts that should wake someone, and alerts that can wait for morning
The most common mistake is sending everything to one channel at one level of urgency. A disk at 80% lands in the same group as the payment system going down. The team learns quickly that most messages in that group need no action, and they mute it. This is what people call alert fatigue.
The fix starts with splitting alerts into at least two levels. A page is something that needs a person to get up and look now. A ticket is work opened for the team to handle during business hours. Anything that fits neither belongs on a dashboard as a graph, not in a message sent to anyone.
Google's SRE book offers a set of questions for deciding whether a condition deserves a page. We read the core of it as this: the condition must be urgent, actionable, affecting users now or about to, and it must need human judgment. If the response can be scripted every time, automate it instead of waking someone (source: Google SRE Book).
A simple test we use: go back through last month's pages one by one and ask whether the person who was woken up actually did anything. If the answer was "looked, then closed it", that alert should be downgraded or its condition narrowed.
Tie alerts to an SLO instead of a CPU number
Once you are measuring from the user's side, the next question is how bad is bad enough to wake someone. The answer that holds up over time is to tie it to a service-level target. Three terms usually travel together.
SLI (Service Level Indicator): what you actually measure, such as the share of requests answered successfully within a set time.
SLO (Service Level Objective): the internal target the team sets for that indicator, such as 99.9% over 30 days.
SLA (Service Level Agreement): the commitment written into a contract with users or a client, usually set looser than the SLO to leave a margin.
We cover SLAs in maintenance contracts in a separate article, What to check before signing an annual maintenance contract (in Thai). Here we focus on using the SLO to set alerts.
If the SLO is 99.9%, the remaining 0.1% is the error budget, the amount of failure you accept in that period. Instead of alerting on every error, you watch how fast the error budget is being spent. That speed is called the burn rate. A burn rate of 1 means the budget runs out exactly at the end of the period.
The Google SRE Workbook gives starting values for a 99.9% SLO in a table: page when 2% of the budget is spent within 1 hour (burn rate 14.4), page when 5% is spent within 6 hours (burn rate 6), and open a ticket when 10% is spent within 3 days (burn rate 1). Each condition has to hold over both a long and a short window at the same time, so the alert stops firing soon after the problem ends (source: Google SRE Workbook, Alerting on SLOs).
These numbers are a starting point, not a finished answer, and the Workbook presents them as defaults to tune for your own system. The benefit of the approach is that small problems which barely touch users overall wake no one, while problems that hit many users in a short time are caught fast.
An alert has to reach a person, not just be sent
The least tested part of any monitoring setup is the last step, between the tool sending an alert and the right person actually reading it. For organizations in Thailand, two cases are worth checking.
Check LINE Notify first. It let a system post into a LINE group with a single token, and it has been used to send alerts in many systems. LINE ended the service on 31 March 2025 and recommends moving to the Messaging API through a LINE Official Account (source: LINE Notify, end-of-service announcement). If any script in your systems still calls LINE Notify, alerts sent that way may be disappearing quietly with nobody noticing. One thing to know about the Messaging API: messages count toward the LINE Official Account's monthly quota, and the free allowance depends on the plan.
The second case is SMS and voice calls from overseas cloud tooling. Azure Monitor action groups, for example, can send email, SMS, voice calls and push notifications. But when we checked in September 2026, Thailand was not on the list of countries supported for SMS or voice. Microsoft suggests using a webhook to a third-party SMS provider that does support the country (source: Microsoft Learn, Azure Monitor action groups). A team that entered phone numbers and considered the job done may never have received a single real SMS.
Both cases have the same fix: test the whole path on a schedule. Fire a test alert from the real tool and check that it reaches the on-call person's phone within the time you expect. Azure Monitor has a built-in test button for action groups; with other tools you can create a test condition you know will trigger. You should also have at least one backup channel, so that no single service carries every notification.
The alert message should tell the reader what to do next
Someone woken at 3am has far less focus than they do in the afternoon. If the message they receive is only a metric name and a number, such as error_rate_5m > 0.02, they spend the first several minutes guessing which service is broken, who is affected, and what to open first. All of that time counts toward the time users cannot use the system.
A good page answers these questions in the message itself: which service has which symptom, when it started, which users or what share of them are affected, with links to that service's dashboard and runbook. The runbook does not need to be long. It only needs to say what to check first, who owns the service, and when to escalate to an incident. If you cannot write a runbook for an alert because you do not know what the person receiving it should do, that is usually a sign it should never have been a page.
The threshold for escalating from an alert to an incident that opens the response plan should use the same numbers as your DR Plan. That way the person holding the pager knows whether they have reached the point of waking others, instead of deciding alone on gut feeling in the middle of the night.
Who picks up the alert at 3am
Every well-designed alert is useless if nobody answers it at 3am. This question needs names and a schedule as its answer, not "the IT team".
Google's SRE book gives several reference numbers here. Typical response times are 5 minutes for user-facing or highly time-critical services and 30 minutes for systems that can wait longer. A sustainable load is at most 2 incidents per 12-hour shift. And a team at a single site needs at least 8 people to run 24-hour on-call with both a primary and a secondary (source: Google SRE Book, Being On-Call).
To be clear, those numbers come from an organization the size of Google. If you do not have eight people for a single system's rota, do not force it. If your system is internal and only used during business hours, a business-hours rota plus a handful of pages for the problems that truly cannot wait until morning may be enough. What you should not cut is naming who holds the pager in each period, and who backs them up when the first person does not respond.
If an outside provider maintains the system under contract, check whether the response time in the contract applies outside business hours, and whether alerts go straight to the provider or wait for one of your staff to see them and phone it in. That gap can take longer than fixing the actual problem.
A common objection: more alerts is safer
The reasoning is understandable. Nobody wants to be the person who did not set up the alert that would have caught the next incident, and teams that have missed a big one tend to respond by adding conditions.
The catch is that alerts are not free. The Google SRE Workbook judges alerting on two measures: precision, the share of alerts that were significant events, and recall, the share of significant events that were caught (source: Google SRE Workbook). Adding alerts indiscriminately does raise recall, but precision falls, and once precision is low enough people stop trusting alerts altogether, including the correct ones.
The better route is to raise recall by measuring the right thing, not more things. One alert tied to a user-facing symptom catches many different causes at once, without a separate alert for every cause you can think of.
If you have nothing today, where to start
For a system with no real monitoring yet, or monitoring nobody trusts, this is the order we recommend. It covers the main points, not every case.
Pick the one to three services users rely on most, and set up external uptime checks for them first.
Measure error rate and latency for those services from the request side, not the machine side.
Set an SLO the team and the service owner can agree on, even if it is a rough number the first time.
Sort existing alerts into two levels, page and ticket, and delete the ones nobody has ever acted on.
Name who holds the pager and who backs them up, then test that alerts actually reach those people's phones.
Review the list of pages every month, and fix or downgrade any that woke someone for nothing.
If your system is still in procurement, monitoring requirements belong in the scope of work from the start, including whose name the monitoring tool's account is held under. We set out that thinking in How to write a TOR for a system you can maintain (in Thai).
Summary
Good monitoring is not measured by the number of graphs or alerts, but by one question: when users start running into problems, does your team know first? If you take one thing from this article, set pages on what users experience, then test that they actually reach the on-call person's phone. Together they close the biggest gap first.
Neither requires buying a new tool.
If your organization already has monitoring but the team has stopped trusting its alerts, or you are not sure where to start measuring, and you would like someone to review it with you, we are happy to talk. No fixed deadline, and no rush to decide.
To discuss a project in detail, reach us at 088-983-9386 or email [email protected]. Our office is in Bang Kapi, Bangkok. For organizations reviewing their alerting, we are glad to set up a call to understand the problem first, and if it turns out what you have is already enough, we will say so plainly.
FAQ: Frequently Asked Questions about This Article
A collection of questions and answers to help you better understand the content of this article.
Tags: