Tired of alerts you learn to ignore? Good network alerting pages you only for conditions that need action: measure normal first, set warning and critical levels, and require a breach to last before firing.
| Item | Value |
|---|---|
| Baseline period before setting thresholds | At least 4-6 weeks |
| Statistic to compare against | 95th percentile (P95), not averages |
| Starting CPU thresholds, new server | 70% warning / 90% critical |
| Starting disk space thresholds, new server | 75% warning / 90% critical |
| Example rearm value for an 80% warning | 65-70% |
These figures come from one vendor’s guidance, ManageEngine’s alerts and thresholds guide. They can change, so check the source before you rely on them, and treat them as starting points rather than rules.
Network alerting basics: what makes an alert useful
An alert is useful when someone can act on it. If the usual response is to glance at it and close it, the alert is noise, and noise teaches people to ignore the next one, including the real one.
Before creating any rule, ask what a person would do when it fires. The same guide advises you to alert only on conditions that require action. Everything else belongs on a dashboard or in a report.
This is the alerting half of network monitoring: collecting data is cheap, but interrupting a human is expensive. Spend those interruptions carefully.
- Page: a service is down or users are affected now.
- Ticket: something will become a problem soon, such as a filling disk, but can wait for working hours.
- Log or dashboard only: diagnostic detail nobody needs to be woken for.
Measure normal before you set a threshold
Thresholds picked from a guess are usually wrong in both directions. The guide says to monitor performance for at least 4-6 weeks before setting thresholds, so the data includes busy and quiet periods.
It also recommends using P95 utilization: “Look at the 95th percentile, rather than averages, to account for normal spikes.” An average hides the peaks that matter, while the highest single sample overreacts to one-off blips.
A network baseline gives you this picture of normal. Set the warning level a sensible distance above typical peak behaviour, then adjust once you see how it behaves in practice.
What to do when you have no history
New devices have no baseline yet. For newly onboarded servers, the guide suggests conservative starting thresholds while the baseline builds:
- CPU: 70% warning / 90% critical.
- Disk space: 75% warning / 90% critical.
Once the 4-6 weeks of data are collected, revise them based on the device’s actual workload. These examples are for servers, so network gear such as switch uplinks or routers needs its own numbers drawn from its own history.
Warning, critical and rearm levels
Use two tiers. A warning tells you a trend is heading the wrong way and goes to a ticket or chat channel. A critical alert means act now and is the only tier that should page someone.
Add a rearm value as well. The guide’s example: if the CPU warning threshold is 80%, set the rearm value to 65-70%. The alert then only resets once the metric drops clearly below the line, so a value hovering at the threshold does not fire again and again.
Filter spikes with consecutive polls
The guide notes that a single spike above a threshold does not always mean a real problem. The fix is a consecutive poll count: the metric must exceed its threshold for a set number of polls in a row before an alert triggers.
Choose the count with care. Too low and you are paged by brief bursts; too high and real outages are reported late. Reserve instant, single-breach alerts for things that are plainly binary, such as a device that stops answering.
- Pick the metric and its baseline P95.
- Set warning and critical levels above that baseline.
- Set a rearm value below the warning level.
- Require consecutive breaches before firing.
- Route each tier to the right channel.
Choosing what to alert on in a network
Start from what users feel, then work backwards. A site link that is down, a gateway that stops answering or a service that cannot be reached deserves attention before a moderately busy interface does.
Typical network candidates, each of which needs thresholds drawn from its own history, include:
- Device reachability: the device stops responding to polls.
- Interface errors and discards: a rising rate on a link that was clean before.
- Link utilization: sustained load near the capacity of an uplink.
- Latency and packet loss: degradation on paths users depend on.
- Device health: CPU, memory and temperature on switches and routers.
Not every item needs a page. Reachability on a core router probably does; a warning about a quiet access port probably belongs in a daily report.
Common mistakes that create noise
Most alert fatigue comes from a few repeating patterns. Check your rules against this list when the inbox fills up.
- Copying default thresholds from a template and never revisiting them.
- Alerting on averages, which miss real peaks, or on single samples, which react to every blip.
- Sending warning and critical alerts to the same channel, so nobody can tell them apart.
- Forgetting maintenance windows, so planned work floods the queue.
- Alerting on a downstream symptom of every device when the one upstream device that failed explains them all.
The last point is a dependency problem. If a core switch fails, the devices behind it all look down. Where your tool supports dependencies or parent-child relationships, use them so one root cause produces one alert instead of a storm.
Plan for planned work and keep tuning
The guide also lists maintenance windows for planned downtime, and escalation policies that trigger after a time window if nobody responds. Both stop predictable events from becoming false alarms and make sure real ones are not left unanswered.
When an alert does fire, the data attached to it speeds up network troubleshooting. Include the device, the metric, the value and the threshold in the message so the responder starts with context.
A simple review habit works well. Once a month, list the alerts that fired most often and ask of each one: did anyone act, was the threshold sensible, and could it be a ticket instead of a page?
Write down the reason for each rule and who owns it. A rule without an owner is the first one people mute, and an unexplained rule is hard to retune months later.
Review rules regularly. Networks grow and traffic shifts, so each alert that was ignored or fired without action is a prompt to retune or delete it.
Frequently asked questions
How long should I collect data before setting thresholds?
Show answer
The ManageEngine guide recommends at least 4-6 weeks, covering peak and off-peak periods. Until then, use conservative starting values.
Should I alert on averages or percentiles?
Show answer
The same guide advises P95 rather than averages, because it accounts for normal spikes without reacting to every single sample.
What is a rearm value?
Show answer
It is the level a metric must fall back to before an alert resets. For example, with an 80% warning threshold, the guide suggests a rearm value of 65-70%.
Why did my alert fire for a few seconds of high usage?
Show answer
Probably the rule triggers on a single poll. Requiring consecutive polls over the threshold filters short-lived spikes.
