Every operating environment is full of checks: health checks, monitoring queries, tests after a deployment, controls after a firewall change. Their value does not lie in being green, but in turning red when something is wrong. It is precisely this second half that is rarely tested. A control that has never fired can mean that everything is fine – or that it cannot fire at all. From the outside, both cases look the same.
The patterns discussed here are widespread and independent of tooling and the size of the environment. The examples are hypothetical, but typical.
Four outcomes, not two
Most checks know two outcomes: passed or not passed. That is not enough. At least two more outcomes belong with them:
- Passed: the measurement took place, and the result matches the expectation.
- Failed: the measurement took place and found a defect.
- Error: the measurement did not take place – the connection was not established, the query was invalid, the response stayed empty.
- Skipped: the check was deliberately left out, with a reason that remains visible in the result.
The decisive difference lies between “error” and “passed”. “The system is broken” and “I could not find out” call for different responses, but both must stand out. It becomes dangerous when the error path and the “nothing found” path return the same value: then the check gives the all-clear precisely at the moment it is least entitled to.
“Skipped” must never pass as green either. A run in which every individual check was left out has measured nothing – and should report exactly that.
How checks lie without lying
The following patterns are unspectacular. That is exactly why they typically survive reviews.
The negation instead of the expectation. Suppose a health check tests whether an address responds “not with status 200”, because a redirect is expected there. But that condition is also met by 403, by 500 – and when no connection was made at all. Check for the expected status by name, and treat a missing response as an outcome of its own.
The error object as zero. Suppose a monitoring query counts hits, such as failed logins. If the query is syntactically invalid, the system returns an error object without a hit count. A script that reads the missing field as 0 then reports “no anomalies”. A missing value is not zero. Make sure a valid response has arrived before you evaluate it.
The text search that finds too much. Anyone searching logs for an error message also finds every place where people talk about that error message – in tickets, chat histories, comments. Structured fields such as source, event type or status code are more reliable than free text.
The probe that finds itself. A classic among process checks: a search such as ps aux | grep dienst often returns a hit – the grep command itself (pgrep, by contrast, excludes itself). The service being looked for may have stopped long ago.
The right question to the wrong recipient
A check can be technically flawless and still miss the point. A common pattern: a check that tests a web application directly at the server sees nothing of the DNS, CDN or certificate problems a visitor would run into. Such direct checks have their purpose, but they answer a different question. Between visitor and server lie name resolution, possibly a load balancer or CDN, and the certificate chain. A fault along this path stays invisible to the check at the server – it reports green while the visitor sees an error page.
Both measurements are legitimate; they are just not interchangeable. Google’s SRE book distinguishes here between white-box monitoring, which looks into the internals of a system, and black-box monitoring, which is symptom-oriented and represents active problems: “The system isn’t working correctly, right now.” A check from the visitor’s point of view – via the public name, with a validated certificate chain – therefore belongs in every release of a publicly reachable application.
“Everything still works” is not a firewall check
After a change to a firewall, the usual check is whether the permitted services are still reachable. That is necessary, but it only proves that nothing broke. Whether the firewall filters at all is not shown by this test – a firewall with no active rule at all passes it just as well.
That takes a negative probe: a connection attempt from a source that must be blocked, to a port that must be closed. Only when this attempt fails is the filtering proven. It is important that the probe comes from outside the system being checked; a test from the host itself can run over paths that the rules do not even affect.
A missing control does not report itself
The most uncomfortable variant concerns the protective mechanisms themselves: approval rules, alert rules, automatic blocks. An example: a company moves its mail server to a new platform. The rule that holds back attachments containing executable files is not carried over. Nobody notices. There is no outage and no warning, and nothing unusual appears in the log – a rule that does not exist cannot record anything either. To everyone involved, operations look like a quiet month.
That is why it is not enough to set up a control. Its effect is only proven once it has demonstrably fired, for example through a deliberate test case. That proof should be repeated whenever the configuration changes fundamentally – after a platform change, a major update or a migration. And it should not come from the checked system alone: what an environment reports about itself can only contain what it knows itself. An independent view – a test message sent from outside, a second tool, another person – uncovers what the self-report misses.
In the IT-Grundschutz module DER.1, the BSI provides for detection systems to be reviewed in regular audits as to “whether they are still up to date and effective”. The same document describes as a threat that faulty configurations can lead to detection systems “not functioning properly”. The SRE book draws a hard consequence from a related observation: data collection, aggregation and alerting configuration that is rarely exercised – for some SRE teams less than once a quarter – should be up for removal.
Checklist: every check has to fail once
- Prove that the check can fail. Run every new check once against a known-bad input and watch it turn red. A check nobody has seen fail is not proven to be a check.
- Give the error its own outcome – separate from passed and not passed, with its own status or exit code.
- Check for the expected, not its negation: a specific status, a specific value, a specific page title.
- Check whether a response arrived before you interpret it. Empty is not green.
- Ask the right recipient: measure origin and visitor view separately, document both.
- Make omissions visible. “Skipped because …” belongs in the result.
- Fix the check, not the threshold. If a check reports wrongly, its error handling needs repairing – not the condition loosened so that the unclear case lands on the green side.
A negative control should also be built so that it still returns the same result tomorrow. Anyone who tests a check against a defect currently present in production has a control that turns green as soon as the defect is fixed – and then proves nothing anymore. Stored test data with a known result are more reliable for this than the live state.
How sector7 supports you
We support the build-up or integration of log and SIEM platforms and define use cases, alert rules and playbooks with you – more under cyber security. Our NOC monitors managed environments around the clock in an automated way; what ongoing operations look like is described under Managed Services & Support.
Sources
- Site Reliability Engineering, Chapter 6: Monitoring Distributed Systems – Google, retrieved 30 September 2026
- IT-Grundschutz-Kompendium, Baustein DER.1 Detektion von sicherheitsrelevanten Ereignissen, Edition 2023 (PDF) – BSI, retrieved 30 September 2026