Start with an operational question
“Why is the error rate rising in only one region?” is a better test than “Is the dashboard finished?”. The first requires context: service, region, revision, time window and a path from the metric to a relevant log. The second can be answered with a visually plausible graph.
For comparison, I would choose a few known investigation paths and follow them in both systems. The purpose is not to preserve every old panel unchanged. The useful question must remain answerable.
Meaning before visual similarity
Timestamps and time zones, field types, service names, labels and parsing determine what a query actually selects. A number arriving as text or a renamed environment field can distort comparisons even while data visibly flows in.
Aggregation and missing data require deliberate choices too. Average, percentile and sum are not interchangeable presentations. A quiet graph may mean low traffic, a different filter or interrupted collection.
Alerts have behaviour, not just thresholds
I would compare evaluation, waiting period, grouping, recovery and notification routing separately. The same threshold number does not guarantee equivalent alert behaviour. During parallel operation, production notifications need deliberate control so a comparison does not trigger duplicate escalations.
If new rules are being compared in the background, they need to be evaluated. Depending on the product, a paused rule is not a substitute. Test or non-paging notification routing may be more suitable; verify the specific tool’s behaviour.
Handover is a functional test
Can someone who did not implement the migration investigate a relevant failure? Do they know the new entry points, terminology, retention limits and remaining differences? Those questions turn an installed tool into something usable in operations.
Work it out: 5.5% or 1.9% errors?
Reproducible learning example using synthetic data. Not an account of an employer or client project.
Two views can use the same requests and show different numbers. In one shared five-minute window, this example has 100 completed requests in region A, including 10 HTTP 5xx responses, and 900 in region B, including 9 HTTP 5xx responses.
Run locally
Python 3 is sufficient. The script creates the 1,000 synthetic records in memory and needs no extra libraries, files, or connection to an observability product. Download it, then run:
Download the Python example · error-rate-aggregation.py
python error-rate-aggregation.pyExample output
Region A: 10/100 = 10.0%
Region B: 9/900 = 1.0%
Mean of regional percentages: 5.5%
All completed requests: 1.9%
No records: unknown5.5% is the unweighted mean of the two regional percentages: (10% + 1%) / 2. The error rate across all recorded requests is instead (10 + 9) / (100 + 900) = 1.9%. Both figures have a mathematical explanation, but answer different questions. For “What fraction of all requests fails?”, the relevant denominator here is 1,000 requests.
Check the denominator too
When comparing systems, first align time window, service and environment filters, and definitions of “request” and “error”. Are retries counted separately? Are timeouts without a completed log record missing? Is there sampling or duplication? Then inspect aggregation. With no records, the rate remains unknown; zero errors among recorded requests is a different claim. And the 1.9% overall rate must not obscure the regional problem at 10%.
Here, one record represents exactly one completed request, and only HTTP 5xx counts as an error. That is a defined exercise assumption, not a universal SLI definition. The comparison validates neither a particular product nor the outcome of a real migration.