The page
A customer-facing login was slow. The front end was in one tool, the Java services in another, infrastructure in a third, and an older platform in a fourth. Each team looked at its own screen and said its part was green.
Triage
To get one answer, someone had to query each tool separately and line the times up by hand.
# New Relic (NRQL): p95 response time per app for the last hour
# SELECT percentile(duration, 95) FROM Transaction FACET appName SINCE 1 hour ago
# Dynatrace API v2: open problems in the last two hours
curl -s -H "Authorization: Api-Token $DT_TOKEN" \
"https://$DT_ENV.live.dynatrace.com/api/v2/problems?from=now-2h" | jq '.problems[].title'
# Grafana: find the dashboards that mention the service
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
"https://grafana.example.com/api/search?query=login" | jq '.[].title'
By the time the times were lined up, the slowness had passed.
Root cause
The tooling wasn't the problem. Each tool was good at its layer. Nobody had a view across the layers, so every incident started with half an hour of assembling the picture.
The fix
I built a single pane of glass: one view that pulled the key signals from all four tools for each customer journey. Alongside that I rolled out the missing coverage (agent-based, agentless, synthetic and scripted monitoring) so every step of the journey had something watching it, and added machine-learning analysis to monitoring and diagnostics.
What I learnt
Coverage per team doesn't add up to coverage of the journey. The customer experiences the whole path, so at least one view has to show the whole path.
What I do now
- Start dashboards from the customer journey, then add the layers under it.
- Keep a shared time axis across tools. Lining up timestamps by hand is where incident time goes.
- Ask "what's not watched?" before "what tool should we buy?"
if your monitoring is spread across tools like this → page me