The page
We added anomaly detection to the monitoring so it would flag unusual behaviour without fixed thresholds. Within a week it had a pattern of its own: every Monday morning, a burst of anomalies across services.
Triage
import pandas as pd
df = pd.read_csv("requests_per_minute.csv", parse_dates=["ts"])
df["hour_of_week"] = df["ts"].dt.dayofweek * 24 + df["ts"].dt.hour
# what the model compared against: one flat baseline over the last 24 hours
flat = df.set_index("ts")["rpm"].rolling("24h").mean()
# what normal actually looks like: the same hour on previous weeks
weekly = df.groupby("hour_of_week")["rpm"].agg(["median", "std"])
print(weekly.loc[24:33]) # Monday 00:00 to 09:00
Monday 9am was always the busiest hour of the week. Compared to Sunday night, it looked like an attack.
Root cause
The baseline used a rolling 24-hour window. It knew what yesterday looked like, and yesterday was Sunday. It had no idea of weekly seasonality.
The fix
Compare each point to the same hour of the week over the previous several weeks, and flag only what falls well outside that range. Holidays went on a short list so they wouldn't train the baseline.
What I learnt
A model is only as good as its idea of normal. Traffic has daily and weekly shapes, and anything that ignores them will call the shape an anomaly.
What I do now
- Check any anomaly baseline against at least a few weeks of history before trusting it.
- Use hour-of-week (or seasonal) baselines for traffic-driven metrics.
- Track false positives as a metric of the detector itself.
stage 1 tunes alerting to how your traffic really behaves → see the engagements