The page
A week after we turned on deeper tracing for a Java service, its response times crept up. No code changes. The team asked, fairly, whether the new monitoring was the cause.
Triage
# which agents is this JVM running with?
ps -o pid,args -p <pid> | tr ' ' '\n' | grep -i javaagent
# JVM flags in effect
jcmd <pid> VM.flags
# GC pressure, sampled every second, ten times
jstat -gcutil <pid> 1000 10
# where threads are spending time
jstack <pid> > threads.txt
grep -A3 "RUNNABLE" threads.txt | head -50
GC was busier than before, and thread dumps showed time inside the agent's instrumentation on a very hot method.
Root cause
The new tracing config instrumented every call to a small utility method that ran thousands of times per request. Each call was cheap. The tracing around it wasn't, at that volume.
The fix
Excluded that method from instrumentation and kept tracing at the service boundaries and the slow paths. To prove it, we ran one instance with the agent config change and compared it against the others for a day before rolling it out.
What I learnt
Monitoring has a cost, and it isn't free just because it's monitoring. Deep instrumentation belongs where the questions are, not everywhere.
What I do now
- Measure agent overhead with a canary, one instance with the change and one without.
- Instrument boundaries and known-slow paths first. Add depth when a question needs it.
- Treat monitoring config like code: reviewed, versioned, rolled out gradually.
stage 1 checks whether your monitoring pays for itself → see the engagements