The page
During a review, someone spotted a database password in a Jenkins console log. It had been there for weeks, readable by anyone with access to the job.
Triage
First, how far it had spread:
# search build logs on the Jenkins controller for likely secrets
grep -rlE "(password=|AKIA[0-9A-Z]{16}|BEGIN RSA PRIVATE KEY)" \
$JENKINS_HOME/jobs/*/builds/*/log
# which jobs, and which builds
grep -rlE "password=" $JENKINS_HOME/jobs/*/builds/*/log | cut -d/ -f5,7 | sort -u
Then why it was printed. The shell step had set -x for debugging, which echoes
every command, including the one with the password in it.
Root cause
The secret was a plain environment variable set in the job config, and a debug flag printed it. Jenkins only masks credentials it knows are credentials.
The fix
Rotated the password straight away, since anything in those logs had to be treated as leaked. Deleted the affected build logs. Then moved the secret into HashiCorp Vault and had the pipeline fetch it at run time through the credentials binding, so Jenkins masks it:
withCredentials([string(credentialsId: 'db-password', variable: 'DB_PASSWORD')]) {
sh 'set +x; ./migrate.sh'
}
vault kv get -field=password secret/app/db
vault token lookup # check the pipeline's token scope and TTL
What I learnt
Masking is a last line of defence. The real fix was that the pipeline shouldn't hold long-lived secrets at all. Short-lived, fetched when needed, scoped to the job.
What I do now
- Treat any secret in a log as leaked. Rotate first, investigate second.
- Keep secrets in a vault and fetch them at run time with a short TTL.
- Scan build logs and repos for secret patterns on a schedule.
if secrets live in your CI config → page me