Skip to content
rajesh.dhar
2021 · lead sre · java, artifactory, sonarqube

Log4Shell: finding every Java app we ran

The patch was the easy part. The hard part of Log4Shell was finding every place log4j-core existed, including inside jars inside jars.

1 min readby Rajesh Dhar

The page

On 9 December 2021 Log4Shell was published: a remote code execution bug in log4j-core 2.x, triggered by logging a string an attacker controls. Versions 2.0-beta9 through 2.14.1 were affected. Almost everything running Java had to be checked, at once, over a weekend.

Triage

Finding it is the work. It hides in fat jars, WAR files and vendor products.

# jars on disk
find / -name "log4j-core-*.jar" 2>/dev/null

# inside an application jar
unzip -l app.jar | grep -i log4j

# in a Maven or Gradle build
mvn dependency:tree -Dincludes=org.apache.logging.log4j
./gradlew dependencies | grep log4j

# in the artifact repository (JFrog CLI)
jf rt search "*/log4j-core-2.1*.jar"

Root cause

Log4j's message lookup feature would resolve ${jndi:ldap://...} strings in logged data, which let an attacker make the server load and run remote code.

The fix

Upgrade log4j-core (2.15 first, then 2.16 and 2.17 as follow-up issues came out), rebuild and redeploy. Where a rebuild wasn't possible straight away, the published stopgap was removing the vulnerable class from the jar:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Then blocking the pattern at the edge while the rebuilds rolled out.

What I learnt

You can only patch what you can find. The teams with a dependency inventory (an SBOM, or just a searchable artifact repo) were done in days. Everyone else spent the first days just finding things.

What I do now

  • Keep a searchable inventory of what's in every build.
  • Route all artifacts through one repository so there's one place to search.
  • Rehearse the "is X anywhere in our estate?" question before the next one.

Advisory: Apache Log4j security vulnerabilities


stage 1 checks how fast you could answer that question → see the engagements

Related

← all posts