The Anatomy of a Supply Chain Attack
A technical reconstruction of the XZ Utils backdoor. How a multi-year social engineering campaign nearly compromised SSH on every major Linux distribution.
On March 29, 2024, Andres Freund posted a message to the oss-security mailing list that stopped the open source world cold. He’d found a backdoor in XZ Utils — specifically in liblzma, a compression library that gets loaded by sshd on most Linux distributions through a systemd dependency chain.
The backdoor wasn’t a sloppy mistake. It was a multi-year, deliberate infiltration of an open source project by a pseudonymous contributor. The technical sophistication and patience involved make this one of the most significant supply chain attacks ever documented.
The Dependency Chain
Understanding why this mattered requires tracing the dependency chain:
sshd
└── libsystemd (via systemd-notify support)
└── liblzma (XZ Utils)
└── BACKDOOREDOn distributions that patch OpenSSH to integrate with systemd (Debian, Fedora, Ubuntu, and others), sshd links against libsystemd, which in turn links against liblzma. The backdoor in liblzma could intercept SSH authentication by hooking the RSA_public_decrypt function at runtime.
This is not a theoretical attack path. It’s the actual mechanism.
Timeline of Infiltration
The social engineering started in 2021. Here’s the reconstructed timeline:
| Date | Event |
|---|---|
| 2021-10 | ”Jia Tan” begins submitting patches to XZ Utils |
| 2022-01 | Pressure campaign begins against maintainer Lasse Collin |
| 2022-03 | Sockpuppet accounts complain about slow maintenance |
| 2022-05 | Jia Tan gains commit access |
| 2023-01 | Jia Tan becomes co-maintainer |
| 2024-02 | Backdoor code lands in test fixtures (tests/files/) |
| 2024-03 | Versions 5.6.0 and 5.6.1 released with backdoor |
| 2024-03-29 | Andres Freund discovers the backdoor |
The gap between first commit and backdoor deployment: over two years. This wasn’t a smash-and-grab. It was an intelligence operation.
Technical Mechanism
The backdoor was hidden in plain sight using a clever multi-stage process:
Stage 1: Obfuscated test fixtures. The malicious payload was embedded in binary test files (bad-3-corrupt_lzma2.xz and good-large_compressed.lzma) committed to the test directory. These look like normal test data for a compression library.
Stage 2: Build system manipulation. The build-to-host.m4 autoconf macro was modified to extract and execute the payload during the build process:
# Simplified version of the extraction logic
gl_am_configmake=`grep -aErls \
"#{4}[[:alnum:]]{5}#{4}$" \
tests/files/ 2>/dev/null`
# This extracts the hidden payload from the test files
eval "$gl_am_configmake" | tr "\t \-_" " \t_\-" | \
xz -d 2>/dev/null | /bin/bashStage 3: Function hooking. The extracted code patches liblzma’s .so file during the build process, adding an IFUNC resolver that intercepts calls to RSA_public_decrypt. This allows the attacker to authenticate to SSH with a specific key, bypassing password and public key authentication entirely.
How It Was Caught
Andres Freund wasn’t doing a security audit. He was benchmarking database performance and noticed that sshd was using significantly more CPU than expected during authentication. The difference: about 500ms per connection attempt.
He investigated with perf:
$ perf record -g -p $(pidof sshd) -- sleep 30
$ perf report --no-children
# Overhead Symbol
# ........ ......
39.21% [.] lzma_crc64
8.42% [.] lzma_crc32Seeing compression functions in the SSH authentication hot path was wrong. CRC functions shouldn’t be anywhere near RSA_public_decrypt. He dug into the library and found the hook.
The backdoor was caught not by a security scanner, not by a code review, not by an audit — but by a developer who noticed a performance regression and had the curiosity to trace it.
Lessons for Defenders
1. Supply chain verification is harder than you think. Code review would not have caught this. The malicious payload was in binary test fixtures, and the extraction happened through autoconf macros that few people read. The build artifact was different from the source code.
2. Reproducible builds matter. If distributions had routinely compared build outputs against builds from verified source, the mismatch between the tarball and the git repository would have been caught earlier.
3. Monitor for behavioral anomalies. Freund caught this through performance monitoring, not security tooling. Behavioral baselines — CPU usage, memory allocation patterns, system call profiles — can reveal compromises that static analysis misses.
4. Single-maintainer projects are critical infrastructure. XZ Utils was maintained by one person. The attacker exploited maintainer burnout as an attack vector. This is a systemic problem in open source.
# Check your own exposure
ldd $(which sshd) | grep lzma
# If this returns a match, your sshd links against liblzma
xz --version
# Versions 5.6.0 and 5.6.1 contained the backdoorThe XZ Utils backdoor is a case study in patience, operational security, and the fragility of trust in open source software. The attacker invested years of legitimate contributions to earn commit access to a critical library. The defense that worked was not a tool — it was human attention to detail.