On 29 March 2024 a PostgreSQL developer, Andres Freund, posted to the oss-security mailing list that the compression library behind xz had been backdoored. Two releases, 5.6.0 and 5.6.1, carried hidden code that hooked into OpenSSH's server on some Linux distributions and would let one specific attacker run commands on the machine before logging in. It got CVE-2024-3094 and a CVSS score of 10.
The code was clever, but the code wasn't the attack. The attack was two and a half years of patient, friendly work to become a trusted maintainer of a small project run by one tired volunteer. Here is how it went, step by step, why it almost worked, and what breaks each step.
Why xz was worth two years
XZ Utils and its library, liblzma, are on almost every Linux system. OpenSSH doesn't use liblzma itself, but Debian, Fedora and others patch sshd to tell systemd when it's ready, and libsystemd links liblzma. So code in liblzma ends up inside the SSH server, the most exposed process on most servers. The project had essentially one maintainer, Lasse Collin, working unpaid.
Step 1: the helpful newcomer (2021-2022)
A contributor using the name Jia Tan sent a first small patch on 29 October 2021 and had a first commit merged in February 2022. Nothing about the early work was malicious. That was the point: months of small, useful fixes build a record.
Step 2: pressure from people who didn't exist (2022)
From April 2022, accounts named "Jigar Kumar" and "Dennis Ens" started complaining on the mailing list that patches sat for years and that the project needed a new maintainer. Neither name has any other footprint online. In June 2022 Lasse Collin replied that his ability to care had been limited by long-term mental health issues, and that Jia Tan might take a bigger role. On 29 June he called Jia Tan "practically a co-maintainer already".
The pressure campaign didn't need a vulnerability. It needed a volunteer who was tired and alone.
Step 3: maintainer, then release manager (2022-2023)
Jia Tan joined the project's GitHub organisation in October 2022, was listed as a maintainer in November and cut their first release, 5.4.2, in March 2023. Along the way came changes that looked routine and later turned out to matter:
• June 2023: support for GNU indirect functions (ifunc) came in, contributed by another newcomer, "Hans Jansen". The backdoor later used an ifunc resolver to run early in the loading process.
• July 2023: ifunc was turned off in the project's OSS-Fuzz builds, "incompatible" with the sanitiser, which also kept fuzzers from seeing that code path.
• January 2024: the project website moved to GitHub Pages under Jia Tan's control.
• February 2024: a single stray . in a build check quietly disabled detection of the Landlock sandbox.
Step 4: a payload that wasn't in the source code
On 23 February 2024 two "test files" were committed, binary .xz archives that test suites legitimately need. Inside them was the encrypted, obfuscated backdoor. On 24 February release 5.6.0 shipped. The release tarball contained a modified build-to-host.m4 build script that was not in the git repository. During a build, that script pulled the payload out of the test files and linked it into liblzma, but only when building on x86-64 Linux with glibc and GCC, inside a Debian or RPM package build. Anyone reading the git history saw nothing.
Once loaded into sshd, the code replaced the function OpenSSH uses to check RSA signatures. A connection carrying data signed by the attacker's own Ed448 key would have its embedded command run as root, before authentication. Without that private key the backdoor did nothing visible, so nobody else could use it, and scanning for it from outside wasn't possible.
Step 5: into the distributions
Jia Tan and "Hans Jansen" then asked distributions to update: Fedora was approached in late February, a Debian bug requesting 5.6.1 was filed on 25 March, and an Ubuntu request followed on 28 March. Version 5.6.1 (9 March) fixed "valgrind errors" the first payload caused. It reached Fedora Rawhide and the Fedora 40 beta, Fedora 41, Debian testing and unstable, openSUSE Tumbleweed and Kali. Stable releases such as Debian 12, RHEL and Ubuntu 22.04 LTS never shipped it. Ubuntu 24.04 was weeks from release.
How it was caught: half a second
Freund was benchmarking PostgreSQL on Debian unstable and noticed SSH logins using a lot of CPU, around 0.8 seconds instead of 0.3, plus valgrind errors in sshd. He profiled it, followed the time into liblzma, and found the hook. He told the distributions privately on 28 March and posted publicly the next day. CISA told users to downgrade to 5.4.6. GitHub suspended the repository while the project was cleaned, and Lasse Collin released 5.6.2 on 29 May 2024.
The backdoor also checked for signs of analysis (debugging environment variables, not running under systemd) and stayed quiet when it saw them. A slow login on one engineer's test machine was the thread that unravelled it.
Check your own servers
All mainstream distributions removed the bad versions in 2024, so this is a two-minute hygiene check, not an emergency:
Controls that break each step
If you run servers
• Run stable releases in production. Every affected system was on a testing, development or rolling branch.
• Don't leave SSH open to the whole internet if you can help it: allow it only from your office or VPN addresses. This backdoor ran before authentication, so key-only SSH alone wouldn't have stopped it, but a firewall allowlist would have. Our guides on SSH hardening and a default-deny firewall with ufw cover both.
• Patch on a schedule and watch distribution security notices; unattended security upgrades install fixes like this one without waiting for you to log in.
• Notice odd performance. A process suddenly slower or busier after an update is worth ten minutes of curiosity.
If you package or build software
• Build from the tagged source, not only the release tarball, or diff the two. The payload's trigger lived only in the tarball.
• Treat binary test fixtures as code: who added them, why, and can they be regenerated from a script?
• Push for reproducible builds, so a second party can rebuild the same bytes.
• Be wary of build-time changes that switch off fuzzing, sanitisers or sandboxes, however good the reason sounds.
If you maintain a project (or depend on one)
• Pressure to hand over access, from accounts with no history, is itself a signal. Slow down and ask other maintainers.
• Require a second reviewer for release scripts and build files, and sign releases.
• Know which small projects your product depends on, and support their maintainers with money or time. The cheapest security control here was a less exhausted Lasse Collin.
• The same trust problem shows up in JavaScript and CI: see Polyfill.io and our npm and GitHub Actions hardening notes.
The lesson
Supply chain attacks are usually told as code stories. XZ was a people story: a lone volunteer, invented critics, and a helpful stranger who waited years for the right release. It was stopped by luck and one curious engineer. The durable fixes are dull ones: stable releases on servers, SSH that only your own addresses can reach, builds that match their source, and maintainers who aren't left alone.
osec.one builds small security tools for sites that don't have a security team: free SSL and email security checks, a header and TLS grade with fixes, uptime and certificate alerts.
See the free checksSources: Andres Freund on oss-security: backdoor in upstream xz/liblzma (29 March 2024); Russ Cox: Timeline of the xz open source attack; Wikipedia: XZ Utils backdoor; CISA: Reported supply chain compromise affecting XZ Utils (CVE-2024-3094); Red Hat: Urgent security alert for Fedora 41 and Fedora Rawhide users; Lasse Collin: XZ Utils backdoor notes.