SolarWinds 2020: the build server that added a backdoor to a signed update

A code review would not have found it. The SolarWinds Orion source code in the repository was clean; the backdoor existed only for the minutes the build ran. A small program on the build server waited for the compiler to start, swapped one source file for a copy with extra code, and put the original back when the build finished. The result was signed with SolarWinds' own certificate and shipped as a normal Orion update from March 2020.

SolarWinds said about 18,000 customers installed one of those updates. The attackers, later attributed by the US government to Russia's foreign intelligence service (the group also called APT29), went further into a much smaller set: the White House said in February 2021 that about nine federal agencies and around 100 companies were actually breached. This piece is about the build step, which is where the attack lived, and the detection, which came from an unexpected place. For a vendor hit through another vendor, see 3CX 2023; for a backdoor added by a trusted maintainer, see XZ Utils.

Step 1: in early, and a dry run

SolarWinds' own investigation puts the first suspicious activity in its network in September 2019. The October 2019 Orion release appears to have contained harmless test changes: the attackers checking that code they added during the build would reach a shipped, signed product without breaking anything or being noticed. It did.

Step 2: SUNSPOT, a program that watched for the compiler

CrowdStrike, which SolarWinds brought in, recovered the build-server implant and called it SUNSPOT. Its build timestamp is 20 February 2020, the date SolarWinds gives for when SUNBURST began going into Orion builds. What it did:

1. Ran at boot from a scheduled task, and made sure only one copy ran.
2. Watched the running processes for MsBuild.exe, Microsoft's build engine.
3. Read that process's command line from memory to see whether it was building the Orion solution.
4. If so, checked the MD5 hash of the real InventoryManager.cs against the version it expected. If someone had changed that file, it did nothing, so a mismatched swap could never break the build and draw attention.
5. Saved the original as a backup, moved the backdoored version into place, and wrapped the added code in #pragma warning disable so the compiler printed no new warnings.
6. When MsBuild.exe exited, put the original file back.
7. Wrote its own log, encrypted, to C:\Windows\Temp\vmware-vmdmp.log, a name chosen to look like a VMware file.

So the source repository, the developers' machines and the code review all saw the real file. Only the compiled DLL, SolarWinds.Orion.Core.BusinessLayer.dll, carried the change, and it went through SolarWinds' normal signing step after the build. The signature was genuine because the signing process was genuine.

Step 3: SUNBURST, a backdoor that waited

FireEye's analysis of the shipped DLL shows how carefully it avoided being noticed at customers:

• It waited 12 to 14 days after the update was installed before doing anything, so a problem would not line up with the update in anyone's mind.
• It ran only inside the real Orion service (SolarWinds.BusinessLayerHost.exe) and only on a machine joined to a Windows domain, so sandboxes and test machines saw nothing.
• It checked for analysis and security tools by hashed process, service and driver names, and went quiet if it found them.
• It phoned home over DNS to a subdomain of avsvmcloud.com built from the victim's encoded domain name. The answer told the operators who they had reached, and only chosen victims got a second stage.
• Its traffic looked like Orion's own telemetry (the Orion Improvement Program), with commands hidden in normal-looking responses.
• On chosen networks, a memory-only dropper (TEARDROP) loaded Cobalt Strike, and the operators moved on with real credentials, naming their servers after the victim's own hostnames.

SolarWinds says the attackers removed the injection code from its build environment in June 2020. By then the updates were out, and they kept working.

Step 4: found from one extra phone

FireEye (now Mandiant) was one of the chosen victims. The intruders had an employee's username and password, but FireEye's VPN also needed a second factor, so they registered a second phone for that employee's MFA. That registration raised a low-priority alert. Someone followed up and called the employee, who had not added a phone.

Kevin Mandia later described it as a "severity zero" alert that turned into the whole investigation. On 8 December 2020 FireEye disclosed that its red-team tools had been stolen; on 13 December it published SUNBURST, SolarWinds published its advisory, and CISA's Emergency Directive 21-01 told federal agencies to disconnect Orion that day. Nobody had found it by looking at the update.

Two lessons in that: an alert for a new MFA device is cheap and worth reading every time, and the people who discovered one of the largest supply chain attacks did it by asking an employee a simple question.

Why the build step is the weak point

• Build servers are trusted and rarely watched. They hold the signing path and network access, and few teams log what runs on them.
• Signing proves who built it, not what went in. A signature from your own key on a tampered build is still a valid signature.
• Long-lived build machines collect implants. A server that runs every build for years gives an intruder a stable place to wait.
• Nobody compares the output with the source. If the DLL and the repository disagree, only a second independent build would show it.

What build integrity means for a small software shop

You are unlikely to face a state intelligence service. You will face stolen CI tokens, a poisoned dependency or a compromised build runner, and the same controls cover both.

1. Build on fresh machines. Use hosted CI runners or containers that are created for one build and thrown away. Nothing persists between builds for an implant to live in.
2. Keep the build definition in the repository and review it like code, so a change to how you build is as visible as a change to what you build.
3. Pin what the build pulls in. Lock files for packages, actions pinned to a commit SHA (the Action pin lookup does that for a list of uses: lines), and a checksum for every tool you download.
4. Record provenance. SLSA provenance or GitHub's artifact attestations write down which commit, which workflow and which runner produced each file, signed by the CI system, not by a key on a build box. Customers (and you) can check a download against it.
5. Make the build reproducible where you can, then build twice on separate systems and compare hashes. SolarWinds now runs builds in parallel in three separate environments for exactly this reason.
6. Keep the signing key off the build machine. A hardware key, a cloud KMS or a separate signing job that only signs artifacts with matching provenance.
7. Alert on the boring things: a new MFA device, a new scheduled task on a build host, a new admin on the CI system. FireEye's discovery started with one of them.

And on the receiving side, for software you install: watch what your monitoring servers talk to. A tool like Orion has no reason to reach random domains; customers who limited its outbound traffic would have cut SUNBURST off from its second stage.

Sources: CrowdStrike: SUNSPOT, an implant in the build process; Mandiant (FireEye): Highly evasive attacker leverages SolarWinds supply chain to compromise multiple global victims with SUNBURST backdoor; SolarWinds: New findings from our investigation of SUNBURST; Dark Reading: FireEye's Mandia, 'severity-zero alert' led to discovery of SolarWinds attack; Kevin Mandia, written testimony to the US Senate Intelligence Committee, February 2021; CISA: Emergency Directive 21-01; SolarWinds: next-generation build system and the NIST SSDF (2023); SLSA: provenance.