3CX 2023: a supply chain attack that began with another supply chain attack

In March 2023 the desktop app of 3CX, a business phone system that the company says is used by 600,000 organisations and 12 million people a day, started doing something a softphone has no reason to do: reading browser history and fetching instructions from icon files on GitHub. The app was the real one. It came from 3CX's own update servers and carried 3CX's own code signature.

A month later Mandiant, hired by 3CX, traced the intrusion back to its start. It wasn't a phishing email or an unpatched server. It was another vendor's installer, tampered with a year earlier, that one employee had put on a personal computer. Mandiant called it the first time it had seen one software supply chain attack lead to another.

Step 1: a retired trading app with a valid signature

X_TRADER was a futures trading program made by Trading Technologies. The company retired it in 2020, but the installer could still be downloaded from its website in 2022.

• Google's Threat Analysis Group had reported that the Trading Technologies site was compromised in February 2022 by a North Korean group.
• The site later served an X_TRADER installer with extra files in it: two tampered DLLs next to a harmless program that loaded them.
• The installer was signed "Trading Technologies International, Inc." with a certificate that stayed valid until October 2022. Windows showed a known publisher; nothing looked wrong.
• Once run, it installed a backdoor Mandiant named VEILEDSIGNAL, which hid its network traffic inside a running Chrome, Firefox or Edge process.

3CX was not the only place it landed. Symantec found the same installer at four more organisations: two energy companies (one in the US, one in Europe) and two firms in financial trading.

Step 2: from a home PC to the company network

According to 3CX's account of Mandiant's findings, an employee installed X_TRADER on a personal computer in 2022. The backdoor gave the intruders that machine, and from it they took the employee's 3CX corporate credentials.

That one sentence carries most of the lesson. A phone-system vendor has no use for futures trading software. The program was on a machine the company didn't manage, installed for a private reason, and that machine also held a working sign-in to the company. No policy about "approved software" at 3CX applied to it, and no company security tool was watching it.

Step 3: the build machines

With a valid staff sign-in the intruders moved through 3CX's network. Mandiant found:

• a copy of Fast Reverse Proxy, an open-source tunnelling tool, saved as MsMpEng.exe, the file name of Microsoft's own antivirus engine;
• on the Windows build environment, a launcher and downloader (TAXHAUL and COLDCAT) kept alive by a hijacked Windows service running as LocalSystem;
• on the macOS build server, a backdoor called POOLRAT, kept alive with a launch daemon.

Build machines are the point of the exercise. Whatever they compile gets signed with the company's certificate and sent to every customer by the product's own updater.

Step 4: a signed update that phoned home

In late March 2023 the Windows and macOS versions of the 3CX desktop app (on Windows, version 18.12.416 and earlier builds in that line) shipped with altered libraries. On Windows the chain was:

1. The signed installer placed the real app plus a modified library that the app loads at start.
2. That library unpacked a second stage, which fetched icon files from a GitHub repository. Extra data appended to each icon decoded to the address of a control server.
3. From there it downloaded a small information stealer. It read the app's config, the computer and domain name, and the browsing history of Chrome, Edge, Brave and Firefox.

Browsing history sounds minor. For the attacker it answers "whose computer is this?" for hundreds of thousands of machines, so the interesting ones could be picked out for a hand-delivered next stage. Mandiant attributes the work to a North Korean cluster it tracks as UNC4736, linked to earlier fake-trading-app campaigns.

Step 5: a week of "false positive"

This is the part every small firm should read twice.

• 22 March 2023: SentinelOne's behavioural detection begins to flag and quarantine the 3CX desktop app across its customers. On 3CX's own community forum a thread opens titled "Threat alerts from SentinelOne for desktop update initiated from desktop client".
• For the next week: customers compare notes. The app is signed by the vendor, it came through the vendor's updater, the alert is behavioural and not a named malware family. Several posters treat it as a probable false positive. A 3CX support reply points customers to SentinelOne, since it is that product raising the alert.
• 29 March: CrowdStrike publishes that it sees the same activity and names a North Korean group. Other vendors follow the same day, and 3CX acknowledges the compromise.
• 30 March: CISA issues an alert; Apple revokes the Mac app's notarisation.

The reasoning that wasted the week was sound-looking: the file is signed by a vendor we trust, so the alert must be wrong. A code signature proves two things only: which company's key signed the file, and that the file hasn't changed since. It says nothing about whether the build machine that produced the file was under someone else's control. Twice in this one story, at Trading Technologies and at 3CX, the signature was real and the software was hostile.

A valid signature answers "who signed this?". It never answered "what does it do?".

Timeline

• February 2022: Trading Technologies' website compromised.
• 2022: a 3CX employee installs the tampered X_TRADER on a personal computer; 3CX credentials stolen.
• Between then and March 2023: intruders inside 3CX reach the Windows and macOS build systems.
• March 2023: backdoored desktop app versions ship through normal updates.
• 22 March: first endpoint detections; forum thread about false positives.
• 29 to 30 March: CrowdStrike and others go public; 3CX acknowledges it; CISA issues its alert.
• 20 April: Mandiant and 3CX publish the X_TRADER link. 21 April: Symantec reports four more X_TRADER victims.

What a small firm can check about the software it installs

You can't audit a vendor's build server. You can change what a poisoned update is able to do, and how you react when a tool complains about one.

1. Keep work sign-ins off machines you don't manage

The first hop was a personal PC that held a company password. Decide which is true for you: staff use company-managed devices for work accounts, or work accounts are reachable only through sign-in that a stolen password can't replay (passkeys or security keys, plus a device check where your identity provider offers one). "Personal laptop with the VPN client and a saved password" is the combination to end.

2. Keep a list of what is installed, and remove what nobody supports

X_TRADER had been retired for two years. Retired software gets no fixes, and its download page gets no attention from its maker.

• Once a quarter, export the installed-programs list from each machine (your endpoint tool or RMM does this; on a handful of PCs, winget list is enough).
• For each item: who asked for it, is it still sold and updated, when was it last updated?
• Uninstall what has no owner or no vendor.

3. Treat an alert on signed software as a question for the vendor, with a deadline

Write this down before you need it:

1. Don't add an exclusion on day one. Leave the block in place. A softphone being quarantined for two days is an inconvenience; use the web client or a phone.
2. Read what the alert says the program did (spawned a shell, injected into a browser, connected to a new domain). Behaviour is evidence; the publisher name is not a rebuttal.
3. Check whether other customers see it: the vendor's forum, your security vendor's advisories.
4. Ask the vendor in writing whether the behaviour is expected in that version. No clear answer in a day or two means the block stays.
5. If an exclusion is ever added, scope it to one file hash, never to a folder or a publisher, and put an expiry date on it.

4. Give desktop apps less to reach

• Staff accounts without local administrator rights. Most business apps install and update without them.
• A softphone needs your phone system's address and little else. If your firewall or DNS filter can limit an app's outbound connections, a request to a code-hosting site from a phone client stands out.
• Where a vendor offers a browser version of the same tool, prefer it for most staff. A web app can't load a tampered library onto the machine.

5. Ask vendors two questions

Before renewing anything that auto-updates on staff machines: do they publish security advisories and where, and how would they tell you about a compromised release? A vendor with a status page and an advisory feed is one you can check in five minutes during the next week like this one.

The lesson

Nobody at 3CX's customers did anything unusual. They ran the vendor's app and accepted its updates, which is what everyone is told to do. The failures were upstream twice over, and the only control downstream that worked was the one that judged the program by its behaviour instead of its signature. The habit to take from this case is small: when a security tool flags trusted software, the burden of proof is on the software. Compare XZ Utils, where the tampering was in the release tarball and not in the reviewed source, and LastPass 2022, which also began on an employee's home computer.

Sources: Mandiant: 3CX software supply chain compromise initiated by a prior software supply chain compromise (20 April 2023); SentinelOne: SmoothOperator, ongoing campaign trojanizes 3CXDesktopApp; CISA: Supply chain attack against 3CXDesktopApp (30 March 2023); Symantec: X_Trader supply chain attack affects critical infrastructure organizations; BleepingComputer: 3CX hack caused by trading software supply chain attack; The Record: energy sector organisations hit by same supply chain attack as 3CX; 3CX community: Threat alerts from SentinelOne for desktop update initiated from desktop client; Techdirt: 3CX knew its app was being flagged by AV platforms.