Ivanti Connect Secure 2024: two chained zero-days on the VPN gateway

A VPN gateway is the one box a company deliberately leaves open to the whole internet, so that staff can reach everything behind it. In the winter of 2023 to 2024 two flaws in Ivanti Connect Secure (the product once sold as Pulse Secure) let a stranger run commands on that box as root, with no username and no password.

The flaws were the easy part to understand. What made this case different is what came after: three weeks with no patch, an integrity checker that the intruders had taught to look away, and an order from the US government to its own agencies to disconnect the product within about 48 hours. Most of the lessons are about those three things, and they apply to any single appliance at the edge of a small network.

Step 1: two bugs that only work as a pair

• CVE-2023-46805 (CVSS 8.2) is an authentication bypass in the appliance's web component. A request shaped the right way reaches pages that should need an administrator's session.
• CVE-2024-21887 (CVSS 9.1) is a command injection in those administrator pages. Input that should have been data ends up in a shell command.
• On its own the second bug needs an admin login, which is why it scores below 10. The first bug supplies that. Chained, they give unauthenticated remote code execution on the gateway.
• All supported versions were affected, 9.x and 22.x, and so was Ivanti Policy Secure.

This is worth remembering when you read an advisory. "Requires administrator privileges" sounds like a small risk. It stops being small the day someone finds a way round the login on the same device.

Step 2: found from the inside, not on the appliance

The security firm Volexity found the attack in the second week of December 2023, and not by looking at the VPN. Its network monitoring at a customer saw lateral movement: someone placing web shells on internal and external web servers. Tracing that traffic back led to the customer's Ivanti appliance, where activity went back to 3 December.

A closer inspection of the ICS VPN appliance showed that its logs had been wiped and logging had been disabled.
— Volexity, 10 January 2024

Volexity recovered the exploit from a memory sample of the device and reported both flaws to Ivanti. It named the group UTA0178 and believes it works for the Chinese state. Mandiant, which investigated other victims, tracks the same activity as UNC5221 and at the time described it only as a suspected espionage actor.

Step 3: living inside the appliance's own files

The intruders didn't install obvious new programs. They edited files the product already had, so the device kept working and nothing looked out of place:

• A legitimate CGI script (compcheckresult.cgi) got a few extra lines that run whatever command arrives in a request. Mandiant calls this web shell LIGHTWIRE.
• A Python file of the web server (visits.py) got a second web shell. Volexity calls it GIFTEDVISITOR, Mandiant WIREFIRE.
• A JavaScript file loaded by the VPN's login page (lastauthserverused.js) was changed to send every username and password typed there to a server the attackers controlled. Mandiant calls it WARPWIRE.
• A dropper script put the web shell back after a system update, so upgrading the appliance did not remove it.

The login-page change is the one that turns a device problem into a company problem. From that moment, every member of staff who signed in to the VPN handed over a working password, and usually it was their main network password.

Step 4: the integrity checker was told to skip those files

Ivanti appliances have a built-in Integrity Checker Tool that compares the files on the device with a list of what should be there. The attackers' script added its changed files to the checker's exclusion list. The checker ran, compared everything it was allowed to compare, and reported nothing.

Ivanti told customers to run an external version of the checker, from a fresh download. That one reboots the device, which wipes the memory an investigator would want. On 29 February 2024 CISA, the FBI and partner agencies in Australia, Canada, New Zealand and the UK published what they had seen in their own incident response and lab work.

The agencies wrote that attackers can "deceive Ivanti's internal and external Integrity Checker Tool (ICT), resulting in a failure to detect compromise". The same advisory says CISA found web shells on devices whose checks showed no mismatched files, and that in its lab an attacker could keep root-level access through a factory reset. A tool that runs on a machine the attacker controls can only report what the attacker lets it see. That's true of every appliance, not only this one.

Step 5: from the gateway into the network

With staff passwords from the login page and the service account the appliance itself used to talk to the directory, the intruders moved on. Volexity's account of one victim:

• Connections to internal systems over RDP, SMB and SSH with stolen logins.
• Memory of the Windows login process dumped on several machines and taken away for offline analysis.
• A backup disk image of a domain controller mounted, and the Active Directory database (ntds.dit) copied out of it. That file holds a password hash for every account in the company.
• Saved credentials pulled from the backup software.
• More web shells left on other web servers as a way back in.

CISA's later lab work adds one detail a small organisation should check today: the directory account saved on the appliance is often a domain administrator, and it can be read out of a compromised device with little effort.

Timeline: three weeks without a patch

• 3 December 2023: earliest known activity on a victim's appliance.
• 10 January 2024: Ivanti and Volexity go public. There is no patch, only a configuration file to import as a mitigation. Ivanti says it knows of fewer than 10 affected customers.
• 11 January: exploitation spreads. By 15 January Volexity has scanned about 30,000 appliances and found the same web shell on more than 1,700, from small businesses to Fortune 500 companies.
• 19 January: CISA issues Emergency Directive 24-01: federal agencies must apply the mitigation by 22 January, run the external checker and report what they find.
• 31 January: the first patches arrive, later than first planned. The same day Ivanti discloses two more flaws, one of them (CVE-2024-21893) already being exploited.
• 31 January: CISA's supplemental direction: disconnect every Connect Secure and Policy Secure device by 11:59 pm on Friday 2 February. To bring one back: export the configuration, factory reset, rebuild on a patched version, then revoke and reissue every certificate, key and password on it.
• 8 February: a fifth flaw (CVE-2024-22024).
• By 1 March: agencies must treat every domain account tied to the appliance as compromised, reset passwords twice and revoke Kerberos tickets.
• March and April: CISA confirms it took two of its own systems offline in February after intruders came in through its Ivanti devices. MITRE says a research network of its own was breached the same way in January.

The mitigation mattered and it wasn't enough. Volexity said it plainly on day one: a mitigation "does not remedy a past or ongoing compromise". If the device was reachable from the internet between early December and the day you imported the file, the question was no longer whether to patch.

What an organisation with one VPN appliance should do

A federal agency has a security team, spare hardware and someone whose job is to read directives. A 30-person firm has one appliance, set up by a contractor years ago, and one person who looks after IT on Tuesdays. The advice below is for that firm, with any vendor's box: Fortinet, Citrix, SonicWall, Palo Alto and Cisco gateways have all had flaws exploited in real attacks too.

Before anything happens

1. Know that you have one, and who gets the vendor's emails. Sign up for the vendor's security advisories with an address a person reads. Many small firms learned about this case from the news.
2. Watch the list of flaws that are really being used. CISA's Known Exploited Vulnerabilities catalog listed both flaws on 10 January, the day they were made public. osec Scan looks for the login pages of twelve VPN, remote-access and file-transfer products on a host and shows what the catalog lists for each. Its known-exploited watch emails you when a product you ticked gets a new entry.
3. Give the appliance the smallest directory account that works. It needs to look up users, not administer the domain. Check this one today.
4. Send its logs somewhere else. A syslog server or a log service, kept for 90 days. The first thing done on the device was wiping its logs.
5. Limit what the appliance can reach. VPN users should land in a network segment with rules, not everywhere. The appliance itself rarely needs to start connections to the internet. Outbound traffic from it to an unknown address is the cheapest alarm you can build.
6. Decide how you'd work without it for three days. Which systems have their own secure sign-in on the web, who needs the office network at all, and who can order the box unplugged at 9 pm on a Friday.

The day a flaw with no patch is announced

1. Apply the vendor's mitigation the same day. Read the instructions first: this one switched off some management features.
2. If you can do without the VPN, unplug it until there's a patch. That is what CISA ended up ordering. It's a business decision, so make it with the owner, not alone.
3. Before any reboot or integrity check, ask for help if you suspect a break-in. The check reboots the device and the evidence in memory is gone. A forensic firm or your insurer's incident line can tell you what to save.
4. Treat a mismatch from the integrity check as a confirmed break-in. Treat a clean result as "nothing found", not "nothing there".

If the device was exposed while the flaw was being used

1. Rebuild, don't clean. Export the configuration, factory reset, install the patched version from the vendor, import the configuration. Where a virtual appliance is an option, a fresh image is safer than a reset.
2. Replace every secret that was on the box: the admin password, local users, the directory service account, API keys, and the TLS certificate with its private key.
3. Reset the password of everyone who signed in through it during the exposed period, and end their sessions. The login page may have been copying them.
4. If the appliance was joined to a Windows domain, follow CISA's instruction to its agencies: reset account passwords twice and revoke Kerberos tickets, and in a hybrid setup revoke cloud tokens too.
5. Look inward. New admin accounts, sign-ins at odd hours, remote desktop sessions that start from the VPN's address, unknown files on web servers.

The wider lesson: fewer doors, and no reusable passwords behind them

Every extra login page facing the internet is another product whose worst day becomes yours. The Ivanti case was bad for its victims because the gateway held reusable passwords and a powerful directory account, and stood in front of a flat network. Take those three things away and the same two bugs give an intruder a box with little on it.

For the web applications a small team uses, a sign-in that has no password to copy (an email code, Google or Apple) leaves a tampered login page with nothing worth keeping. For the gateway itself, the list above is the work. Related reading: Change Healthcare 2024, where the open door was a Citrix portal without MFA, and MOVEit 2023, another internet-facing product hit before a patch existed.

Sources: Volexity: Active exploitation of two zero-day vulnerabilities in Ivanti Connect Secure VPN (10 January 2024); Volexity: Ivanti Connect Secure VPN exploitation goes global (15 January 2024); Mandiant: Suspected APT targets Ivanti Connect Secure VPN in new zero-day exploitation; CISA: Emergency Directive 24-01 and its supplemental directions; CISA and partners: advisory AA24-060B, threat actors exploit multiple vulnerabilities in Ivanti Connect Secure and Policy Secure gateways (29 February 2024); TechTarget: Ivanti confirms 2 zero-day vulnerabilities are under attack; The Record: CISA takes two systems offline following Ivanti compromise; Decipher: nation-state actors exploited Ivanti bugs to hit MITRE.