osec.one case analysis cover: PAN-OS GlobalProtect 2024, a session cookie that wrote a file, and a firewall that ran it as a command

CVE-2024-3400 is two harmless-looking things joined together. The first: Palo Alto's GlobalProtect VPN service took a file path from the session cookie and created an empty file there, as root, without checking it. The second: a scheduled telemetry job on the same firewall put file names into a shell command. Put a shell command in the file name and the firewall runs it, as root, for anyone on the internet who can reach the VPN login page.

Volexity caught it on 10 April 2024 at a customer, with the earliest attempt dated 26 March. Palo Alto published the advisory on 12 April with a CVSS score of 10.0, shipped the first hotfixes on 14 April and the last ones on 18 April. In between, the first suggested workaround, turning telemetry off, turned out not to help. This write-up goes through each step, what the attacker Volexity calls UTA0218 did once inside, and what a small team with one Palo Alto firewall should check and rotate.

Attack chain: a crafted SESSID cookie carries a path; gpsvc creates a zero-byte file there as root with an attacker-chosen name; the scheduled telemetry upload builds a shell command that includes the file name; the name runs as a root command (exploited from 26 March 2024); the UPSTYLE backdoor reads commands from 404 lines in the web server error log; the attacker uses the firewall's domain admin service account to reach the internal network and copy NTDS.dit and browser logins
Neither half was enough alone: an empty file is harmless, and the telemetry job only read file names the system wrote itself.

The timeline

text
CVE-2024-3400, 2024 (Volexity, Palo Alto, Unit 42, watchTowr, CISA, Shadowserver)

26 Mar        earliest attempt Volexity found: zero-byte test files
10 Apr        Volexity's sensor flags a customer's firewall; zero-day found
11 Apr        same exploitation at a second Volexity customer
12 Apr        Palo Alto advisory (CVSS 10.0), Unit 42 "Operation MidnightEclipse",
              Volexity write-up; CISA adds it to KEV (federal due date 19 Apr)
14 Apr        first hotfixes: 10.2.9-h1, 11.0.4-h1, 11.1.2-h3
15-18 Apr     hotfixes for the older maintenance releases
16 Apr        watchTowr explains the bug; "disable telemetry" withdrawn as a fix
~15-19 Apr    public exploit code; "numerous threat actors" (BleepingComputer)
17 Apr        Palo Alto adds a log grep to check for attempts
25 Apr        clean-up advice by level of compromise
3 May         Enhanced Factory Reset made available through support

Who was exposed

PAN-OS 10.2, 11.0 and 11.1 on firewalls with a GlobalProtect portal or gateway configured (Network > GlobalProtect > Portals / Gateways). Not affected: Cloud NGFW, Panorama, Prisma Access, and PAN-OS 10.1 and older. Firewall VMs in your own cloud account count as firewalls. The counts once the bug was public:

text
Date (2024)   Count                                          Source
~12 Apr       ~82,000 firewalls said to be vulnerable        Y. Sejiyama scan
week of 15    >156,000 PAN-OS devices on the internet        Shadowserver
18 Apr        ~22,500 "possibly vulnerable"                  Shadowserver
              (US 9,620; Japan 960; India 890;
               Germany 790; UK 780)
21 Apr        ~6,200 still vulnerable                        Shadowserver (SC Media)

From ~22,500 to ~6,200 in three days is a 72% drop: most owners patched within a week of the first fix. The remaining 6,200 were three days into a period when public exploit code existed.

Step 1: a path in the session cookie

GlobalProtect's web requests reach a backend service written in Go, gpsvc. It stores session data in files, and it built the file name from the SESSID cookie the browser sent. watchTowr Labs found it didn't clean that value, so a cookie containing ../ sequences pointed the file somewhere else on the disk.

If nothing existed at that path, gpsvc created the folders and an empty file, running as root. The attacker chose the name and the place, but not the contents. On its own that is a nuisance, not a takeover: watchTowr called it a "primitive". That's why Volexity's earliest finds, from 26 March, were zero-byte files: someone was testing whether the first half worked.

Step 2: a telemetry job that put file names into a shell command

PAN-OS has a scheduled Python job that uploads device telemetry to Palo Alto. watchTowr traced it to a script that built a curl command line, including names of files from a working directory, and ran that line through a helper that interprets it as shell. The job assumed only the system itself wrote files there.

Join the halves: an empty file whose name is a shell command, placed where the telemetry job looks, and the next scheduled run executes the command as root. Spaces were stripped from the cookie, which public write-ups note was worked around with the shell's ${IFS} variable. No sign-in was needed at any point; the cookie went to the VPN login page.

Why "disable telemetry" stopped being advice

The advisory of 12 April said exposure needed GlobalProtect and device telemetry, and listed turning telemetry off as a stopgap. By 16 April the advisory said: "Disabling device telemetry is no longer an effective mitigation", and that telemetry did not need to be on for a firewall to be exposed. watchTowr's post was updated the same way. The file-creation half lived in gpsvc whatever the telemetry setting, and Palo Alto didn't publish what else could act on the file. Owners who had relied on the stopgap for four days were exposed for those four days.

The stopgap that did hold was the IPS signature: Threat IDs 95187, 95189 and 95191 (content 8836-8695 or later), but only on a firewall with a Threat Prevention subscription and a Vulnerability Protection profile applied to the GlobalProtect interface. Without the profile on that interface, the signature was installed and did nothing.

Step 3: UPSTYLE, a backdoor that read the error log

Volexity's first customer had a backdoor installed with the bug, which it named UPSTYLE. How it hid is the useful part for anyone reading logs:

text
Piece                What it did (Volexity, Unit 42)
install              update.py writes /usr/lib/python3.6/site-packages/system.pth;
                     Python runs .pth import lines whenever it starts, so
                     the backdoor starts with every Python process
command in           the attacker requests a page that doesn't exist, with an
                     encoded command in the URL; the 404 lands in
                     /var/log/pan/sslvpn_ngx_error.log; UPSTYLE reads it
command out          output appended to the portal's bootstrap.min.css,
                     a file any browser can fetch
cleanup              the CSS file restored after 15 seconds, the log line
                     removed, file timestamps put back

Unit 42 saw other attempts on other firewalls: three UPSTYLE installs that failed, copies of running_config.xml placed in the web folder and fetched over HTTP, a script downloaded with wget ... | bash, and a cron entry that fetched and ran a remote policy file every minute.

Step 4: from the firewall into the network

At Volexity's customer the firewall itself was not the goal. UTA0218, which Volexity judged "highly likely" state-backed, installed the GOST tunnelling tool and a reverse SSH shell on port 31289, then used a highly privileged service account configured on the firewall, a domain admin, to move over SMB and WinRM to internal Windows machines. From there it took:

1. NTDS.dit, the Active Directory database with every user's password hash;
2. DPAPI keys and Windows Terminal Services event logs;
3. Chrome and Edge Login Data, cookies and Local State from users' machines: enough to decrypt their saved passwords and reuse their sessions.

The firewall had stored a domain admin's credentials so it could look up users for the VPN. That account turned root on one box into the keys to the domain. Volexity's network signs worth an alert: the firewall making direct-to-IP wget calls, SMB or RDP from the firewall to internal hosts, and HTTP requests from it to worldtimeapi.org.

Clean-up by level

Unit 42 sorted what it saw into four levels; Palo Alto's knowledge base gave the clean-up for each on 25 April:

text
Level              What was found                         Clean-up (KB, 25 Apr 2024)
0 Probe            failed attempt                         hotfix
1 Test             zero-byte file, no command seen        hotfix
2 Potential        a file (usually running_config.xml)    hotfix + Private Data Reset
  exfiltration     copied to a web-readable place
3 Interactive      backdoor, commands run, files          hotfix + factory reset
  access           downloaded

Two later changes. On 3 May Palo Alto added an Enhanced Factory Reset, done with support, for anyone who hadn't applied the fix or the signature with the profile by 25 April, or who worried about persistence; on 29 April it had noted public proof-of-concepts of "post-exploit persistence techniques that survive resets and upgrades". And a private data reset does not wipe the disks. The current knowledge base article points compromised hardware to the enhanced reset and VM firewalls to a fresh instance.

The hotfix builds, per branch

Fixed in each build below and every later release (Palo Alto advisory):

text
Branch  Hotfix builds (release date, 2024)
11.1    11.1.2-h3 (14 Apr), 11.1.1-h1 (16 Apr), 11.1.0-h3 (16 Apr)
11.0    11.0.4-h1 (14 Apr), 11.0.4-h2 (17 Apr), 11.0.3-h10 (16 Apr),
        11.0.2-h4 (16 Apr), 11.0.1-h4 (18 Apr), 11.0.0-h3 (18 Apr)
10.2    10.2.9-h1 (14 Apr), 10.2.8-h3 (15 Apr), 10.2.7-h8 (15 Apr),
        10.2.6-h3 (16 Apr), 10.2.5-h6 (16 Apr), 10.2.4-h16 (18 Apr),
        10.2.3-h13 (18 Apr), 10.2.2-h5 (18 Apr), 10.2.1-h2 (18 Apr),
        10.2.0-h3 (18 Apr)

Azure Marketplace names add a 0: 11.1.2-h3 is listed as 11.1.203.

Hotfix numbering is where people slip: 10.2.9-h1 is fixed, 10.2.9 is not, and 10.2.8-h2 is not either. The osec Firmware branch check reads hotfix suffixes and lists each known-exploited PAN-OS CVE that still applies to the build you type, with the fixed build on your branch.

What breaks each step: one firewall, small team

text
Step                          Control
1 cookie path, empty file      patch to the hotfix for your release; until then the
                               signatures 95187/95189/95191 with a Vulnerability
                               Protection profile on the GlobalProtect interface
2 telemetry runs the name      same patch; don't rely on a feature toggle the vendor
                               hasn't confirmed as a fix
3 UPSTYLE in the logs          forward logs off the box as written (syslog); alert
                               on threat IDs above; take a support file (TSF)
                               before rebooting into the fix
4 domain admin on the firewall a read-only LDAP bind account, never a domain admin;
                               block the firewall's own outbound SMB/RDP/WinRM to
                               the LAN; alert on its direct-to-IP downloads

If your firewall ran an affected build with GlobalProtect on

1. Check the logs on the PAN-OS CLI with Palo Alto's command: grep pattern "failed to unmarshal session(.\+.\/" mp-log gpsvc.log*. A normal line has a GUID between session( and ); a path or a shell command there is an attempt.
2. Open a support case and upload a TSF. Palo Alto checks it for indicators.
3. Level 2 or 3, or unsure: export the configuration, read it for changes you didn't make, then have support run the Enhanced Factory Reset (hardware) or build a new VM.
4. Rotate everything the firewall held: local admin passwords, the LDAP/AD bind account (the path into the network here), RADIUS secrets, IPsec pre-shared keys, the master key (switch to AES-256-GCM), and every certificate under Device > Certificates, the GlobalProtect portal and gateway certificates included.
5. Reset VPN users' passwords if the firewall saw their logins while it was compromised.

The same pattern runs through the other gateway cases on this site: a bug on the box, then credentials and sessions that outlive the patch. In Ivanti Connect Secure 2024 two chained bugs gave the same unauthenticated root on a VPN gateway in the same season; in Citrix Bleed 2023 patching didn't log the attackers out.

Type your PAN-OS build, hotfix included, and see which known-exploited CVEs still apply and the fixed build on your branch. Save it as a watch and get one email when a new one starts applying.

Check your PAN-OS build

Sources: Palo Alto Networks: CVE-2024-3400 advisory (published 12 April 2024, updated 3 May 2024); Unit 42: Threat Brief, Operation MidnightEclipse (12 April 2024, updated 20 May 2024); Volexity: Zero-Day Exploitation of Unauthenticated Remote Code Execution Vulnerability in GlobalProtect (12 April 2024); watchTowr Labs: Palo Alto, Putting The Protecc In GlobalProtect (16 April 2024); CISA: Palo Alto Networks releases guidance for CVE-2024-3400 (12 April 2024); CISA Known Exploited Vulnerabilities catalog; BleepingComputer: 22,500 Palo Alto firewalls possibly vulnerable (19 April 2024); SC Media: 6.2K Palo Alto firewalls still at risk (April 2024); SecurityWeek: Palo Alto shares remediation advice for hacked firewalls (25 April 2024); The Hacker News: Palo Alto outlines remediation (26 April 2024); Palo Alto knowledge base: remediation for CVE-2024-3400.