osec.one case analysis cover: Citrix Bleed 2023, session tokens leaked from the gateway

Most gateway breaches start with something stolen: a password, a phished code, a tired employee who taps "approve". Citrix Bleed (CVE-2023-4966) needed none of that. One unauthenticated web request to a NetScaler gateway returned a chunk of the device's memory, and in that memory sat the session cookie of someone who was already signed in. Paste the cookie into a browser and you are that person, past the password and past MFA.

The flaw was patched on 10 October 2023. The lesson most organisations learned the hard way came after: the patch closed the leak, but every session token already copied stayed valid. Attackers who had collected tokens before the update kept using them after it. This is how the leak worked, who was hit, and what "patch and terminate sessions" means for any login gateway, including the sign-in of your own web app.

Attack chain: a GET request with a very long Host header to the gateway's OpenID configuration page, the device returns its buffer plus the memory after it, a live NSC_AAAC session cookie is in those bytes, the attacker replays it with no password or MFA, the device is patched but the stolen session stays valid, and the attacker moves into the network with remote-management tools, credential dumps and ransomware
The bug leaked a login. The patch didn't take it back.

Step 1: a page that answers anyone, with the wrong length

NetScaler ADC and NetScaler Gateway can act as an OpenID Connect provider, so they publish a small JSON document describing themselves at /oauth/idp/.well-known/openid-configuration (and an /oauth/rp/ twin). Like every discovery document, it is public by design: no login needed.

Researchers at Assetnote compared the patched and unpatched builds and published the cause on 24 October 2023:

• The device builds that JSON with snprintf into a fixed buffer of 0x20000 bytes (128 KB).
• The hostname it writes into the document comes from the request's Host header, which the client controls, and it is written six times.
• snprintf stops writing when the buffer is full but returns the length it would have written. The code used that return value as the length of the response.
• So a request with a very long Host header got back the full buffer plus whatever memory followed it. In Assetnote's test the reply was 147,441 bytes from a 131,072-byte buffer.

The leaked bytes weren't random. Because the buffer sits at a fixed place in memory, the same neighbours came back each time, and in almost every device they tested a 32- or 65-character hex string appeared at the same spot: a valid NSC_AAAC session cookie for a signed-in user. No exploit code is needed to see why this is serious; the whole "attack" is an oversized header.

An attacker with access to a valid cookie can establish an authenticated session to the NetScaler appliance without knowledge of the username, password, or access to a multi-factor authentication token or device.
— Mandiant, 31 October 2023

Step 2: a session cookie is a finished login

A session cookie is what a site gives you after you have proved who you are. It doesn't ask the question again. MFA protects the moment of sign-in; it does nothing for a token that was minted when the real user passed MFA an hour earlier. That's why the CISA advisory files this under both "steal web session cookie" and "modify authentication process": the attacker doesn't defeat MFA, they arrive after it.

The gateway was the door to everything behind it: Citrix virtual desktops and apps, internal web tools, remote desktop. A hijacked gateway session put the attacker on a desktop inside the company network as an ordinary employee.

Step 3: a zero-day for six weeks, then everyone

• Late August 2023: Mandiant later found the flaw was being used as a zero-day against professional services, technology and government organisations.
• 10 October: Citrix publishes the bulletin and fixed builds (14.1-8.50, 13.1-49.15, 13.0-92.19 and the FIPS/NDcPP builds). Version 12.1 is end of life and gets no fix.
• 17 October: Citrix updates its bulletin: exploitation seen on unpatched devices. Mandiant publishes remediation advice the same day.
• 18 October: CISA adds the CVE to its Known Exploited Vulnerabilities catalog.
• 23 October: Citrix's blog spells out the next step: after upgrading, kill all active and persistent sessions.
• 24 October: Assetnote publishes the technical cause. Mass exploitation follows within days.
• 21 November: CISA, the FBI, MS-ISAC and Australia's ACSC publish advisory AA23-325A on LockBit 3.0 affiliates using the flaw, with indicators shared by Boeing.

Step 4: the patch that didn't log anyone out

Here is the gap. Patching changes the code that leaks memory. It doesn't touch the session table. A cookie copied on 9 October was still a live session on 11 October, on a fully patched device, until it expired or someone ended it.

Mandiant warned that compromised sessions persist after patching, and its CTO later described cases where session secrets were stolen before an organisation patched and the sessions were hijacked after. That's why Citrix's own guidance came as a pair: install the fixed build, then run:

text
kill aaa session -all
kill icaconnection -all
kill rdp connection -all
kill pcoipConnection -all
clear lb persistentSessions

Every user signs in again, with password and MFA. Anyone holding a stolen cookie is left holding nothing. NetScaler also advised rotating the credentials of identities used through a vulnerable device, because the device kept too few logs to prove which sessions had been taken.

Step 5: what attackers did with the sessions

The CISA advisory, built partly on what Boeing saw at Boeing Distribution Inc., its parts business on a separate network, describes the LockBit 3.0 affiliates' routine after the session was theirs:

• A PowerShell script drops a DLL into C:\Users\Public\ and runs it with rundll32; it talks to a domain made up to look like an Adobe update site.
• Remote-management tools for a way back in: AnyDesk and Splashtop, with others on the list such as ScreenConnect and Atera.
• Credential theft: LSASS memory dumps, Mimikatz, secretsdump.py; then movement with wmiexec, PsExec and Plink tunnels.
• Data taken, then ransomware.

Other groups used the same flaw. Comcast's Xfinity found that intruders had been inside between 16 and 19 October 2023, before it applied the fixes on 23 October, and its filing with the Maine attorney general lists 35,879,455 affected people: usernames and hashed passwords for all of them, and for some, contact details, birth dates, the last four digits of Social Security numbers and security questions.

Why it was hard to see

• The device's web server didn't log requests to the vulnerable page, so the leak itself left no trace on the box. Only a WAF or proxy in front of it would have the requests.
• A replayed session looks like the real user, already past MFA, so there is no failed login to alert on.
• By default ns.log rotates at about 2.5 MB, so on a busy gateway the useful days were gone quickly unless logs were sent elsewhere.
• The best tell Mandiant found: in session records, the client IP and the source IP don't match, and one unfamiliar address uses several different accounts within hours.

What "patch and terminate sessions" means for any login gateway

Citrix Bleed is about one product, but the shape repeats: Ivanti, Fortinet and other gateways have all had flaws that leak or forge sessions. And your own web app's sign-in is a login gateway too. The checklist is the same.

If you run a VPN or remote-access gateway

1. Know you have one and where its login page is. Sign up for the vendor's advisories with an address a person reads.
2. Patch within a day for a flaw that is already being exploited, and plan the sign-out at the same time: the patch is half the fix.
3. End every session after patching with the vendor's commands, then rotate credentials for accounts that used the device in the exposed window, starting with administrators and service accounts.
4. Hunt before you reboot. A reboot clears memory evidence. For NetScaler, Mandiant's advice was to take a core dump first.
5. Send gateway logs to another system and keep them for months, not megabytes. The CISA advisory asks for 180 days of retention for enhanced PowerShell logging on Windows; the gateway's own logs deserve at least as long.
6. Alert on session oddities: one session used from two IP addresses, several accounts from one address, desktop sessions starting at 3 a.m.
7. Assume full compromise if the box was exposed during the zero-day window. CISA's advice for LockBit cases was to assume the attackers have administrator access to the network.

If you run a web app with sign-in

1. Keep sessions on the server, or keep a session version per user, so that "sign everyone out" is one statement, not a wait for cookies to expire.
2. Build the global sign-out before you need it, and test it. On the day of an incident nobody wants to write a migration.
3. Keep session lifetimes short for admin and sensitive areas, and ask again (a fresh code) before changing an email address, a payout account or a password.
4. Notice when a session moves: a token minted in one country and used from a data centre an hour later deserves a fresh sign-in.
5. Leave nothing reusable to steal at the login: sign-in by email code or Google/Apple leaves no password to reuse elsewhere, so a copied session is the only prize, and it's one you can revoke.

More on stolen sessions after MFA: Session cookie theft after MFA: why passkeys don't fully fix it. Two more gateway cases: Change Healthcare 2024, where a Citrix portal had no MFA at all, and Ivanti Connect Secure 2024, two chained zero-days on the VPN gateway.

osec Scan finds exposed VPN and remote-access login pages on a host, Citrix NetScaler Gateway among them, and lists what CISA's known-exploited catalog says about each. Tick the products you run and get one email when a new exploited flaw is listed.

Scan a host

Sources: Citrix (Cloud Software Group): CVE-2023-4966, critical security update for NetScaler ADC and NetScaler Gateway; CISA: guidance for the Citrix NetScaler ADC and Gateway vulnerability CVE-2023-4966; CISA, FBI, MS-ISAC and ACSC: advisory AA23-325A, LockBit 3.0 ransomware affiliates exploit CVE-2023-4966 (21 November 2023); Assetnote: Citrix Bleed, leaking session tokens with CVE-2023-4966 (24 October 2023); Mandiant: investigation of session hijacking via Citrix NetScaler ADC and Gateway vulnerability; Help Net Security: Citrix Bleed, mass exploitation in progress; Cybersecurity Dive: Comcast's Xfinity data breach tied to CitrixBleed; Dark Reading: Comcast Xfinity breached via CitrixBleed, 35M customers affected.