When a vendor you use is breached, the advice is always the same two words: rotate credentials. In October 2023 Cloudflare was one of the customers caught in Okta's support-system breach, and it did rotate. It stopped the first attempt to use a stolen session, wrote it up, and moved on.
It had missed four. One service token and three service accounts were left as they were, because the team believed they were no longer in use. On 14 November someone started trying them. Nine days later, on Thanksgiving Day in the US, an automated alert caught the intruder inside Cloudflare's Atlassian server. The clean-up took until January and included wiping every machine in the company's global network.
Cloudflare published a detailed report. It is one of the best public examples of a rotation that was 99% done.
The four credentials
Cloudflare's report lists them by what each could reach:
• a Moveworks service token that gave remote access through Cloudflare's gateway to its Atlassian system (Moveworks is an IT help-desk assistant);
• a Smartsheet service account with administrative access to the Jira instance;
• a Bitbucket service account with access to the source code management system;
• a credential for an AWS environment behind the Cloudflare Apps marketplace, with no access to the global network and no customer data.
Cloudflare is blunt that none of those vendors did anything wrong. The credentials had been exposed in the Okta incident and were "mistakenly believed to be unused".
Look at the kind of credential on the list. None belongs to a person. They are the logins of integrations: a help-desk bot, a spreadsheet connector, a code-hosting sync. Nobody types them, nobody gets a sign-in prompt for them, none of them has a second factor, and nobody's morning is ruined if one is forgotten. They are missed for exactly those reasons.
Day by day
All times are from Cloudflare's report (UTC).
• 18 October: the Okta compromise. Cloudflare's credentials are among the material taken from Okta's support system.
• 14 November, 09:22: probing starts. Attempts to sign in to Cloudflare's Okta and to the Cloudflare dashboard are refused. The AWS credential works, and reaches an environment with nothing of value.
• 15 November, 16:28: the Moveworks token gets through the gateway and the Smartsheet account signs in to Jira and Confluence.
• 16 November: the intruder creates a user account made to look like an ordinary Cloudflare employee and adds it to several groups, so that access survives if the Smartsheet account is noticed.
• 17 to 21 November: almost nothing. A couple of brief sign-ins to check the access still works.
• 22 November: using the ScriptRunner for Jira plugin, which lets Jira administrators run scripts on the server, the intruder installs Sliver, an open-source remote-control framework. With that foothold and the Bitbucket account they go through source code, and try to reach a console server in a new data centre in São Paulo that isn't in production yet. That attempt fails.
• 23 November, 15:58: the Smartsheet account is added to an administrators group. 16:00: an automated alert fires. By 16:35 the account is disabled; by 17:23 the fake employee account is found and disabled; at 21:31 firewall rules block the intruder's addresses.
• 24 November: last activity at 10:44. Sliver is removed an hour later.
What they looked at
The numbers are small, and the searches say what the intruder wanted:
• Confluence: 202 wiki pages out of 194,100. Search terms included "remote access", "secret", "token" and "cloudflared".
• Jira: 36 tickets out of more than 2 million.
• Bitbucket: 120 repositories viewed out of 11,904; 76 downloaded as archives. Most were about backups, network configuration, identity, remote access, Terraform and Kubernetes. A few held encrypted secrets, which were rotated anyway.
No customer data, no production network. Cloudflare's reading was that a patient and probably state-backed intruder was mapping how to get lasting access to the global network. The outside forensics firm it brought in found nothing Cloudflare had missed.
Nobody was reading the wiki for fun. They were searching it for the next credential.
Why it stopped where it did
Three things limited the damage, and all three are within reach of a small company:
1. The stolen credentials couldn't sign in to the identity provider or the admin dashboard. Those need hardware security keys at Cloudflare. A token for one integration opened that integration's door and nothing else.
2. Network rules held. The Atlassian server could not talk to most of the network. The one gap, the São Paulo console server, was still protected by its own access control.
3. A change to an admin group raised an alert within two minutes. That single rule ended a nine-day intrusion.
The cost of four
Cloudflare called the response "Code Red" and put much of its technical staff on it from 27 November to 5 January:
• more than 5,000 production credentials rotated, this time all of them;
• 4,893 systems forensically checked;
• every machine in the global network reimaged and rebooted, plus all the Atlassian servers;
• the hardware in the São Paulo data centre sent back to the manufacturers and replaced, although no access was found;
• Jira tickets and source code searched for stored secrets; HAR files uploaded to the wiki deleted.
That is the price list for "we think that one is dead".
A rotation checklist that starts from an inventory
Rotation fails in one common way: the list of what to rotate is rebuilt from memory during the incident. People remember their own passwords and the big API keys. They don't remember the connector someone set up in 2021.
1. Keep the list before you need it
One table, kept where the team will see it. For every non-human credential:
Build it from the systems, not from people's recollection: the service accounts, API tokens, OAuth apps and deploy keys pages of your identity provider, code host, cloud accounts, help desk, CRM and anything else with an "integrations" menu. Each of those pages has an export or at least a list.
The rows with "none" under Owner and "?" under How to rotate are the Cloudflare four. Find them on a quiet day.
2. "Unused" means deleted
A credential you believe is unused has two honest states: proven in use, or revoked. Check its last-used date (most systems show one). If there is none, or it is older than 90 days, disable it and wait for someone to complain. Nobody complaining for a month is your proof; then delete it. "Probably unused, leave it" is not a state.
3. When a vendor is breached, rotate by scope, not by guess
Ask what the vendor held or could see, and rotate that whole set:
1. Everything stored in or passed through the vendor (for a support system: anything ever pasted or uploaded into a ticket).
2. Every credential the vendor's product used to reach your systems.
3. Every session it could have minted (sign everyone out; reset the integration's sessions).
Work down the inventory rows where "Lives in" or "Used by" names that vendor. Tick each row with a time and a name. A row can't be skipped because someone is sure it's dead; see step 2.
4. Check that the old value stopped working
A new credential issued is not an old credential revoked. Some systems keep both valid for a grace period, some forever unless you delete the old one. For each row, try the old value once and expect a refusal.
5. Give integrations small permissions and an end date
• The Smartsheet account had Jira administrator rights. A connector that reads tickets needs read access to some projects.
• Where the system allows it, restrict an integration's token to the vendor's published IP ranges.
• Set an expiry on every token that supports one. A yearly forced renewal is also a yearly check that the thing is still needed.
6. Alert on the three changes intruders make
New user created outside your normal joiner process; any account added to an admin group; a new plugin, app or API token on a core system. Most identity providers and Atlassian, GitHub and Google Workspace admin consoles can email on these. That is the alert that ended this incident.
The lesson
Cloudflare did the hard parts well: hardware keys on the identity provider, a segmented network, an alert that worked, and a report that hid nothing. It still lost five weeks of engineering to four rows missing from a list. The fix is clerical. Know every credential that isn't a person, who owns it and how to kill it, before the next vendor writes to say they've had an incident. The Snowflake customer breaches show the same failure with ordinary passwords: credentials stolen years earlier still worked because nothing had ever made them change.
osec.one makes small security tools for sites without a security team: passwordless sign-in, login rules, scans and CSP reports, each added with a pasted snippet and no DNS move.
See the tools