A CI service is a vault you don't think of as one. To build, test and deploy, it holds your cloud keys, your package-registry tokens, your database passwords and an OAuth grant to read your code. You typed them in once, into a project's environment variables, and forgot them.
On 4 January 2023 CircleCI emailed every customer: rotate any and all secrets stored in CircleCI. Its incident report a week later explained why. An intruder had been inside its production systems for several days in December and had copied customers' environment variables, tokens and keys, along with the encryption keys that protected them. The way in was not a password. It was a browser session cookie from one engineer's laptop.
Step 1: one laptop, one session
On 16 December 2022 malware landed on a CircleCI engineer's laptop. CircleCI's antivirus did not detect it. The malware took the engineer's single sign-on session cookie, a session that had already been through two-factor authentication, and the intruder used it from somewhere else. To the sign-on system, it looked like the engineer still at work.
This is the same trick behind many breaches on this site: the session cookie theft after MFA piece covers why a stolen cookie skips every factor, including passkeys, and LastPass 2022 also started with one engineer's computer.
Step 2: the engineer could create production tokens
The engineer's normal job included generating production access tokens. So did the intruder's, from that moment. On 19 December they looked around; on 22 December they copied data from a subset of databases and stores, including customers' environment variables, tokens and keys.
Those secrets were encrypted at rest. It didn't help: the intruder pulled the encryption keys from the memory of a running process, which is where a service has to keep them to use them. Encryption at rest protects a stolen disk, not a stolen identity that the service trusts.
Step 3: a customer noticed first
On 29 December a customer told CircleCI about suspicious activity on their GitHub OAuth token. CircleCI confirmed the token had been compromised on 30 December, started rotating every customer's GitHub OAuth token on 31 December, and found the intruder's path back to the laptop. Then, on 4 January:
• 16:35 UTC: the engineer's access shut off.
• 18:30: production access cut for nearly all employees.
• 22:30: every potentially exposed production host rotated.
• Evening (US time): the email to all customers. Over the next three days CircleCI revoked all project API tokens and older personal API tokens, finished rotating Bitbucket and GitHub tokens, and worked with AWS to warn customers whose AWS keys might be exposed.
CircleCI said fewer than five customers told it of unauthorized access to their own systems as a result. Everyone else had to assume they could be next, which is the expensive part.
Step 4: the rotate-everything day, from the customer side
Rotating one key takes minutes. Rotating every key stored in a CI system turned out to take weeks for many teams, because nobody had a list. The UK Ministry of Justice's cloud platform team published its own incident record: they found service accounts that could read every secret in a namespace, some shared across namespaces, and rotated in five phases. They declared the incident on 5 January and closed it on 1 February 2023, almost four weeks later.
The slow parts were the same everywhere:
• Finding the secrets. Project variables, contexts, SSH keys, tokens pasted years ago by someone who has left.
• Finding where each one is used. A database password may be in CI, in the app's config and on a developer's laptop.
• Rotating without breaking production. Some keys can't overlap, so the new one has to go everywhere at once.
• Checking for misuse. Which of these was used from an address that isn't yours, between 21 December and 4 January?
Make the next one short
You can't stop your CI provider from being breached. You can make it a two-hour job instead of a four-week one.
1. Stop storing cloud keys in CI at all. Use OIDC: GitHub Actions, GitLab and CircleCI can each present a short-lived signed token that AWS, Google Cloud or Azure exchange for credentials valid for one job. Nothing long-lived sits in the CI system to steal. CircleCI listed this first in its own advice.
2. Keep a secrets inventory in the repository: name, where it's stored, what it unlocks, who owns it, how to rotate it. One line each. If you can't write the rotation step, you'll find out on the bad day.
3. Scope each secret to one job. A deploy key that can only deploy one service, a registry token that can only publish one package, and CI contexts restricted to the branches or groups that need them. The Ministry of Justice's write-up names broad, shared service accounts as what widened the blast radius.
4. Rotate one thing on a schedule so the process stays practised: quarterly is enough for a small team. A rotation you've done before takes minutes.
5. Know where to look for misuse: cloud audit logs (CloudTrail and the like) filtered by access key, the registry's publish history, GitHub's audit log for the OAuth app. Save the queries now.
6. Lock CI identities to CI. Restrict tokens by source IP where the service allows it, and alert when a deploy credential is used from anywhere else.
7. Treat engineers' browser sessions as production credentials. Short session lifetimes for admin tools, re-authentication with a security key for production access (CircleCI added step-up authentication afterwards), and managed devices for anyone who can mint tokens.
8. Check what your workflows let outsiders do. A pull request that can run code next to your secrets makes rotation day come sooner; the free Workflow risk check reads a public repository's workflows and flags pull_request_target checkouts, injected PR text, missing permissions: blocks and unpinned actions.
Two more CI stories on this site cover the other ways in: Codecov 2021, where a tampered script read secrets inside customers' own builds, and Cloudflare's Thanksgiving 2023 intrusion, where four credentials missed in a rotation were enough.
The workflow files in your repository decide what a CI job can reach. osec's Action pin lookup turns uses: lines into commit-pinned ones, so a moved tag can't change what runs next to your secrets. Free, paste your lines.
Sources: CircleCI: Incident report for January 4, 2023 security incident; CircleCI community: Security alert, rotate any secrets stored in CircleCI (FAQ); UK Ministry of Justice Cloud Platform: 2023-01-05 CircleCI security incident; The Hacker News: CircleCI urges customers to rotate secrets; GitHub Docs: About security hardening with OpenID Connect; CircleCI Docs: Using OpenID Connect tokens in jobs.