osec.one cover: most AI crawlers on our server were fake

The short version

• We run about twenty small sites on one server. For 14 days (3 to 17 October 2026) we checked every request that called itself GPTBot, ClaudeBot, PerplexityBot, Googlebot or one of six other crawler names against the address list the company behind that name publishes.
• 63,603 requests carried one of those ten names. 50,647 of them (80%) came from addresses that aren't on the company's list. For the eight AI names alone it was 73%.
• Not one request in the name of PerplexityBot, Perplexity-User or Claude-SearchBot was real. 3,159 said ChatGPT-User; 6 were.
• The fakes weren't reading our pages. 71% asked for things like /.env, /.git/config, /terraform.tfstate and private SSH keys.
• 207 addresses sent all of them, and 177 of those wore nine or ten different crawler names in turn. It's one crowd changing hats, not ten different bots.
• If your bot rules wave through anything that says GPTBot, they wave through this crowd. Check the address, not the name.

Why we looked

We wanted to know whether ChatGPT, Claude or Perplexity ever read our pages for a real person. Counting that sounded easy: grep the access log for ChatGPT-User. The first count said 2,888 such requests in two weeks, which would have been great news. Then we looked at what those requests asked for. Almost none were pages. They were /.env.production and /admin/.env.

So we stopped trusting the user agent and started checking where each request came from. OpenAI, Anthropic, Perplexity, Google and Microsoft all publish the address ranges their crawlers use, as plain JSON files, exactly so site owners can tell the real ones from impostors.

How we counted

1. Read the nginx access logs on our server for 3 October 00:00 to 17 October 13:00 UTC: 981,484 lines across all our sites.
2. Pick out every request whose user agent contains one of ten crawler names. When a user agent holds two (every Claude agent also carries claudebot@anthropic.com), the first name in the string wins.
3. Download each company's published list (links at the end) and test the request's address against the list for that crawler.
4. On the list: real. Not on it: fake. All ten names have a published list.
5. Print only totals. No address leaves the server in this write-up.

The lists are the ones published on 14 October, and they change now and then, so a crawler that moved address during the fortnight could be counted as fake. The gap is far too large for that to matter.

The numbers, name by name

text
Name               Claimed    Real   Real share
Googlebot           26,437   3,131    12%
Claude-User          6,846   3,592    52%
ClaudeBot            6,642   3,460    52%
GPTBot               5,266   2,021    38%
OAI-SearchBot        3,811     545    14%
PerplexityBot        3,167       0     0%
ChatGPT-User         3,159       6    0.2%
Perplexity-User      3,117       0     0%
Claude-SearchBot     3,091       0     0%
bingbot              2,067     201    10%
All ten             63,603  12,956    20%

Three things stand out.
• Googlebot is the most faked name by far. Seven out of eight requests in its name weren't Google. That's an old trick, and it still works on sites that allowlist Googlebot so their search ranking doesn't suffer.
• The AI names are now faked just as evenly. PerplexityBot, Perplexity-User, Claude-SearchBot and ChatGPT-User each got about 3,100 fake requests, almost the same number. That evenness is a clue: one tool is cycling through a list.
• The real ones behave. Real GPTBot requests got a normal answer 99% of the time and read pages and sitemaps, nothing else. Real ClaudeBot spent most of its time on robots.txt and sitemaps.

Diagram: a rented cloud machine borrows a crawler's name, the site trusts the name, the request asks for .env and similar files, and a readable .env leaks database and API keys
What the fake crawlers were trying to do, step by step.

One crowd, ten hats

All 50,647 fake requests came from 207 addresses. 88 of them used nine of the crawler names and 89 used all ten, so 177 of the 207 rotated through the whole list. The ten busiest addresses sent 65% of the fake requests, and the busiest one alone sent 22%.

The same 207 addresses also sent 419,171 requests that carried no crawler name at all: ordinary-looking browser strings, empty ones, scripting libraries. That's 43% of every line in our logs for the fortnight. Wearing GPTBot is just one costume they try.

The volume swings a lot from day to day: 486 fake crawler requests on 15 October, 8,871 on 11 October. It comes in bursts, not a steady drip, which fits a scanner working through a long list of targets and coming back to each one now and then.

What the fakes asked for

35,840 of the fake requests (71%) went for files that should never be on a public web server. The top of the list, in order:

text
Requests  Path
     639  /.env
     501  /.env.local
     460  /@fs/.env
     454  /__vite_rsc_findSourceMapURL
     357  /.env.production
     241  /.env.development
     236  /@fs/app/.env
     218  /@fs/../.env
     163  /.env.prod
     151  /@fs/var/run/secrets/kubernetes.io/serviceaccount/token
     150  /.htpasswd
     149  /secrets.json
     145  /.git/config
     145  /credentials.json
     144  /terraform.tfstate
     143  /.ssh/id_ed25519

• /.env in every flavour. A framework's environment file holds the database password, the payment key and the mail key. One readable .env is usually the whole company.
• /@fs/... and /__vite_rsc_.... These are routes of the Vite development server, not of a built site. Vite's /@fs/ route serves files from disk, and it had several bypass bugs in 2025 that let it read files outside the project. The scanners are betting someone left vite dev facing the internet. The Kubernetes service-account token path is the same bet one step further: read the token, then talk to the cluster.
• terraform.tfstate, .git/config, SSH keys. Deployment leftovers. A Terraform state file can hold cloud credentials in plain text.
• The rest (29%) were home pages, /userfiles, a few admin consoles and file-read APIs of popular self-hosted apps.

Almost none of it worked here. 40,049 fake requests got a 503 and 9,271 a 404. 1,078 (2%) got a 200, and a few hundred of those were for /.env itself, which made us sit up. We checked each site that answered: they were single-page apps that send their own HTML shell for any address, so the “200” was a web page, not a file. Harmless, but a scanner can't tell the difference without reading the body, so it comes back. And every one of those paths does exist on some site, which is why the list keeps growing.

What the real assistants read

The part we were after in the first place. In 14 days, real ChatGPT-User (a person's chat fetching a page live) came six times: home pages three times, an article on how big to print a QR code twice, and one health news item. Real Claude-User came 3,592 times, but only 245 of those got a normal answer, and those were robots.txt and image files of one of our apps, not articles. Real Perplexity-User: never.

Six honest visits are worth more to us than 3,153 fake ones, but only if we can tell them apart. With the name alone we would have reported 3,159.

Check your own logs in five minutes

1. Download the lists you care about (the URLs are in the sources below). Each one is JSON with a prefixes array of ipv4Prefix and ipv6Prefix entries.
2. Pull the requests that claim the name, with their addresses.
3. Test each address against the list. Python's ipaddress module does it in a few lines.

python
import ipaddress, json, urllib.request
from collections import Counter

url = "https://openai.com/gptbot.json"
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in json.load(urllib.request.urlopen(url))["prefixes"]]

seen = Counter()
for name in ["/var/log/nginx/access.log", "/var/log/nginx/access.log.1"]:
    for line in open(name, errors="replace"):
        if "GPTBot" not in line:
            continue
        ip = ipaddress.ip_address(line.split(" ", 1)[0])
        seen["real" if any(ip in n for n in nets if n.version == ip.version) else "fake"] += 1
print(seen)

Swap in searchbot.json, chatgpt-user.json, Anthropic's bots.json or Google's googlebot.json for the other names. On a busy server, expect the fake count to be the bigger number.

Let the real ones in, and only them

If you allowlist AI crawlers in nginx, tie the rule to the address list, not to the user agent. A geo block reads the ranges from a file, a map combines them with the name, and anything that claims GPTBot from elsewhere is closed without an answer:

nginx
# http {} context, e.g. /etc/nginx/conf.d/gptbot.conf
geo $gptbot_ip {
    default 0;
    include /etc/nginx/ranges/gptbot.conf;   # lines like: 192.0.2.0/24 1;
}
map "$http_user_agent|$gptbot_ip" $fake_gptbot {
    default 0;
    "~*gptbot.*\|0
#34; 1;
}

# in each server {} block
if ($fake_gptbot) { return 444; }
bash
# refresh the ranges once a day (cron), then reload only if nginx -t passes
curl -fsS https://openai.com/gptbot.json \
  | jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' > /etc/nginx/ranges/gptbot.conf.new \
  && mv /etc/nginx/ranges/gptbot.conf.new /etc/nginx/ranges/gptbot.conf \
  && nginx -t && systemctl reload nginx

• Repeat the pair for each name you allow. If the download fails, the && chain keeps yesterday's file.
• Do the rest regardless of names. Nothing that starts with a dot should be served, vite dev should never face the internet, and .git and terraform.tfstate don't belong in a web root. Our earlier write-ups cover closing probe paths with return 444 and banning repeat knockers with fail2ban.
• Not sure whether your site serves one of those files today? osec Scan checks a URL for exposed files, headers and TLS in about ten seconds.

What we're changing

• Our daily traffic numbers count an AI visit only when its address is on the company's list. Requests that just wear the name are counted separately as fakes and never mixed into the totals.
• We stopped treating any user agent as a reason to relax a rule.
• We'll rerun this count in a month and add the result here.

Sources

• Our own nginx access logs, 3 to 17 October 2026, read on the server; totals only.
• OpenAI crawler ranges: https://openai.com/gptbot.json, https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json
• Anthropic crawler ranges: https://claude.com/crawling/bots.json, plus Anthropic's outbound range from https://platform.claude.com/docs/en/api/ip-addresses
• Perplexity: https://www.perplexity.com/perplexitybot.json, https://www.perplexity.com/perplexity-user.json
• Google: https://developers.google.com/static/search/apis/ipranges/googlebot.json
• Microsoft: https://www.bing.com/toolbox/bingbot.json
• nginx geo and map modules: https://nginx.org/en/docs/http/ngx_http_geo_module.html, https://nginx.org/en/docs/http/ngx_http_map_module.html

Bot Gate lets real crawlers and people through and stops the rest with a snippet on your own server. No DNS move.

See Bot Gate