osec Bot Gate cover: AI scrapers on small sites

The problem

• AI training crawlers walk every page, every revision, every filter combination.
• Many ignore robots.txt and rotate through residential IPs.
• Small servers fall over: wikis, forums, Git forges, shops with faceted search.
• The usual fix is moving your DNS behind a big proxy. Not everyone can or wants to.

What others did

• Anubis (open source, January 2025): a proof-of-work wall in front of the site. Adopted by GNOME, FFmpeg, Wine, the Linux kernel mailing list archives and many more.
• It works, mostly: Codeberg reported in August 2025 that some scrapers learned to solve it, but it still blocked the bulk.
• Lesson: proof-of-work doesn't make scraping impossible; it makes it expensive.

How osec Bot Gate works

1. A visitor without a gate cookie asks for a page.
2. Your middleware answers with a tiny "Checking your browser…" page.
3. The browser solves one osec Captcha puzzle (about a quarter second).
4. Your server verifies it with osec.one once, then sets a signed cookie for 7 days.
5. Every later page view is checked locally against the cookie. No call to us.

osec Bot Gate check page: Checking your browser, This takes a second and happens once a week
What a new visitor sees for about a second, once a week.

Add it

1. Sign in at osec.one → Console → Site keys → New key. Keep the secret in an environment variable.
2. Download the middleware: osec-gate.js (Node), osec_gate.py (Python), osec-gate.php (PHP, WordPress).
3. Node: app.use(osecGate({ sitekey, secret })).
4. FastAPI: app.add_middleware(OsecGateASGI, sitekey=…, secret=…). Flask: wrap app.wsgi_app.
5. WordPress: two lines at the top of wp-config.php.
6. Deploy and open your site in a private window.

osec.one console Site keys page with Bot Gate snippets for Node.js, Python and PHP
Console → Site keys → Snippets → Bot Gate.

What passes without a check

• Search engines and link previews: Google, Bing, DuckDuckGo, Apple, Yandex, Baidu, Facebook, LinkedIn, X, Slack, Discord.
• Turn on verifyCrawlers to confirm them by reverse DNS (user agents can be faked).
• APIs, assets, feeds and form posts: only HTML page views are gated. Protect forms with osec Captcha.
• If osec.one is unreachable or your quota is used up, visitors get through for an hour (fail open, on by default).

Good to know

• Cost: one request per visitor per week. 500 a day free; credits when you grow.
• The cookie is signed with your secret and tied to the browser's user agent.
• No proxy, no DNS change: your traffic never passes through osec.one.
• Honest limit: it raises the cost of scraping; scrapers running full browsers can still pay it.

A two-line bot gate for Node, Python and PHP. No DNS move.

Set up osec Bot Gate

Sources

• LWN.net, Anubis and AI scrapers: https://lwn.net/Articles/1028558/
• Ben Tasker, deploying Anubis to block AI bots: https://www.bentasker.co.uk/posts/blog/the-internet/deploying-anubis-to-block-ai-bots.html
• Google, verifying Googlebot by reverse DNS: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot