A Python report of bot pings, from your own nginx access log

Blocking without measuring usually means blocking the wrong thing. A few lines of Python over the access log tell you which addresses send the most requests, which of them probe for files that don't exist, and which user agents they claim to be.

Know the log format

The default nginx combined format writes one line per request with the client address, time, request, status, bytes, referer and user agent. The script reads the fields it needs and ignores the rest.

The report script

Save it as nginx_bot_report.py. It takes the log path as an argument and reads the default path when none is given.

python

#!/usr/bin/env python3
"""Summarise bot probes in an nginx combined access log."""
import re
import sys
from collections import Counter
from pathlib import Path

LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[[^\]]+\] '
    r'"(?P<method>\S+) (?P<path>\S+) [^"]*" '
    r'(?P<status>\d{3}) \S+ "[^"]*" "(?P<agent>[^"]*)"'
)
PROBE = re.compile(r"/(\.env|\.git|\.aws|wp-login\.php|xmlrpc\.php|phpmyadmin|vendor/phpunit)", re.I)


def main(log_path: str) -> None:
    requests: Counter = Counter()
    probes: Counter = Counter()
    for line in Path(log_path).read_text(encoding="utf-8", errors="replace").splitlines():
        match = LINE.match(line)
        if not match:
            continue
        ip = match["ip"]
        requests[ip] += 1
        if PROBE.search(match["path"]) or match["status"] in ("403", "404"):
            probes[ip] += 1

    print(f"{'requests':>8}  {'probes':>6}  ip")
    for ip, total in requests.most_common(10):
        print(f"{total:8}  {probes[ip]:6}  {ip}")


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "/var/log/nginx/access.log")

Run it

bash

python3 nginx_bot_report.py /var/log/nginx/access.log

Find the most requested missing paths

A 404 count by path shows what the scanners are after. Field 9 is the status and field 7 the path in the combined format.

bash

sudo awk '$9 == 404 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -15

Turn the numbers into action

• An address with many requests and almost all probes: ban it, or add it to a deny list.
• An address with many requests and no probes: rate limit it, it may be a legitimate crawler.
• A path that every scanner asks for: block it at nginx, as in the 444 article.

Run the report from cron each morning and keep the output. A bot that appears on Monday and is gone by Friday is easier to spot in a list than in a screen of raw log lines.

Forms are the next thing bots reach for. osec Captcha adds a private, invisible check without moving your DNS.

Flow: access log, parse, count, rank, act, repeat — Measure before you block.

Sources

• nginx log module (log_format): https://nginx.org/en/docs/http/ngx_http_log_module.html
• Python re module: https://docs.python.org/3/library/re.html