Close probe paths and bad user agents in nginx with return 444

A 404 is still a response. The scanner reads it, your app may render a page for it, and your log gets a line. nginx's return 444 closes the connection without sending anything. The request never reaches the application, and the scanner learns nothing from it.

Map the bad user agents

Put the user-agent map in the http context. The default 0 line keeps everything else allowed, and the empty user agent is flagged because real browsers always send one.

nginx

# /etc/nginx/conf.d/bot-filters.conf
map $http_user_agent $bad_agent {
    default       0;
    ~*sqlmap      1;
    ~*nikto       1;
    ~*masscan     1;
    ~*zgrab       1;
    ""            1;
}

Match the probe paths in the server block

Dotfiles outside /.well-known/ and the usual admin paths get closed. The if block only returns, which is the one form of if nginx documents as safe.

nginx

server {
    listen 443 ssl;
    server_name example.com;

    # dotfiles, except the well-known paths used by certificates and standards
    location ~* /\.(?!well-known) {
        return 444;
    }

    # admin and CMS probes for software this site doesn't run
    location ~* (wp-login\.php|xmlrpc\.php|phpmyadmin|/vendor/phpunit) {
        return 444;
    }

    if ($bad_agent) {
        return 444;
    }

    location / {
        proxy_pass http://127.0.0.1:8020;
    }
}

Test the config and reload

bash

sudo nginx -t && sudo systemctl reload nginx

Check that the probes are closed

curl reports 000 when the connection closes without a response, which is what you want for a probe. The Python check below runs the same probes and reports what each one got back.

python

import urllib.request
from urllib.error import HTTPError, URLError

BASE = "https://example.com"
PROBES = ["/.env", "/.git/config", "/wp-login.php", "/xmlrpc.php", "/phpmyadmin/"]


def probe(path: str) -> str:
    try:
        with urllib.request.urlopen(BASE + path, timeout=5) as resp:
            return str(resp.status)
    except HTTPError as err:
        return str(err.code)
    except URLError as err:
        return f"closed ({err.reason})"


if __name__ == "__main__":
    for path in PROBES:
        print(f"{path:18} {probe(path)}")

Don't block the crawlers you want

Search engines and uptime checks send their own user agents. Before you add a pattern to the map, check the log for the exact string, and keep the list short. A deny list that catches a real crawler costs more than the probes it stops.

For the scrapers that ignore robots.txt, osec Bot Gate adds one more layer without moving your DNS.

Flow: probe, match, return 444, no app work, verify, keep real bots — Matched probes close the connection with no response.

Sources

• nginx rewrite module (return, including 444): https://nginx.org/en/docs/http/ngx_http_rewrite_module.html
• nginx map module: https://nginx.org/en/docs/http/ngx_http_map_module.html