Bot Fight Mode

Bot Fight Mode keeps automated traffic off a site: crawlers you did not invite, vulnerability scanners, scripts, headless browsers, and the tools that request /.env and /wp-login.php from every server on the internet around the clock. It judges each request by its User-Agent, its address, its URL and its behaviour.

Turn it on in the site’s settings: Advanced tab, Bot Fight Mode. Blocked requests get a 404 (or your own block page).

Bot categories

A User-Agent with generic crawler words (“SomethingBot/1.0”, “crawler”, “spider”, a +https:// contact link) that matches no known name counts as an unknown bot. It is blocked when either AI crawlers or SEO crawlers are blocked.

Search engines and link previews are off by default because blocking them has visible costs. Without search engine indexers the site drops out of search results. Without link preview bots, a link to your site shared in a messenger shows no title or picture. Turn them on only for sites that should stay out of sight, such as staging copies.

Monitoring and feed readers are never matched

Uptime monitors and feed readers are usually services you signed up for yourself, and many of them send bot-like User-Agents. UptimeRobot, Pingdom, StatusCake, Better Uptime (Better Stack), Feedly, Inoreader, NewsBlur and Feedbin fall under no category. To block one of them anyway, put its name in your own list (below).

Your own list

Also block User-Agents containing: one entry per line, case-insensitive, up to 200 entries. Any User-Agent that contains one of them is blocked. This list is checked before every other User-Agent rule, so it also catches monitors and bots you have allowed by category.

Strict browser check

Off by default. When on, a request whose User-Agent matches no known bot must look like a real browser: it has to send Accept and Accept-Language headers and a User-Agent that starts with Mozilla/5.0. Every modern browser does; many scrapers copy a browser’s User-Agent and forget the other headers. The check can block rare old browsers and some unusual clients, so try it first on a site whose statistics you watch.

Known bots that you allow (for example search engines) skip this check.

Generated robots.txt

With Serve robots.txt that disallows blocked bots (on by default), a site that has no robots.txt of its own gets a generated one. It disallows the whole site for the AI crawlers, SEO crawlers and search engines you block, by their official robots.txt names, hides the honeypot path from everyone else and allows the rest. Crawlers that respect robots.txt then stop coming; the others are blocked anyway.

If your site has its own robots.txt, it is served as is and nothing is generated.

Probes for vulnerable files

Ban IPs probing for vulnerabilities (on by default) bans a visitor that asks for a path only an attacker would ask for, if that file does not exist on your site:

• secrets and repositories: /.env, /.git/..., /.svn/, /.aws/, /.ssh/, /.docker/, .sql, .bak, .old, .swp, .htpasswd, docker-compose.yml, credentials.yml;
• scripts scanners try: /wp-login.php, /wp-config.php, /xmlrpc.php, /phpinfo.php, /info.php, /shell.php, /cmd.php, /adminer.php, /c99.php, /wso.php, /filemanager.php and similar names;
• well-known application paths: /wp-admin/, /wp-json/, /phpmyadmin/, /cgi-bin/, /actuator, /server-status, /vendor/phpunit, /etc/passwd;
• path traversal with .. (a browser resolves .. itself, only tools send it).

A file that really exists is never a probe: if you have a /info.php, asking for it is fine.

A script extension alone is not enough to ban. A site that moved from PHP or another platform keeps getting visitors who follow old links to /contact.php, /index.php or /downloads/setup.sh, and banning them would lock out your own readers. Only the names that scanners actually try are matched; a scanner trying many other names is caught by scan detection.

Probes, attack payloads, fake crawlers and the honeypot share one ban time: Ban, minutes in Bot Fight Mode, 60 by default (0 to 10 080; 0 means refuse without a ban).

Non-GET requests

Block non-GET requests (on by default). A static site only ever needs GET, HEAD and OPTIONS. POST, PUT, DELETE, PROPFIND and other methods sent to a static site come from automated tools, so they are refused (no ban).

Sites with the PHP or Node.js handler take forms and API calls. There POST, PUT, PATCH and DELETE are allowed too, and only unusual methods such as PROPFIND or TRACE are refused.

Attack payloads in URLs

Block attack payloads in URLs (on by default) looks for SQL injection, cross-site scripting, path traversal, php:// wrappers, Log4Shell (${jndi:...}), template injection and shell commands in the path and in each query parameter. Encoded and double-encoded forms (%2e%2e, %252e) and SQL comments used as spaces (UNION/**/SELECT) are decoded before matching.

The rules have three levels, so that a real person is never banned for something they typed:

• An unmistakable attack gets a ban. That covers SQL syntax around the words (' or '1'='1, UNION SELECT after a quote, sleep(5) after a quote), <script>, an event handler that fires by itself (onerror=), ${jndi:, .. in the path, ../ reaching /etc or /proc, or three levels up at once (../../../).
• Something that looks like an attack but could come from a person is refused, and only that one request: a search for {{7*7}}, a link like ?next=../page, the text etc/passwd in a parameter, onclick=foo().
• Plain text matches no rule: words like select ... from in an article’s address, “Sleep (2009)”, {{name}} or jQuery’s $(...).

Search parameters (q, s, search, query, keyword, term, find and a few more) are treated more gently, because someone searching your documentation for php:// or <div onclick=...> is a reader. In a search parameter only an unmistakable exploit bans; shell one-liners and stacked SQL there are refused without a ban.

Limits that cannot be padded around

Every rule runs in linear time, so a crafted URL cannot slow the server down. The path and each query parameter are checked separately, up to 4 KB each. So that an attack cannot hide behind padding, a request is refused (without a ban) when:

• the path or one parameter is longer than 4 KB,
• it has more than 200 parameters,
• or the query string is longer than 64 KB.

Scan detection

Ban IPs scanning for files (on by default) catches directory brute-forcing tools such as gobuster, ffuf and dirbuster: a visitor that collects many 404 answers in a short time is banned. The default is 25 missing paths in 60 seconds (adjustable: 3 to 10 000 paths, 5 to 3600 seconds).

A missing image, stylesheet, script, font or source map counts as only 0.1 of a miss. A reader who opens an old page with 30 broken images adds up 3 misses, far below the threshold.

PHP and Node.js sites

On a site with a dynamic handler some paths that look like probes may be real routes: WordPress serves /wp-json/ and /wp-admin/ from its own code. So on PHP and Node.js sites:

• the well-known application paths (/wp-admin/, /wp-json/, /phpmyadmin/, /cgi-bin/ and the rest of that list) are not treated as probes. Secrets (/.env, /.git), scanner script names (/shell.php, /phpinfo.php) and traversal still are;
• forms and APIs (POST, PUT, PATCH, DELETE) are allowed, as described above;
• a path that .htaccess rules would serve counts as existing.

Scanning is still caught, because scan detection counts the 404 answers PHP returns.

Crawler verification and IP lists

Three more switches in Bot Fight Mode work by address instead of User-Agent: Verify crawlers by IP (on by default), Block those crawlers by IP too and Use the public IP blocklists. They are described in Verified crawlers and IP feeds. The Honeypot path has its own page: Honeypot.