Verified crawlers and IP feeds

A User-Agent filter is easy to get around: anyone can put Googlebot into their User-Agent, and scrapers often do, because many sites let Googlebot through. WardPane checks such requests against the address ranges that the crawler operators publish themselves.

Crawler verification by IP

These operators publish the address ranges their crawlers use:

WardPane downloads these lists once a day and keeps a copy on disk. With Verify crawlers by IP on (the default, in the site’s Bot Fight Mode), a request that uses one of these names from an address the operator does not own is a fake: it is refused and the address is banned for the Bot Fight ban time (60 minutes by default). This happens whether or not you allow that crawler on the site, since nobody ends up with a Googlebot User-Agent on a non-Google address by accident.

A real Googlebot from Google’s own addresses is not affected by this check; whether it gets in depends only on the “Search engine indexers” category.

When a list is out of date

If a list cannot be refreshed, the last good copy stays in use and the log records the failure. Once a list is more than a month old, verification for that operator is switched off until a fresh copy arrives. An old list may lack new ranges, and banning the real Googlebot because of it would do far more damage than letting one fake through. An operator whose list was never downloaded is not verified at all.

Downloaded lists are checked before use: entries that cannot be read, or that are broader than /8 (IPv4) or /19 (IPv6), are dropped, so a broken or tampered list cannot block half the internet.

Blocking crawlers by IP

Some crawlers come from their operator’s addresses but with an ordinary browser User-Agent. With Block those crawlers by IP too (off by default), requests from the published ranges are blocked even when the User-Agent looks like a browser.

The option covers only the categories you block on the site. If “AI crawlers” is blocked, every request from the crawler ranges of OpenAI, Anthropic and Perplexity is refused. Blocking “SEO crawlers” adds the Ahrefs ranges, and the ranges of Google, Bing, DuckDuckGo and Apple are blocked only if you block search engines.

Looking up an address

The panel-wide tools are under Advanced > Security in the main menu. In Whose crawler is this IP?, type an address and press Look up to see which operator’s published range it belongs to, if any, and whether it is on one of the enabled public blocklists. Known bot IP ranges lists every operator’s ranges with their count and the date of the last update, and has an Update now button.

For the country and network of an address, use the lookup in Geo-blocking.

Public IP blocklists

Separately from the crawler checks, WardPane can use public lists of known malicious addresses:

They are off by default, and turning one on takes two steps:

1. Enable the feeds for the server. In Settings > Integrations, tick the feeds you want, or list them in /opt/wardpane/.env and restart the panel:

   IP_FEEDS=spamhausDrop,fireholLevel1
   

   sudo systemctl restart wardpane
   

   The panel downloads each feed once for all sites and refreshes it daily. A download with fewer than 10 usable entries is rejected, and the previous copy stays in use. A feed you turn off stops applying.

2. Turn them on for a site. In the site’s Bot Fight Mode, tick Use the public IP blocklists. This is set separately for every site.

A request from a listed address is refused and counted as “blocklisted” in the site’s statistics. There is no ban on top of that, because the list already catches every request from that address. The Public IP blocklists card in Advanced > Security shows each feed with its size, its last update and whether it is enabled.

Feeds and the crawler checks need the real visitor address. If a site is behind Cloudflare, turn on Behind Cloudflare; without it these checks are skipped for requests that arrive from Cloudflare’s addresses.