Honeypot path

A honeypot is a path that nobody has a reason to visit. Well-behaved crawlers stay away because robots.txt disallows it, and people never open it because no page links to it. That leaves crawlers that ignore robots.txt and scrapers that read robots.txt to find the paths a site wants hidden, which are the bots you want gone.

Setting it up

In the site’s settings, Advanced tab, Bot Fight Mode:

1. Tick Honeypot path.
2. Enter the path in the Honeypot path field. The default is /secret-archive/. Pick something that sounds worth a look and does not exist on your site.
3. Make sure Serve robots.txt that disallows blocked bots is on, and save.

The path may contain letters, digits, /, _, . and -, up to 100 characters, and no ... A leading / is added if you leave it out.

What happens

• Any request whose path starts with the honeypot path, for example /secret-archive/ or /secret-archive/2019/backup.zip, is refused and the visitor is banned at once.
• The ban lasts the Bot Fight ban time (Ban, minutes, 60 by default) and counts as a probe in the site’s statistics.
• The check runs early, before the User-Agent rules, so a bot that imitates a browser perfectly is caught too.
• Addresses in the site’s allow list are never trapped.

robots.txt

The generated robots.txt (served only when your site has no robots.txt of its own) contains the line that hides the trap from honest crawlers:

User-agent: *
Disallow: /secret-archive/
Allow: /

If your site has its own robots.txt, WardPane serves it unchanged, and you need to add the Disallow: line for the honeypot path yourself. Without it, a legitimate search engine that somehow finds the path has no reason to avoid it and would be banned too.

Do not link to the honeypot from your pages, and do not use a path that a real section of your site starts with: every path under it is a trap.