From: prx <prx@si3t.ch>
Subject: Re: httpd: add header block/drop rules for request filtering
To: Rafael Sadowski <rafael@sizeofvoid.org>
Cc: tech@openbsd.org
Date: Mon, 21 Sep 2026 20:14:58 +0200

* Rafael Sadowski <rafael@sizeofvoid.org> le [21-09-2026 16:02:08 +0200]:
> Hi tech@
> 
> during EuroBSDCon26, Purple Rain (secbsd.com) came up to me and showed
> me his httpd diff. The idea was to block Ai scrapers by the user-agents
> header because his site had crashed under the load.
> 
> My answer was that you can also do this with relayd(8) in front of
> http(8). However, I understand that not everyone wants to run relayd(8)
> for a single httpd.
> 
> His idea was to do this for user agents. I incorporated the whole thing
> into our "header" syntax and made it generic. Now you can block (with an
> HTTP status code) or drop anything request you want based on key/value
> header pairs.
> 
> Here are a few use cases:
> 
> 1.) Drop all the Ai scrapers
> 
> $ cat ai_scrapers.conf
> # BLOCK AI CRAWLERS AND TRAINING
> 
> header drop "user-agent" "*bot*"
> header block "user-agent" "addsearchbot*" 403
> header block "user-agent" "agenttimes*" 403
> header block "user-agent" "ai2bot*" 403
> header block "user-agent" "aihitbot*" 403
> header block "user-agent" "aiwebindex*" 403
> header block "user-agent" "amazon*" 403
> header block "user-agent" "amzn*" 403
> header block "user-agent" "andibot*" 403
> header block "user-agent" "anomura*" 403
> header drop "user-agent" "anthropic*"
> header block "user-agent" "apify*" 403
> header block "user-agent" "applebot*" 403
> header block "user-agent" "aranet*" 403
> header block "user-agent" "atlassian-bot*" 403
> header block "user-agent" "awario*" 403
> header block "user-agent" "azureai*" 403
> ...
> 
> server "default" {
> 	listen on * port 80
> 	# block Ai and crawlers
> 	include "/etc/ai_scrapers.conf"
>  	location "/*" {
> 		root "/htdocs/localhost"
> 	}
> }
> 
> 2.) block with redirect
> 
> header block "user-agent" "amazon*" 301 "https://amazon.com"
> 
> 2.) block with message
> 
> header block "user-agent" "amazon*" 404 "bye bye my love"
> 
> I'm not sure if we want this in 8.0 or if we should wait until after the
> release. Of course, it would be useful to have it in the release.
> 

Great idea.
I like the redirect and the little message too :)

FWIW, I was running this script on my logs to add scrapers to a pf
table:
https://sr.ht/~prx/ban-ai/

It seems much more efficient with httpd directly. Thanks!