If the goal is to defend against scrapers, running httpd naked is probably
a scenario for failure. As stated, this can be done with relayd already
in base, do we really want to pull all kitchen sink functions into httpd?

One feature looks like it's missing is a logging/tagging function. E.g.
the following is a relayd rule I use:

block request quick header log "User-Agent" value "*cohere-ai/*" tag "AI_USER_AGENT"

I see more value in logging/analyzing the traffic than 301 to goatse.

I would more prefer a connection to pf where a dynamic blacklist table
can be updated by relayd/httpd directly. Maybe it exists already, but
for me it's a manual process after log parsing.

The second worst scrapers, of course, are using a Chrome user-agent.

The worst scrapers are spewing random data into the user-agent header.

I'm not convinced it's malicious as much as it's shitty bot coding.

I hope all commits parsing headers from clients are being fuzzed.

Regards
Lloyd

Kirill A. Korinsky <kirill@korins.ky> wrote:

> On Mon, 21 Sep 2026 16:02:08 +0200,
> Rafael Sadowski <rafael@sizeofvoid.org> wrote:
> >
> > Hi tech@
> >
> > during EuroBSDCon26, Purple Rain (secbsd.com) came up to me and showed
> > me his httpd diff. The idea was to block Ai scrapers by the user-agents
> > header because his site had crashed under the load.
> >
> > My answer was that you can also do this with relayd(8) in front of
> > http(8). However, I understand that not everyone wants to run relayd(8)
> > for a single httpd.
> >
> > His idea was to do this for user agents. I incorporated the whole thing
> > into our "header" syntax and made it generic. Now you can block (with an
> > HTTP status code) or drop anything request you want based on key/value
> > header pairs.
> >
> > Here are a few use cases:
> >
> > 1.) Drop all the Ai scrapers
> >
> > $ cat ai_scrapers.conf
> > # BLOCK AI CRAWLERS AND TRAINING
> >
> > header drop "user-agent" "*bot*"
> > header block "user-agent" "addsearchbot*" 403
> > header block "user-agent" "agenttimes*" 403
> > header block "user-agent" "ai2bot*" 403
> > header block "user-agent" "aihitbot*" 403
> > header block "user-agent" "aiwebindex*" 403
> > header block "user-agent" "amazon*" 403
> > header block "user-agent" "amzn*" 403
> > header block "user-agent" "andibot*" 403
> > header block "user-agent" "anomura*" 403
> > header drop "user-agent" "anthropic*"
> > header block "user-agent" "apify*" 403
> > header block "user-agent" "applebot*" 403
> > header block "user-agent" "aranet*" 403
> > header block "user-agent" "atlassian-bot*" 403
> > header block "user-agent" "awario*" 403
> > header block "user-agent" "azureai*" 403
> > ...
> >
> > server "default" {
> > 	listen on * port 80
> > 	# block Ai and crawlers
> > 	include "/etc/ai_scrapers.conf"
> >  	location "/*" {
> > 		root "/htdocs/localhost"
> > 	}
> > }
> >
> > 2.) block with redirect
> >
> > header block "user-agent" "amazon*" 301 "https://amazon.com"
> >
> > 2.) block with message
> >
> > header block "user-agent" "amazon*" 404 "bye bye my love"
> >
> > I'm not sure if we want this in 8.0 or if we should wait until after the
> > release. Of course, it would be useful to have it in the release.
> >
> > Purple Rain tested this diff in production. (Thanks)
> >
> > Feedback welcome.
> >
> 
> It reads interesting not sure how it can be used against AI crawlers,
> because shity one never respects User Agent, and sane one moves to proove
> who they are by using Web Bot Auth.
> 
> Anyway, I don't object from this feature, and actually it can be useful for someone.