Hello,

I've tested a bit more. I have the problem but only if i load configuration second time.

(16:31:08):~
# pfctl -f /etc/pf.conf                                                                                                                                                                                                                                                                 
(16:31:11):~
# pfctl -f /etc/pf.conf 
pfctl: Current pool size exceeds requested anchors limit 512
pfctl: Current pool size exceeds requested anchors limit 512
pfctl: Current pool size exceeds requested anchors limit 512

# grep 'set limit' /etc/pf.conf                                                                                                                                                                                                                                                         
set limit states 700000
set limit src-nodes 500000
set limit tables 2000
set limit anchors 2000


In pf.conf I have

anchor "vpn_users" on { tun0 tun1 }
load anchor vpn_users from "/etc/anchors/vpn_users.conf"

And in vpn_users.conf I have:

anchor "user1" in from X.X.X.X
load anchor user1 from "/etc/anchors/vpn_users/user1"
anchor "user2" in from X.X.X.X
load anchor user1 from "/etc/anchors/vpn_users/user2"
(...).

And in some of the user1 files there are more anchors loaded.

Like in: /etc/anchors/vpn_users/user1
anchor "project1" in from X.X.X.X
load anchor project1 from "/etc/anchors/vpn_users/projects/project1"

At total I have:
# pfctl -vsA |wc -l
     863

Maybe this is the problem that configuration is loaded from multiple nested files?


I think I'm using correct kernel and pfctl. Earalier I used kernel/pfctl from 7.3 where 
this problem appeared for the first time. Now I've tested this on 7.7 + your patch.
I have and I had current version of CVS checked out like like described in 
https://www.openbsd.org/faq/faq5.html

On 7.7 with your patch I think I do not have problem with interruptiong load of configuration
but rather I have problem with queues. Whene those four directives with queues are present in
configuration I have:

# pfctl -f /etc/pf.conf 
pfctl: DIOCADDQUEUE: Operation not supported by device
 

Additionaly I think that I may have found another problem (or feature). I've 
commented-out the two directives of creating and loading the verry first anchor in 
main configuration file. So no other files were processed. Despite that all anchors 
previsusly loaded anchors remined in memory after reload using
pfctl -f /etc/pf.conf. Shoudn't be all the anchors removed from the memory
in that case?


----- Original Message -----
From: "Alexandr Nedvedicky" <sashan@fastmail.net>
To: "Rafał Ramocki" <rafal.ramocki@eo.pl>
Cc: "tech" <tech@openbsd.org>
Sent: Tuesday, May 6, 2025 6:58:13 AM
Subject: Re: Error when at startup more than 512 anchors are loaded from pf.conf

Hello,


On Mon, May 05, 2025 at 03:28:39PM +0200, Rafa?? Ramocki wrote:
> Hello,
> 
> In this order, with your diff it will get loaded. But if you will have:
> 
> ----8<-------8<-------8<-------8<-----------8<----
> set limit states 700000
> set limit src-nodes 500000
> set limit tables 2000
> set limit anchors 2000
> 
> anchor test_0
> anchor test_1
> anchor test_2
> ----8<-------8<-------8<-------8<-----------8<----
> 

It looks like it works for me. This is the pf.conf I'm using
to test it:

----8<-------8<-------8<-------8<-----------8<----
set limit states 700000
set limit src-nodes 500000
set limit tables 2000
set limit anchors 2048

anchor "test_1"
anchor "test_2"
anchor "test_3"
anchor "test_4"
anchor "test_5"
...
anchor "test_1024"
----8<-------8<-------8<-------8<-----------8<----

the pf is enabled:

    src# pfctl -si |grep Enabled
    Status: Enabled for 0 days 00:06:38              Debug: err

it runs with default limits after boot:

    src# pfctl -sm
    states        hard limit   100000
    src-nodes     hard limit    10000
    frags         hard limit    65536
    tables        hard limit     1000
    table-entries hard limit   200000
    pktdelay-pkts hard limit    10000
    anchors       hard limit      512

loading the file above I'm seeing it works:

    src# pfctl -f pf-anchors.conf
    src# pfctl -sm
    states        hard limit   700000
    src-nodes     hard limit   500000
    frags         hard limit    65536
    tables        hard limit     2000
    table-entries hard limit   200000
    pktdelay-pkts hard limit    10000
    anchors       hard limit     2048

> you will have three errors for three limits set before anchors.

    sorry if question sounds dumb: are you using the right kernel
    with matching pfctl tool? (I need to try it myself to see if
    it might be case or not)

</snip>
> error = qif->pfqops->pfq_addqueue(qif->disc, q);
> 
> It looks like it is verry low level error.
> 
> I also found out that this strange state is triggered when rules are loaded
> but there were no commit triggered. I the configuration I have verry few queues.
> Only three actualy.
> 
> queue dl_XXXXX_limit on $if_XXXXX_ipsec bandwidth "120Mb"
> queue XXXXX_limit parent dl_XXXXX_limit bandwidth "50Mb"
> queue dl_XXXXXX_limit_default parent dl_XXXXX_limit bandwidth "40Mb" default
> 
> pass in quick on $if_XXXXX_ipsec proto tcp from { X.X.X.X Y.Y.Y.Y } to Z.Z.Z.Z port { AAAA } rdr-to X.X.X.X port AAAA queue XXXXX_limit

    I keep forgetting about queues. thank you for narrowing it down.
    I will try to reproduce it and see what I can figure out.

thanks and
regards
sashan