From: Stuart Henderson Subject: Re: pf_lb port range retries [Re: CVS: cvs.openbsd.org: src] To: Alexandr Nedvedicky Cc: tech Date: Mon, 9 Nov 2026 12:58:49 +0000 On 2026/11/09 13:25, Alexandr Nedvedicky wrote: > Hello, > > On Mon, Nov 09, 2026 at 11:13:20AM +0000, Stuart Henderson wrote: > > > > The fix introduces a limit on retries. The limit is set to 128. > > > If no free port number is found within 128 attempts, the function > > > gives up. > > > > I think we need to watch for issues with this, it seems like it will > > be bad on systems with a lot of machines behind a nat (though it is > > true that they will already be locking a lot..) > > > > There is a DPFPRINTF for port allocation failure for every new state, > > but it's LOG_NOTICE which is not enabled by default (and I think it > > ends up pretty noisy if you do enable it). I don't think it's > > included in any pfctl -si counter. > > > > I agree adding a counter is good idea. I will prepare the diff shortly. thank you. > > > The packet is dropped and PF_LOCK() is released so other > > > packets waiting for a rule check can be inspected by pf_test(). > > > > This isn't correct. The nat rule is not matched but ruleset traversal > > continues. If the ruleset allows it to pass (common where the nat-to > > is on a match rule), the packet is sent untranslated unless there's > > a subsequent translate rule which matches and has a free port. > > I have not tested the change for 'match ... nat-to' case, but > if I understand code correct, the packet gets always dropped > if pf_get_sport() fails. That's how things work (according > to my understanding of code in pf). > > if number of attempts is reached pf_get_sport() returns 1 > to indicate failure. > > the caller, pf_get_transaddr(), then returns -1 to further > propagate the error. > > this is intercepted in pf_match_rule() function. The block > that handles the 'match ... nat-to' rule reads as follows: > > 4805 if (r->action == PF_MATCH) { > 4806 if ((ctx->ri = pool_get(&pf_rule_item_pl, > 4807 PR_NOWAIT)) == NULL) { > 4808 REASON_SET(&ctx->reason, PFRES_MEMORY); > 4809 return (PF_TEST_FAIL); > 4810 } > 4811 ctx->ri->r = r; > 4812 /* order is irrelevant */ > 4813 SLIST_INSERT_HEAD(&ctx->rules, ctx->ri, entry); > 4814 ctx->ri = NULL; > 4815 pf_rule_to_actions(r, &ctx->act); > 4816 if (r->rule_flag & PFRULE_AFTO) > 4817 ctx->pd->naf = r->naf; > 4818 if (pf_get_transaddr(r, ctx->pd, ctx->sns, > 4819 &ctx->nr) == -1) { > 4820 REASON_SET(&ctx->reason, > 4821 PFRES_TRANSLATE); > 4822 return (PF_TEST_FAIL); > 4823 } > > the code bails out with PF_TEST_FAIL error. the in turn makes > pf_test_rule() here: > > 4979 ruleset = &pf_main_ruleset; > 4980 rv = pf_match_rule(&ctx, ruleset); > 4981 if (rv == PF_TEST_FAIL || ctx.limiter_drop == 1) { > 4982 REASON_SET(reason, ctx.reason); > 4983 goto cleanup; > 4984 } > > to take caleanup path and rule traversal stops. The value of > local variable 'action' is PF_DROP. so there should be no difference > between the way how 'pass ... nat-to' and 'mach ... nat-to' rules are > handled. hmm - I'm confused by "match log(matches)" pflog output with this then. And I'm pretty certain I've seen "packets sent out untranslated" before (but that may have been some time ago) so thought it was the same thing. Here's tcpdump pflog output from 3 connection attempts with the following as part of the ruleset: match log(matches) proto tcp to 1.1.1.1 match out on wg0 proto tcp to 1.1.1.1 port 80 nat-to em1 port 1234:1235 rtable 2 (a bit more complicated than is ideal, but that's the router I can test on easily). The first two connection attempts succeeded (3way handshake accepted), the third failed (telnet stayed at 'Trying 1.1.1.1'). # tcpdump -nevvipflog0 host 1.1.1.1 tcpdump: WARNING: snaplen raised from 116 to 160 tcpdump: listening on pflog0, link-type PFLOG 12:46:03.325939 rule 3/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 64, id 7950, len 64) 12:46:03.325949 rule 8/(match) [uid 0, pid 12426] block in on vlan6: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 64, id 7950, len 64) 12:46:03.325960 rule 12/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 64, id 7950, len 64) 12:46:03.325981 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 64, id 7950, len 64) 12:46:03.325985 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 64, id 7950, len 64) 12:46:03.326066 rule 3/(match) [uid 0, pid 12426] match out on wg0: xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:03.326082 rule 4/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:03.326088 rule 8/(match) [uid 0, pid 12426] block out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:03.326093 rule 10/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:03.326105 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:03.326108 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.28114 > 1.1.1.1.80: S 2943402084:2943402084(0) win 16384 (DF) [tos 0x10] (ttl 63, id 7950, len 64, bad ip cksum fb3a! -> fc3a) 12:46:06.558187 rule 3/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 64, id 63279, len 64) 12:46:06.558200 rule 8/(match) [uid 0, pid 12426] block in on vlan6: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 64, id 63279, len 64) 12:46:06.558206 rule 12/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 64, id 63279, len 64) 12:46:06.558227 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 64, id 63279, len 64) 12:46:06.558231 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 64, id 63279, len 64) 12:46:06.558307 rule 3/(match) [uid 0, pid 12426] match out on wg0: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:06.558325 rule 4/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:06.558333 rule 8/(match) [uid 0, pid 12426] block out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:06.558337 rule 10/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:06.558350 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:06.558353 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 63279, len 64, bad ip cksum 2319! -> 2419) 12:46:09.656296 rule 3/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 64, id 42865, len 64) 12:46:09.656308 rule 8/(match) [uid 0, pid 12426] block in on vlan6: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 64, id 42865, len 64) 12:46:09.656315 rule 12/(match) [uid 0, pid 12426] match in on vlan6: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 64, id 42865, len 64) 12:46:09.656336 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 64, id 42865, len 64) 12:46:09.656340 rule 44/(match) [uid 0, pid 12426] pass in on vlan6: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 64, id 42865, len 64) 12:46:09.656418 rule 3/(match) [uid 0, pid 12426] match out on wg0: xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:09.656435 rule 4/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:09.656442 rule 8/(match) [uid 0, pid 12426] block out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:09.656447 rule 10/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:09.656459 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:09.656462 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1234, dst 1.1.1.1:80] xxx.xx.xxx.2.3235 > 1.1.1.1.80: S 278373808:278373808(0) win 16384 (DF) [tos 0x10] (ttl 63, id 42865, len 64, bad ip cksum 72d7! -> 73d7) 12:46:12.561512 rule 3/(match) [uid 0, pid 12426] match out on wg0: xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) 12:46:12.561534 rule 4/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1235, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) 12:46:12.561543 rule 8/(match) [uid 0, pid 12426] block out on wg0: [rewritten: src 10.15.12.2:1235, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) 12:46:12.561549 rule 10/(match) [uid 0, pid 12426] match out on wg0: [rewritten: src 10.15.12.2:1235, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) 12:46:12.561569 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1235, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) 12:46:12.561574 rule 30/(match) [uid 0, pid 12426] pass out on wg0: [rewritten: src 10.15.12.2:1235, dst 1.1.1.1:80] xxx.xx.xxx.2.13233 > 1.1.1.1.80: S 1319576370:1319576370(0) win 16384 (DF) [tos 0x10] (ttl 63, id 30220, len 64, bad ip cksum a43c! -> a53c) ^C So I'm not sure quite what's going on...