Index | Thread | Search

From:
Kristof Provost <kp@FreeBSD.org>
Subject:
Re: pf af-to breaks traceroute
To:
Alexandr Nedvedicky <sashan@fastmail.net>
Cc:
tech@openbsd.org
Date:
Wed, 26 Feb 2025 23:44:52 +0100

Download raw body.

Thread
This is the fix I’m likely to go with: 
https://reviews.freebsd.org/D49143
And if that’s useful, here are the test changes to check it: 
https://reviews.freebsd.org/D49144
The automated tests won’t translate easily to OpenBSD because they 
rely on FreeBSD’s vnet jails.

Best regards,
Kristof

On 26 Feb 2025, at 8:56, Kristof Provost wrote:

> Hi Alexandr,
>
> Thanks for trying to reproduce.
>
> An important note is that you’re going to need multiple routing hops 
> after the pf af-to host to demonstrate the problem.
>
> The issue is with the outer packet address, not the ICMP payload. We 
> need an intermediate router between the pf box and the final 
> destination to see this issue.
>
> I’m a little confused by the setup, given that I expected ‘SRC’ 
> to be the LAN client and ‘ECO’ the WAN server we’re 
> tracerouting, but from your capture it looks like it’s the other way 
> around.
> That means we don’t have a routing hop beyond the af-to host, and 
> won’t actually see the bug.
>
> I’ve attached lan and wan captures from an openbsd box in this 
> situation. There’s a traceroute6 to 64:ffb9:8.8.8.8 from a lan 
> client. We’ll see the correct source addresses on the ICMPv4 packets 
> on the wan side, but incorrect ICMPv6 source addresses on the lan 
> side. (All ICMPv6 errors appear to originate from 64:ffb9::808:808, 
> but the wan clearly shows multiple different source addresses for the 
> ICMPv4 errors.)
>
> That produced a tracerout6 output (though on a different run) like 
> this one:
>
>     traceroute6 to 64:ffb9::8.8.8.8 (64:ffb9::808:808) from 
> 2001:db8::2, 64 hops max, 28 byte packets
>      1  * * *
>      2  64:ffb9::808:808 (64:ffb9::808:808)  2.029 ms  1.252 ms  0.769 
> ms
>      3  64:ffb9::808:808 (64:ffb9::808:808)  4.543 ms  7.018 ms  4.863 
> ms
>      4  * * *
>      5  * * *
>      6  * * *
>      7  64:ffb9::808:808 (64:ffb9::808:808)  27.341 ms  29.157 ms  
> 19.023 ms
>      8  * * *
>      9  64:ffb9::808:808 (64:ffb9::808:808)  21.435 ms  19.053 ms  
> 20.534 ms
>
>
> Best regards,
> Kristof
>
> On 26 Feb 2025, at 0:50, Alexandr Nedvedicky wrote:
>> Hello Kristof,
>>
>> I did try to check the issue on a testbed I'm using. My set up
>> matches configuration for PF regression tests described here:
>>
>> https://github.com/openbsd/src/blob/master/regress/sys/net/pf_forward/Makefile
>>
>> There are four hsts:
>>     SRC PF RT ECO
>>
>> To check the issue hosts ECO, RT and PF interface facing towards
>> RT are running IPv6.  host SRC is connected to PF using IPv4.
>>
>> host PF uses rule as follows:
>>
>>     pass in on vio2 inet6 from any to 64:ff9b::/96 af-to \
>>     inet from 10.188.64.64 to 0.0.0.0/0
>>
>> I'm doing a traceroute from ECO to SRC:
>>
>>     eco# traceroute6  64:ff9b::10.188.210.10
>>     traceroute6 to 64:ff9b::10.188.210.10 (64:ff9b::abc:d20a), 64 
>> hops max, 60 byte packets
>>      1  fdd7:e83e:66bc:212:5054:ff:fe12:3450 
>> (fdd7:e83e:66bc:212:5054:ff:fe12:3450)  2.584 ms  0.85 ms  0.95 ms
>>      2  * * *
>>      3  64:ff9b::abc:d20a (64:ff9b::abc:d20a)  3.508 ms  1.967 ms  
>> 1.947 ms
>>
>> hop number 2 (* * *) is firewall itself. I suspect it fails to
>> send ICMP error message. firewall's ip stack sees packet as 
>> translated
>> with source address changed to 10.188.64.64, there is no route where
>> to send those packet. Adding a route does not help either
>> because reply sent by firewall's ip stack does not match state
>> created by af-to so we see ICMPv4 error travelling on wire.
>>
>> back to the issue you are reporting here.  I was intercepting packets 
>> at IPv6
>> interface on PF hosts. packet dump is attached. It looks like ICMPb6 
>> payload is
>> handled correctly by af-to. I'm using OpenBSD-current.
>> I'm attaching the packet dump.
>>
>> thanks you for reaching tech@
>>
>> regards
>> sashan