On Thu, Nov 13, 2025 at 3:18 AM Claudio Jeker <cjeker@diehard.n-r-g.com>
wrote:

> On Thu, Nov 13, 2025 at 01:05:37AM -0600, Abel Abraham Camarillo Ojeda
> wrote:
> > Hi Claudio,
> >
> > But I just tested, if I remove secX and use ipsec directly (with flows?)
> I
> > get around 350-400Mbps
> > stable in tcpbench between both nodes:
> >
> > vpn-us-lax$ tcpbench -b 172.31.255.1 172.31.255.0
> > Conn:   1 Mbps:      371.654 Peak Mbps:      371.654 Avg Mbps:
> 371.654
> >        62077       47108592      376.869  100.00%
> > Conn:   1 Mbps:      376.869 Peak Mbps:      376.869 Avg Mbps:
> 376.869
> >        63077       47609600      380.877  100.00%
> > Conn:   1 Mbps:      380.877 Peak Mbps:      380.877 Avg Mbps:
> 380.877
> >        64077       48309904      386.479  100.00%
> > Conn:   1 Mbps:      386.479 Peak Mbps:      386.479 Avg Mbps:
> 386.479
> > ^C
> >
> > host A # cat /etc/iked.conf
> >  ikev2 "vmrouter--horizoniq-us-dfw" active esp \
> >          from 172.31.255.1 to 172.31.255.0 \
> >          peer 206.191.155.20 \
> >          psk "mypsk"
> >
> >  Host B: iked.conf
> >  ikev2 "vpn--linode-us-lax" active esp \
> >          from 172.31.255.0 to 172.31.255.1 \
> >         peer 172.235.57.61 \
> >          psk "mypsk"
> >
> > In the sec0 case I increased MTU in the sec interface to 1438 which gave
> me
> > (according to tcpdump) 1500 bytes
> > length packets in the real nic, and could reproduce the 10-20-30Mbps
> speed.
> >
> > With ipsec flows I get the following tcpdump:
> >
> > vpn-us-lax# tcpdump -envs 1500 -i enc0 -c 10
> > tcpdump: listening on enc0, link-type ENC
> > 07:01:53.766356 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.44312 > 172.31.255.0.12345: S [tcp sum ok]
> > 175319794:175319794(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
> > 6,nop,nop,timestamp 350528551 0> (DF) (ttl 64, id 43823, len 64) (DF)
> (ttl
> > 64, id 14908, len 84, bad ip cksum 0! -> b06d)
> > 07:01:53.800774 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20
> >
> > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.44312: R [tcp sum ok]
> > 0:0(0) ack 175319795 win 0 (DF) (ttl 64, id 63842, len 40) (DF) (ttl 45,
> id
> > 13975, len 60)
> > 07:02:01.939020 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: S [tcp sum ok]
> > 2725127438:2725127438(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
> > 6,nop,nop,timestamp 67364463 0> (DF) (ttl 64, id 39872, len 64) (DF) (ttl
> > 64, id 7049, len 84, bad ip cksum 0! -> cf20)
> > 07:02:01.973428 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20
> >
> > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.5974: S [tcp sum ok]
> > 3777116094:3777116094(0) ack 2725127439 win 16384 <mss
> > 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 2680597555 67364463>
>
> The reported MSS in both SYN and SYN+ACK are 1460 which is more than the
> MTU of the sec interface. So this requires some PMTU fixup.
>
> > (DF) (ttl 64, id 35316, len 64) (DF) (ttl 45, id 8937, len 84)
> > 07:02:01.973442 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> ack
> > 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) (ttl 64, id 52792,
> > len 52) (DF) (ttl 64, id 14980, len 72, bad ip cksum 0! -> b031)
> > 07:02:01.973558 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > 1:1385(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF)
> > (ttl 64, id 65155, len 1436) (DF) (ttl 64, id 9579, len 1456, bad ip
> cksum
> > 0! -> bfe2)
> > 07:02:01.973570 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > 1385:2769(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> (DF)
> > (ttl 64, id 13314, len 1436) (DF) (ttl 64, id 31149, len 1456, bad ip
> cksum
> > 0! -> 6ba0)
> > 07:02:01.973577 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > 2769:4153(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> (DF)
> > (ttl 64, id 21174, len 1436) (DF) (ttl 64, id 22254, len 1456, bad ip
> cksum
> > 0! -> 8e5f)
> > 07:02:01.973583 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > 4153:5537(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> (DF)
> > (ttl 64, id 45015, len 1436) (DF) (ttl 64, id 7335, len 1456, bad ip
> cksum
> > 0! -> c8a6)
> > 07:02:01.973589 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > 5537:6921(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> (DF)
> > (ttl 64, id 13552, len 1436) (DF) (ttl 64, id 15735, len 1456, bad ip
> cksum
> > 0! -> a7d6)
>
> The packets here use a len of 1456 which probably includes the outer IP
> header and therefor fits the sec(4) MTU.
>
> >
> > vpn-us-lax# tcpdump -envs 1500 -i vio0 -c 10 port 4500
> > tcpdump: listening on vio0, link-type EN10MB
> > 07:04:44.365057 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 142:
> > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp
> spi
> > 0xfa1451da seq 703531 len 100 (DF) (ttl 64, id 48683, len 128)
> > 07:04:44.399477 fe:ff:ff:ff:ff:ff 22:00:3b:22:c5:b8 0800 142:
> > 206.191.155.20.56971 > 172.235.57.61.4500: [no udp cksum] udpencap: esp
> spi
> > 0x6625e92a seq 402519 len 100 (DF) (ttl 45, id 25467, len 128)
> > 07:04:44.399515 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 130:
> > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp
> spi
> > 0xfa1451da seq 703532 len 88 (DF) (ttl 64, id 59220, len 116)
> > 07:04:44.399722 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703533 len 1472 (DF) (ttl 64, id 35434, len 1500)
> > 07:04:44.399724 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703534 len 1472 (DF) (ttl 64, id 3323, len 1500)
> > 07:04:44.399725 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703535 len 1472 (DF) (ttl 64, id 657, len 1500)
> > 07:04:44.399727 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703536 len 1472 (DF) (ttl 64, id 17938, len 1500)
> > 07:04:44.399729 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703537 len 1472 (DF) (ttl 64, id 41008, len 1500)
> > 07:04:44.399730 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703538 len 1472 (DF) (ttl 64, id 12734, len 1500)
> > 07:04:44.399731 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> seq
> > 703539 len 1472 (DF) (ttl 64, id 6637, len 1500)
> >
> > Maybe I'm not understanding something, may this be secX related?
>
> I use sec(4) between to APU boxes and I get ~100Mbps over that tunnel
> without problem. This is why I think this is something in your setup and
> not a generic sec(4) problem. Now I do run my sec(4) with MTU 1500 because
> I'm tired of PMTU issues.
>
>
Thanks Claudio, will try to to more testing later, so, you send fragments?


> --
> :wq Claudio
>