From: Claudio Jeker <cjeker@diehard.n-r-g.com>
Subject: Re: poor ipsec (10-20Mbps) performance between two openbsd hosts
To: Abel Abraham Camarillo Ojeda <acamari@verlet.org>
Cc: Openbsd Tech <tech@openbsd.org>
Date: Thu, 13 Nov 2025 10:29:30 +0100

On Thu, Nov 13, 2025 at 03:23:51AM -0600, Abel Abraham Camarillo Ojeda wrote:
> On Thu, Nov 13, 2025 at 3:18 AM Claudio Jeker <cjeker@diehard.n-r-g.com>
> wrote:
> 
> > On Thu, Nov 13, 2025 at 01:05:37AM -0600, Abel Abraham Camarillo Ojeda
> > wrote:
> > > Hi Claudio,
> > >
> > > But I just tested, if I remove secX and use ipsec directly (with flows?)
> > I
> > > get around 350-400Mbps
> > > stable in tcpbench between both nodes:
> > >
> > > vpn-us-lax$ tcpbench -b 172.31.255.1 172.31.255.0
> > > Conn:   1 Mbps:      371.654 Peak Mbps:      371.654 Avg Mbps:
> > 371.654
> > >        62077       47108592      376.869  100.00%
> > > Conn:   1 Mbps:      376.869 Peak Mbps:      376.869 Avg Mbps:
> > 376.869
> > >        63077       47609600      380.877  100.00%
> > > Conn:   1 Mbps:      380.877 Peak Mbps:      380.877 Avg Mbps:
> > 380.877
> > >        64077       48309904      386.479  100.00%
> > > Conn:   1 Mbps:      386.479 Peak Mbps:      386.479 Avg Mbps:
> > 386.479
> > > ^C
> > >
> > > host A # cat /etc/iked.conf
> > >  ikev2 "vmrouter--horizoniq-us-dfw" active esp \
> > >          from 172.31.255.1 to 172.31.255.0 \
> > >          peer 206.191.155.20 \
> > >          psk "mypsk"
> > >
> > >  Host B: iked.conf
> > >  ikev2 "vpn--linode-us-lax" active esp \
> > >          from 172.31.255.0 to 172.31.255.1 \
> > >         peer 172.235.57.61 \
> > >          psk "mypsk"
> > >
> > > In the sec0 case I increased MTU in the sec interface to 1438 which gave
> > me
> > > (according to tcpdump) 1500 bytes
> > > length packets in the real nic, and could reproduce the 10-20-30Mbps
> > speed.
> > >
> > > With ipsec flows I get the following tcpdump:
> > >
> > > vpn-us-lax# tcpdump -envs 1500 -i enc0 -c 10
> > > tcpdump: listening on enc0, link-type ENC
> > > 07:01:53.766356 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.44312 > 172.31.255.0.12345: S [tcp sum ok]
> > > 175319794:175319794(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
> > > 6,nop,nop,timestamp 350528551 0> (DF) (ttl 64, id 43823, len 64) (DF)
> > (ttl
> > > 64, id 14908, len 84, bad ip cksum 0! -> b06d)
> > > 07:01:53.800774 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20
> > >
> > > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.44312: R [tcp sum ok]
> > > 0:0(0) ack 175319795 win 0 (DF) (ttl 64, id 63842, len 40) (DF) (ttl 45,
> > id
> > > 13975, len 60)
> > > 07:02:01.939020 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: S [tcp sum ok]
> > > 2725127438:2725127438(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
> > > 6,nop,nop,timestamp 67364463 0> (DF) (ttl 64, id 39872, len 64) (DF) (ttl
> > > 64, id 7049, len 84, bad ip cksum 0! -> cf20)
> > > 07:02:01.973428 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20
> > >
> > > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.5974: S [tcp sum ok]
> > > 3777116094:3777116094(0) ack 2725127439 win 16384 <mss
> > > 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 2680597555 67364463>
> >
> > The reported MSS in both SYN and SYN+ACK are 1460 which is more than the
> > MTU of the sec interface. So this requires some PMTU fixup.
> >
> > > (DF) (ttl 64, id 35316, len 64) (DF) (ttl 45, id 8937, len 84)
> > > 07:02:01.973442 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > ack
> > > 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) (ttl 64, id 52792,
> > > len 52) (DF) (ttl 64, id 14980, len 72, bad ip cksum 0! -> b031)
> > > 07:02:01.973558 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > > 1:1385(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF)
> > > (ttl 64, id 65155, len 1436) (DF) (ttl 64, id 9579, len 1456, bad ip
> > cksum
> > > 0! -> bfe2)
> > > 07:02:01.973570 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > > 1385:2769(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> > (DF)
> > > (ttl 64, id 13314, len 1436) (DF) (ttl 64, id 31149, len 1456, bad ip
> > cksum
> > > 0! -> 6ba0)
> > > 07:02:01.973577 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > > 2769:4153(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> > (DF)
> > > (ttl 64, id 21174, len 1436) (DF) (ttl 64, id 22254, len 1456, bad ip
> > cksum
> > > 0! -> 8e5f)
> > > 07:02:01.973583 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > > 4153:5537(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> > (DF)
> > > (ttl 64, id 45015, len 1436) (DF) (ttl 64, id 7335, len 1456, bad ip
> > cksum
> > > 0! -> c8a6)
> > > 07:02:01.973589 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 >
> > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok]
> > > 5537:6921(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555>
> > (DF)
> > > (ttl 64, id 13552, len 1436) (DF) (ttl 64, id 15735, len 1456, bad ip
> > cksum
> > > 0! -> a7d6)
> >
> > The packets here use a len of 1456 which probably includes the outer IP
> > header and therefor fits the sec(4) MTU.
> >
> > >
> > > vpn-us-lax# tcpdump -envs 1500 -i vio0 -c 10 port 4500
> > > tcpdump: listening on vio0, link-type EN10MB
> > > 07:04:44.365057 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 142:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp
> > spi
> > > 0xfa1451da seq 703531 len 100 (DF) (ttl 64, id 48683, len 128)
> > > 07:04:44.399477 fe:ff:ff:ff:ff:ff 22:00:3b:22:c5:b8 0800 142:
> > > 206.191.155.20.56971 > 172.235.57.61.4500: [no udp cksum] udpencap: esp
> > spi
> > > 0x6625e92a seq 402519 len 100 (DF) (ttl 45, id 25467, len 128)
> > > 07:04:44.399515 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 130:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp
> > spi
> > > 0xfa1451da seq 703532 len 88 (DF) (ttl 64, id 59220, len 116)
> > > 07:04:44.399722 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703533 len 1472 (DF) (ttl 64, id 35434, len 1500)
> > > 07:04:44.399724 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703534 len 1472 (DF) (ttl 64, id 3323, len 1500)
> > > 07:04:44.399725 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703535 len 1472 (DF) (ttl 64, id 657, len 1500)
> > > 07:04:44.399727 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703536 len 1472 (DF) (ttl 64, id 17938, len 1500)
> > > 07:04:44.399729 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703537 len 1472 (DF) (ttl 64, id 41008, len 1500)
> > > 07:04:44.399730 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703538 len 1472 (DF) (ttl 64, id 12734, len 1500)
> > > 07:04:44.399731 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514:
> > > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da
> > seq
> > > 703539 len 1472 (DF) (ttl 64, id 6637, len 1500)
> > >
> > > Maybe I'm not understanding something, may this be secX related?
> >
> > I use sec(4) between to APU boxes and I get ~100Mbps over that tunnel
> > without problem. This is why I think this is something in your setup and
> > not a generic sec(4) problem. Now I do run my sec(4) with MTU 1500 because
> > I'm tired of PMTU issues.
> >
> >
> Thanks Claudio, will try to to more testing later, so, you send fragments?

Yes, my ISP does not drop fragments even at high packet rates and the tunnel
endpoint is on a direct peering so there is very little errors because of
that.

-- 
:wq Claudio