Download raw body.
poor ipsec (10-20Mbps) performance between two openbsd hosts
poor ipsec (10-20Mbps) performance between two openbsd hosts
poor ipsec (10-20Mbps) performance between two openbsd hosts
On Thu, Nov 13, 2025 at 01:05:37AM -0600, Abel Abraham Camarillo Ojeda wrote: > Hi Claudio, > > But I just tested, if I remove secX and use ipsec directly (with flows?) I > get around 350-400Mbps > stable in tcpbench between both nodes: > > vpn-us-lax$ tcpbench -b 172.31.255.1 172.31.255.0 > Conn: 1 Mbps: 371.654 Peak Mbps: 371.654 Avg Mbps: 371.654 > 62077 47108592 376.869 100.00% > Conn: 1 Mbps: 376.869 Peak Mbps: 376.869 Avg Mbps: 376.869 > 63077 47609600 380.877 100.00% > Conn: 1 Mbps: 380.877 Peak Mbps: 380.877 Avg Mbps: 380.877 > 64077 48309904 386.479 100.00% > Conn: 1 Mbps: 386.479 Peak Mbps: 386.479 Avg Mbps: 386.479 > ^C > > host A # cat /etc/iked.conf > ikev2 "vmrouter--horizoniq-us-dfw" active esp \ > from 172.31.255.1 to 172.31.255.0 \ > peer 206.191.155.20 \ > psk "mypsk" > > Host B: iked.conf > ikev2 "vpn--linode-us-lax" active esp \ > from 172.31.255.0 to 172.31.255.1 \ > peer 172.235.57.61 \ > psk "mypsk" > > In the sec0 case I increased MTU in the sec interface to 1438 which gave me > (according to tcpdump) 1500 bytes > length packets in the real nic, and could reproduce the 10-20-30Mbps speed. > > With ipsec flows I get the following tcpdump: > > vpn-us-lax# tcpdump -envs 1500 -i enc0 -c 10 > tcpdump: listening on enc0, link-type ENC > 07:01:53.766356 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.44312 > 172.31.255.0.12345: S [tcp sum ok] > 175319794:175319794(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale > 6,nop,nop,timestamp 350528551 0> (DF) (ttl 64, id 43823, len 64) (DF) (ttl > 64, id 14908, len 84, bad ip cksum 0! -> b06d) > 07:01:53.800774 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20 > > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.44312: R [tcp sum ok] > 0:0(0) ack 175319795 win 0 (DF) (ttl 64, id 63842, len 40) (DF) (ttl 45, id > 13975, len 60) > 07:02:01.939020 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: S [tcp sum ok] > 2725127438:2725127438(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale > 6,nop,nop,timestamp 67364463 0> (DF) (ttl 64, id 39872, len 64) (DF) (ttl > 64, id 7049, len 84, bad ip cksum 0! -> cf20) > 07:02:01.973428 (authentic,confidential): SPI 0x6625e92a: 206.191.155.20 > > 172.235.57.61: 172.31.255.0.12345 > 172.31.255.1.5974: S [tcp sum ok] > 3777116094:3777116094(0) ack 2725127439 win 16384 <mss > 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 2680597555 67364463> The reported MSS in both SYN and SYN+ACK are 1460 which is more than the MTU of the sec interface. So this requires some PMTU fixup. > (DF) (ttl 64, id 35316, len 64) (DF) (ttl 45, id 8937, len 84) > 07:02:01.973442 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] ack > 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) (ttl 64, id 52792, > len 52) (DF) (ttl 64, id 14980, len 72, bad ip cksum 0! -> b031) > 07:02:01.973558 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] > 1:1385(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) > (ttl 64, id 65155, len 1436) (DF) (ttl 64, id 9579, len 1456, bad ip cksum > 0! -> bfe2) > 07:02:01.973570 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] > 1385:2769(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) > (ttl 64, id 13314, len 1436) (DF) (ttl 64, id 31149, len 1456, bad ip cksum > 0! -> 6ba0) > 07:02:01.973577 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] > 2769:4153(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) > (ttl 64, id 21174, len 1436) (DF) (ttl 64, id 22254, len 1456, bad ip cksum > 0! -> 8e5f) > 07:02:01.973583 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] > 4153:5537(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) > (ttl 64, id 45015, len 1436) (DF) (ttl 64, id 7335, len 1456, bad ip cksum > 0! -> c8a6) > 07:02:01.973589 (authentic,confidential): SPI 0xfa1451da: 172.235.57.61 > > 206.191.155.20: 172.31.255.1.5974 > 172.31.255.0.12345: . [tcp sum ok] > 5537:6921(1384) ack 1 win 256 <nop,nop,timestamp 67364493 2680597555> (DF) > (ttl 64, id 13552, len 1436) (DF) (ttl 64, id 15735, len 1456, bad ip cksum > 0! -> a7d6) The packets here use a len of 1456 which probably includes the outer IP header and therefor fits the sec(4) MTU. > > vpn-us-lax# tcpdump -envs 1500 -i vio0 -c 10 port 4500 > tcpdump: listening on vio0, link-type EN10MB > 07:04:44.365057 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 142: > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp spi > 0xfa1451da seq 703531 len 100 (DF) (ttl 64, id 48683, len 128) > 07:04:44.399477 fe:ff:ff:ff:ff:ff 22:00:3b:22:c5:b8 0800 142: > 206.191.155.20.56971 > 172.235.57.61.4500: [no udp cksum] udpencap: esp spi > 0x6625e92a seq 402519 len 100 (DF) (ttl 45, id 25467, len 128) > 07:04:44.399515 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 130: > 172.235.57.61.4500 > 206.191.155.20.56971: [no udp cksum] udpencap: esp spi > 0xfa1451da seq 703532 len 88 (DF) (ttl 64, id 59220, len 116) > 07:04:44.399722 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703533 len 1472 (DF) (ttl 64, id 35434, len 1500) > 07:04:44.399724 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703534 len 1472 (DF) (ttl 64, id 3323, len 1500) > 07:04:44.399725 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703535 len 1472 (DF) (ttl 64, id 657, len 1500) > 07:04:44.399727 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703536 len 1472 (DF) (ttl 64, id 17938, len 1500) > 07:04:44.399729 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703537 len 1472 (DF) (ttl 64, id 41008, len 1500) > 07:04:44.399730 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703538 len 1472 (DF) (ttl 64, id 12734, len 1500) > 07:04:44.399731 22:00:3b:22:c5:b8 fe:ff:ff:ff:ff:ff 0800 1514: > 172.235.57.61.4500 > 206.191.155.20.56971: udpencap: esp spi 0xfa1451da seq > 703539 len 1472 (DF) (ttl 64, id 6637, len 1500) > > Maybe I'm not understanding something, may this be secX related? I use sec(4) between to APU boxes and I get ~100Mbps over that tunnel without problem. This is why I think this is something in your setup and not a generic sec(4) problem. Now I do run my sec(4) with MTU 1500 because I'm tired of PMTU issues. -- :wq Claudio
poor ipsec (10-20Mbps) performance between two openbsd hosts
poor ipsec (10-20Mbps) performance between two openbsd hosts
poor ipsec (10-20Mbps) performance between two openbsd hosts