From: Claudio Jeker <cjeker@diehard.n-r-g.com>
Subject: Re: poor ipsec (10-20Mbps) performance between two openbsd hosts
To: Abel Abraham Camarillo Ojeda <acamari@verlet.org>
Cc: Openbsd Tech <tech@openbsd.org>
Date: Wed, 12 Nov 2025 15:25:57 +0100

On Wed, Nov 12, 2025 at 08:11:54AM -0600, Abel Abraham Camarillo Ojeda wrote:
> Hi to all,
> 
> I'm having around 10-20Mbps between two of my OpenBSD hosts (A and B) over
> secX interfaces.
> If I tcpbench them directly (via public ips not ipsec) I get around 400Mbps.
> 
> Over hosts B and (openbsd) C I have 800Mbps even over ipsec (but not using
> secX iface)
> 
> I think this might be related to TCP using 500 byte segments?

You have PMTU issues. Note, sec(4) by default uses a MTU of 1280.
You can bump it to 1500 and have the stack fragment the sec packets but
that could also cause issues when hops down the line use lower MTU.
 
> on A: # tcpdump -ns 1500 -i sec0 port 12345
> 13:12:39.764873 172.31.255.1.23396 > 172.31.255.0.12345: S
> 4281699220:4281699220(0) win 16384 <mss 1240,nop,nop,sackOK,nop,wscale
> 6,nop,nop,timestamp 302695704 0>
> 13:12:39.798569 172.31.255.0.12345 > 172.31.255.1.23396: S
> 3255871517:3255871517(0) ack 4281699221 win 16384 <mss
> 1240,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 917141583 302695704> (DF)
> 13:12:39.798690 172.31.255.1.23396 > 172.31.255.0.12345: . ack 1 win 256
> <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798698 172.31.255.1.23396 > 172.31.255.0.12345: . 1:501(500) ack 1
> win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798706 172.31.255.1.23396 > 172.31.255.0.12345: . 501:1001(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798711 172.31.255.1.23396 > 172.31.255.0.12345: . 1001:1501(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798716 172.31.255.1.23396 > 172.31.255.0.12345: . 1501:2001(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798721 172.31.255.1.23396 > 172.31.255.0.12345: . 2001:2501(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798726 172.31.255.1.23396 > 172.31.255.0.12345: . 2501:3001(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798731 172.31.255.1.23396 > 172.31.255.0.12345: . 3001:3501(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798736 172.31.255.1.23396 > 172.31.255.0.12345: . 3501:4001(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798741 172.31.255.1.23396 > 172.31.255.0.12345: . 4001:4501(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798746 172.31.255.1.23396 > 172.31.255.0.12345: . 4501:5001(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.798750 172.31.255.1.23396 > 172.31.255.0.12345: . 5001:5501(500)
> ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
> 13:12:39.832674 172.31.255.0.12345 > 172.31.255.1.23396: . ack 1001 win 253
> <nop,nop,timestamp 917141623 302695734> (DF)
> 13:12:39.832714 172.31.255.0.12345 > 172.31.255.1.23396: . ack 2001 win 237
> <nop,nop,timestamp 917141623 302695734> (DF)
> 13:12:39.832719 172.31.255.0.12345 > 172.31.255.1.23396: . ack 3001 win 221
> <nop,nop,timestamp 917141623 302695734> (DF)
> 13:12:39.832722 172.31.255.0.12345 > 172.31.255.1.23396: . ack 4001 win 206
> <nop,nop,timestamp 917141623 302695734> (DF)
> 13:12:39.832725 172.31.255.0.12345 > 172.31.255.1.23396: . ack 5001 win 190
> <nop,nop,timestamp 917141623 302695734> (DF)
> 13:12:39.832764 172.31.255.1.23396 > 172.31.255.0.12345: . 5501:6001(500)
> ack 1 win 256 <nop,nop,timestamp 302695764 917141623>
> 13:12:39.832776 172.31.255.1.23396 > 172.31.255.0.12345: . 6001:6501(500)
> ack 1 win 256 <nop,nop,timestamp 302695764 917141623>
> 
> host A # cat /etc/iked.conf
> ikev2 "vmrouter--horizoniq-us-dfw" active esp \
>         from 172.31.255.1 to 172.31.255.0 \
>         peer 206.191.155.20 \
>         psk "mypsk" \
>         iface sec0
> 
> Host B: iked.conf
> ikev2 "vpn--linode-us-lax" active esp \
>         from 172.31.255.0 to 172.31.255.1 \
>         peer 172.235.57.61 \
>         psk "mypsk" \
>         iface sec0
> 
> dmesg of A and B attached.

> OpenBSD 7.7 (GENERIC.MP) #625: Sun Apr 13 08:30:20 MDT 2025
>     deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 1056800768 (1007MB)
> avail mem = 998440960 (952MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5980 (10 entries)
> bios0: vendor SeaBIOS version "1.15.0-1" date 04/01/2014
> bios0: QEMU Standard PC (i440FX + PIIX, 1996)
> acpi0 at bios0: ACPI 1.0
> acpi0: sleep states S5
> acpi0: tables DSDT FACP APIC WAET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 94.49 MHz, 06-3f-02
> cpu0: cpuid 1 edx=f8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS> ecx=f7fab223<SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV>
> cpu0: cpuid 6 eax=4<ARAT>
> cpu0: cpuid 7.0 ebx=7ab<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID> ecx=4<UMIP> edx=ac000400<MD_CLEAR,IBRS,IBPB,STIBP,SSBD>
> cpu0: cpuid a vers=2, gp=4, gpwidth=48, ff=3, ffwidth=48
> cpu0: cpuid d.1 eax=1<XSAVEOPT>
> cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=21<LAHF,ABM>
> cpu0: msr 10a=48<SKIP_L1DFL,IF_PSCHANGE>
> cpu0: MELTDOWN
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1000MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 40.73 MHz, 06-3f-02
> cpu1: smt 0, core 0, package 1
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpiprt0 at acpi0: bus 0 (PCI0)
> "ACPI0006" at acpi0 not configured
> acpipci0 at acpi0 PCI0
> com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> com0: console
> "PNP0303" at acpi0 not configured
> "PNP0F13" at acpi0 not configured
> acpicmos0 at acpi0
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "QEMU0002" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> cpu0: using VERW MDS workaround
> pvbus0 at mainbus0: KVM
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
> cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
> pciide0: channel 1 disabled (no drives)
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
> iic0 at piixpm0
> vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
> wsdisplay at vga1 not configured
> virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio0: 1 queue, address 52:54:00:f1:74:94
> virtio0: msix per-VQ
> virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
> vio1 at virtio1: 1 queue, address 52:54:00:c8:a8:fb
> virtio1: msix per-VQ
> uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
> uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
> uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
> ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio2
> virtio2: msix per-VQ
> scsibus2 at vioblk0: 1 targets
> sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
> sd0: 20480MB, 512 bytes/sector, 41943040 sectors
> virtio3 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
> viomb0 at virtio3
> virtio3: apic 0 int 11
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0 mux 1
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
> uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2
> uhidev0: iclass 3/0
> ums0 at uhidev0: 3 buttons, Z dir
> wsmouse1 at ums0 mux 0
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (83249d93442c310d.a) swap on sd0b dump on sd0b
> fd0 at fdc0 drive 1: density unknown

> OpenBSD 7.8 (GENERIC) #54: Sun Oct 12 12:45:58 MDT 2025
>     deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 1056796672 (1007MB)
> avail mem = 998363136 (952MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf51b0 (10 entries)
> bios0: vendor Linode
> bios0: Linode Compute Instance
> acpi0 at bios0: ACPI 3.0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP APIC HPET MCFG WAET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD EPYC 7713 64-Core Processor, 2000.65 MHz, 19-01-01
> cpu0: cpuid 1 edx=78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2> ecx=f7fa3203<SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV>
> cpu0: cpuid 6 eax=4<ARAT>
> cpu0: cpuid 7.0 ebx=219c03ab<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> ecx=40060c<UMIP,PKU> edx=ac000010<IBRS,IBPB,STIBP,SSBD>
> cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
> cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG> ecx=8003f3<LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR>
> cpu0: cpuid 80000008 ebx=300d205<IBPB,IBRS,STIBP,SSBD,VIRTSSBD>
> cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
> cpu0: 512KB 64b/line 16-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1000MHz
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpihpet0 at acpi0: 100000000 Hz
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xb0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> "ACPI0006" at acpi0 not configured
> acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "QEMU0002" at acpi0 not configured
> "PNP0303" at acpi0 not configured
> "PNP0F13" at acpi0 not configured
> com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> acpicmos0 at acpi0
> "ACPI0010" at acpi0 not configured
> acpicpu0 at acpi0: C1(@1 halt!)
> pvbus0 at mainbus0: KVM
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
> vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> virtio0 at pci0 dev 2 function 0 "Qumranet Virtio SCSI" rev 0x00
> vioscsi0 at virtio0: qsize 256
> virtio0: msix per-VQ
> scsibus1 at vioscsi0: 255 targets
> sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
> sd0: 184MB, 512 bytes/sector, 376832 sectors, thin
> virtio1 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00
> vioscsi1 at virtio1: qsize 256
> virtio1: msix per-VQ
> scsibus2 at vioscsi1: 255 targets
> uk0 at scsibus2 targ 1 lun 0: <QEMU, QEMU TARGET, 2.5>
> sd1 at scsibus2 targ 1 lun 2: <QEMU, QEMU HARDDISK, 2.5+>
> sd1: 496MB, 512 bytes/sector, 1015808 sectors, thin
> virtio2 at pci0 dev 4 function 0 "Qumranet Virtio SCSI" rev 0x00
> vioscsi2 at virtio2: qsize 256
> virtio2: msix per-VQ
> scsibus3 at vioscsi2: 255 targets
> uk1 at scsibus3 targ 2 lun 0: <QEMU, QEMU TARGET, 2.5>
> sd2 at scsibus3 targ 2 lun 3: <QEMU, QEMU HARDDISK, 2.5+>
> sd2: 24872MB, 512 bytes/sector, 50937856 sectors, thin
> virtio3 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio3: 1 queue, address 22:00:3b:22:c5:b8
> virtio3: msix per-VQ
> virtio4 at pci0 dev 6 function 0 "Qumranet Virtio Network" rev 0x00
> vio1 at virtio4: 1 queue, address 90:de:01:22:c5:b8
> virtio4: msix per-VQ
> pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
> ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
> scsibus4 at ahci0: 32 targets
> ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16
> iic0 at ichiic0
> isa0 at pcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vscsi0 at root
> scsibus5 at vscsi0: 256 targets
> softraid0 at root
> scsibus6 at softraid0: 256 targets
> root on sd2a (e7158e2d81ea591f.a) swap on sd2b dump on sd2b


-- 
:wq Claudio