Hi to all,

I'm having around 10-20Mbps between two of my OpenBSD hosts (A and B) over
secX interfaces.
If I tcpbench them directly (via public ips not ipsec) I get around 400Mbps.

Over hosts B and (openbsd) C I have 800Mbps even over ipsec (but not using
secX iface)

I think this might be related to TCP using 500 byte segments?

on A: # tcpdump -ns 1500 -i sec0 port 12345
13:12:39.764873 172.31.255.1.23396 > 172.31.255.0.12345: S
4281699220:4281699220(0) win 16384 <mss 1240,nop,nop,sackOK,nop,wscale
6,nop,nop,timestamp 302695704 0>
13:12:39.798569 172.31.255.0.12345 > 172.31.255.1.23396: S
3255871517:3255871517(0) ack 4281699221 win 16384 <mss
1240,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 917141583 302695704> (DF)
13:12:39.798690 172.31.255.1.23396 > 172.31.255.0.12345: . ack 1 win 256
<nop,nop,timestamp 302695734 917141583>
13:12:39.798698 172.31.255.1.23396 > 172.31.255.0.12345: . 1:501(500) ack 1
win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798706 172.31.255.1.23396 > 172.31.255.0.12345: . 501:1001(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798711 172.31.255.1.23396 > 172.31.255.0.12345: . 1001:1501(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798716 172.31.255.1.23396 > 172.31.255.0.12345: . 1501:2001(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798721 172.31.255.1.23396 > 172.31.255.0.12345: . 2001:2501(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798726 172.31.255.1.23396 > 172.31.255.0.12345: . 2501:3001(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798731 172.31.255.1.23396 > 172.31.255.0.12345: . 3001:3501(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798736 172.31.255.1.23396 > 172.31.255.0.12345: . 3501:4001(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798741 172.31.255.1.23396 > 172.31.255.0.12345: . 4001:4501(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798746 172.31.255.1.23396 > 172.31.255.0.12345: . 4501:5001(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.798750 172.31.255.1.23396 > 172.31.255.0.12345: . 5001:5501(500)
ack 1 win 256 <nop,nop,timestamp 302695734 917141583>
13:12:39.832674 172.31.255.0.12345 > 172.31.255.1.23396: . ack 1001 win 253
<nop,nop,timestamp 917141623 302695734> (DF)
13:12:39.832714 172.31.255.0.12345 > 172.31.255.1.23396: . ack 2001 win 237
<nop,nop,timestamp 917141623 302695734> (DF)
13:12:39.832719 172.31.255.0.12345 > 172.31.255.1.23396: . ack 3001 win 221
<nop,nop,timestamp 917141623 302695734> (DF)
13:12:39.832722 172.31.255.0.12345 > 172.31.255.1.23396: . ack 4001 win 206
<nop,nop,timestamp 917141623 302695734> (DF)
13:12:39.832725 172.31.255.0.12345 > 172.31.255.1.23396: . ack 5001 win 190
<nop,nop,timestamp 917141623 302695734> (DF)
13:12:39.832764 172.31.255.1.23396 > 172.31.255.0.12345: . 5501:6001(500)
ack 1 win 256 <nop,nop,timestamp 302695764 917141623>
13:12:39.832776 172.31.255.1.23396 > 172.31.255.0.12345: . 6001:6501(500)
ack 1 win 256 <nop,nop,timestamp 302695764 917141623>

host A # cat /etc/iked.conf
ikev2 "vmrouter--horizoniq-us-dfw" active esp \
        from 172.31.255.1 to 172.31.255.0 \
        peer 206.191.155.20 \
        psk "mypsk" \
        iface sec0

Host B: iked.conf
ikev2 "vpn--linode-us-lax" active esp \
        from 172.31.255.0 to 172.31.255.1 \
        peer 172.235.57.61 \
        psk "mypsk" \
        iface sec0

dmesg of A and B attached.
OpenBSD 7.7 (GENERIC.MP) #625: Sun Apr 13 08:30:20 MDT 2025
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056800768 (1007MB)
avail mem = 998440960 (952MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5980 (10 entries)
bios0: vendor SeaBIOS version "1.15.0-1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 94.49 MHz, 06-3f-02
cpu0: cpuid 1 edx=f8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS> ecx=f7fab223<SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0 ebx=7ab<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID> ecx=4<UMIP> edx=ac000400<MD_CLEAR,IBRS,IBPB,STIBP,SSBD>
cpu0: cpuid a vers=2, gp=4, gpwidth=48, ff=3, ffwidth=48
cpu0: cpuid d.1 eax=1<XSAVEOPT>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=21<LAHF,ABM>
cpu0: msr 10a=48<SKIP_L1DFL,IF_PSCHANGE>
cpu0: MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 40.73 MHz, 06-3f-02
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay at vga1 not configured
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: 1 queue, address 52:54:00:f1:74:94
virtio0: msix per-VQ
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: 1 queue, address 52:54:00:c8:a8:fb
virtio1: msix per-VQ
uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
virtio2: msix per-VQ
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
virtio3 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio3
virtio3: apic 0 int 11
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (83249d93442c310d.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown
OpenBSD 7.8 (GENERIC) #54: Sun Oct 12 12:45:58 MDT 2025
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056796672 (1007MB)
avail mem = 998363136 (952MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf51b0 (10 entries)
bios0: vendor Linode
bios0: Linode Compute Instance
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC 7713 64-Core Processor, 2000.65 MHz, 19-01-01
cpu0: cpuid 1 edx=78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2> ecx=f7fa3203<SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0 ebx=219c03ab<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> ecx=40060c<UMIP,PKU> edx=ac000010<IBRS,IBPB,STIBP,SSBD>
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG> ecx=8003f3<LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR>
cpu0: cpuid 80000008 ebx=300d205<IBPB,IBRS,STIBP,SSBD,VIRTSSBD>
cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 2 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio0: qsize 256
virtio0: msix per-VQ
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 184MB, 512 bytes/sector, 376832 sectors, thin
virtio1 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi1 at virtio1: qsize 256
virtio1: msix per-VQ
scsibus2 at vioscsi1: 255 targets
uk0 at scsibus2 targ 1 lun 0: <QEMU, QEMU TARGET, 2.5>
sd1 at scsibus2 targ 1 lun 2: <QEMU, QEMU HARDDISK, 2.5+>
sd1: 496MB, 512 bytes/sector, 1015808 sectors, thin
virtio2 at pci0 dev 4 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi2 at virtio2: qsize 256
virtio2: msix per-VQ
scsibus3 at vioscsi2: 255 targets
uk1 at scsibus3 targ 2 lun 0: <QEMU, QEMU TARGET, 2.5>
sd2 at scsibus3 targ 2 lun 3: <QEMU, QEMU HARDDISK, 2.5+>
sd2: 24872MB, 512 bytes/sector, 50937856 sectors, thin
virtio3 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio3: 1 queue, address 22:00:3b:22:c5:b8
virtio3: msix per-VQ
virtio4 at pci0 dev 6 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio4: 1 queue, address 90:de:01:22:c5:b8
virtio4: msix per-VQ
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
scsibus4 at ahci0: 32 targets
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus5 at vscsi0: 256 targets
softraid0 at root
scsibus6 at softraid0: 256 targets
root on sd2a (e7158e2d81ea591f.a) swap on sd2b dump on sd2b

• dmesgB.txt (5K)
• dmesgA.txt (4K)