Download raw body.
vmx(4): TCP Large Receive Offload
On 22.5.2024. 22:47, jan@openbsd.org wrote:
> Hi,
>
> This diff introduces TCP Large Receive Offload (LRO) for vmx(4).
>
> The virtual device annotates LRO packets with receive descriptors of
> type 4. We need this additional information to calculate a valid MSS
> for this packet. Thus, we are able to route this kind of packets.
> But, we just get type 4 descriptors if we pretend support vmxnet3
> in revision 2.
>
> I tested it on ESXi 8 with Linux guests and external hosts. It
> increases the single TCP performance up to 20 GBit/s in my setup.
>
> Tests are welcome, especially with different ESXi versions and IP
> forwarding.
Hi,
I'm running this diff with forwarding and iperf setup at the same time
and box seems fine. Vm is on ESXi8.
With LRO diff I'm getting from 3Gbps to 30Gbps per input tcp stream.
Forwarding seems the same.
Thank you :)
while testing in a loop I run this
ifconfig vmx0 -tcplro && ifconfig vmx1 -tcplro && sleep 5 && ifconfig
vmx0 tcplro && ifconfig vmx1 tcplro && sleep 5 && ifconfig vmx0 mtu 1500
&& sleep 5 && ifconfig vmx1 mtu 1500 && sleep 5 && ifconfig vmx0 mtu
9000 && sleep 5 && ifconfig vmx1 mtu 9000 && sleep 5
vm-rtr1# netstat -sp tcp | egrep -i "tso|lro"
0 output TSO packets software chopped
6618284 output TSO packets hardware processed
22455610 output TSO packets generated
0 output TSO packets dropped
0 input LRO packets passed through pseudo device
23997654 input LRO generated packets from hardware
210350140 input LRO coalesced packets by network device
0 input bad LRO packets dropped
vm-rtr1# vmstat -i
interrupt total rate
irq49/ahci0 1 0
irq51/vmwpvs0 45538 14
irq65/vmx0:0 16717669 5263
irq66/vmx0:1 16716842 5263
irq67/vmx0:2 17246752 5430
irq68/vmx0:3 16712965 5262
irq69/vmx0:4 17417522 5484
irq70/vmx0:5 16713046 5262
irq71/vmx0:6 16636308 5238
irq72/vmx0:7 17007849 5355
irq78/vmx1:0 18168771 5720
irq79/vmx1:1 18471945 5816
irq80/vmx1:2 18651520 5872
irq81/vmx1:3 18374943 5785
irq82/vmx1:4 18931416 5960
irq83/vmx1:5 18596833 5855
irq84/vmx1:6 18827128 5927
irq85/vmx1:7 18343129 5775
irq144/pckbc0 624 0
irq0/clock 64897420 20433
irq0/ipi 419786152 132174
Total 768264373 241896
OpenBSD 7.5-current (GENERIC.MP) #1: Wed May 22 23:58:23 CEST 2024
hrvoje@vm-rtr1.netlab:/sys/arch/amd64/compile/GENERIC.MP
real mem = 8571977728 (8174MB)
avail mem = 8291057664 (7906MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (260 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 11/12/2020
bios0: VMware, Inc. VMware Virtual Platform
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3)
S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3)
S1F0(S3) PE50(S3) S1F0(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 534.33 MHz, 06-55-04
cpu0: cpuid 1
edx=1f8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT>
ecx=f7fa3203<SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0
ebx=d19f27eb<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL>
ecx=8<PKU> edx=bc000400<MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD>
cpu0: cpuid a vers=1, gp=4, gpwidth=48
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG>
ecx=121<LAHF,ABM,3DNOWP>
cpu0: cpuid 80000007 edx=100<ITSC>
cpu0: msr 10a=c<RSBA,SKIP_L1DFL>
cpu0: MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB
64b/line 16-way L2 cache, 22MB 64b/line 11-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 16MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 388.03 MHz, 06-55-04
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 456.38 MHz, 06-55-04
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 560.71 MHz, 06-55-04
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 4 (application processor)
cpu4: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 655.75 MHz, 06-55-04
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 5 (application processor)
cpu5: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 742.09 MHz, 06-55-04
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 6 (application processor)
cpu6: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 838.49 MHz, 06-55-04
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, 957.67 MHz, 06-55-04
cpu7: smt 0, core 7, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf0000000, bus 0-127
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"PNP0A05" at acpi0 not configured
acpiac0 at acpi0: AC unit online
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: VMware
vmt0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus
disabled
"VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
pci2 at ppb1 bus 2
ahci0 at pci2 dev 2 function 0 "VMware AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware SATA CD00, 1.00> removable
ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01: msi
pci3 at ppb2 bus 3
vmwpvs0 at pci3 dev 0 function 0 "VMware PVSCSI" rev 0x02: msi
scsibus2 at vmwpvs0: 65 targets
sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 2.0>
sd0: 81920MB, 512 bytes/sector, 167772160 sectors
ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01: msi
pci4 at ppb3 bus 4
ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01: msi
pci5 at ppb4 bus 5
ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01: msi
pci6 at ppb5 bus 6
ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01: msi
pci7 at ppb6 bus 7
ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01: msi
pci8 at ppb7 bus 8
ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01: msi
pci9 at ppb8 bus 9
ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01: msi
pci10 at ppb9 bus 10
ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01: msi
pci11 at ppb10 bus 11
vmx0 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: msix, 8
queues, address 00:0c:29:32:46:1c
ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01: msi
pci12 at ppb11 bus 12
ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01: msi
pci13 at ppb12 bus 13
ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01: msi
pci14 at ppb13 bus 14
ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01: msi
pci15 at ppb14 bus 15
ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01: msi
pci16 at ppb15 bus 16
ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01: msi
pci17 at ppb16 bus 17
ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01: msi
pci18 at ppb17 bus 18
ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01: msi
pci19 at ppb18 bus 19
vmx1 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: msix, 8
queues, address 00:0c:29:32:46:26
ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01: msi
pci20 at ppb19 bus 20
ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01: msi
pci21 at ppb20 bus 21
ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01: msi
pci22 at ppb21 bus 22
ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01: msi
pci23 at ppb22 bus 23
ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01: msi
pci24 at ppb23 bus 24
ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01: msi
pci25 at ppb24 bus 25
ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01: msi
pci26 at ppb25 bus 26
ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01: msi
pci27 at ppb26 bus 27
ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01: msi
pci28 at ppb27 bus 28
ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01: msi
pci29 at ppb28 bus 29
ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01: msi
pci30 at ppb29 bus 30
ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01: msi
pci31 at ppb30 bus 31
ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01: msi
pci32 at ppb31 bus 32
ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01: msi
pci33 at ppb32 bus 33
ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01: msi
pci34 at ppb33 bus 34
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (da40fdbc862b9ac4.a) swap on sd0b dump on sd0b
vm-rtr1# sysctl hw
hw.machine=amd64
hw.model=Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
hw.ncpu=8
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=cd0:,sd0:da40fdbc862b9ac4
hw.diskcount=2
hw.sensors.acpiac0.indicator0=On (power supply)
hw.sensors.vmt0.timedelta0=351.564210 secs, OK, Thu May 23 10:28:12.993
hw.cpuspeed=534
hw.vendor=VMware, Inc.
hw.product=VMware Virtual Platform
hw.version=None
hw.serialno=VMware-56 4d 6c d4 72 d6 92 13-1c f2 44 6a af 32 46 1c
hw.uuid=564d6cd4-72d6-9213-1cf2-446aaf32461c
hw.physmem=8571977728
hw.usermem=8571961344
hw.ncpufound=8
hw.allowpowerdown=1
hw.smt=0
hw.ncpuonline=8
hw.power=1
hw.ucomnames=
vmx(4): TCP Large Receive Offload