From: Hrvoje Popovski Subject: Re: em(4) multiqueue To: Jan Klemkow , Jonathan Matthew Cc: Brian Conway , tech@openbsd.org, Kevin Lo Date: Tue, 16 Jun 2026 16:33:46 +0200 On 25.4.2023. 13:28, Jan Klemkow wrote: > On Fri, Apr 14, 2023 at 10:26:14AM +0800, Kevin Lo wrote: >> On Thu, Apr 13, 2023 at 01:30:36PM -0500, Brian Conway wrote: >>> Reviving this thread, apologies for discontinuity in mail readers: https://marc.info/?t=165642193500008 >>> >>> After rebasing on 7.3, my results have mirrored Hrvoje's testing at >>> the end of that thread. No issues with throughput, unusual latency, >>> or reliability. `vmstat -i` shows some level of balancing between >>> the queues. I've been testing on as many em(4) systems as I have >>> access to, some manually, some in a packet forwarder/firewall >>> scenarios: >> Last time I tested (about a year go) on I211, rx locked up if I tried something >> like iperf3 or tcpbench. Don't know if you have a similar problem. > I rebased the rest to current and tested it with tcpbench between the > following interfaces: > > em0 at pci7 dev 0 function 0 "Intel 82580" rev 0x01, msix, 4 queues, address 90:e2:ba:df:d5:2c > em0 at pci5 dev 0 function 0 "Intel I350" rev 0x01, msix, 8 queues, address 00:25:90:eb:b3:c2 > > After a second the connection stucked. As far as I can see, the > sending side got a problem. Hi all, I've searched tech@ for multiqueue em and it seems that this is the last mail. Is this still active topic? I've got little supermicro box with few onboard em that could be nice box for testing forwarding ... em0 at pci8 dev 0 function 0 "Intel I210" rev 0x03: msi, address em1 at pci9 dev 0 function 0 "Intel I210" rev 0x03: msi, address em2 at pci12 dev 0 function 0 "Intel I350" rev 0x01: msi, address em3 at pci12 dev 0 function 1 "Intel I350" rev 0x01: msi, address em4 at pci12 dev 0 function 2 "Intel I350" rev 0x01: msi, address em5 at pci12 dev 0 function 3 "Intel I350" rev 0x01: msi, address OpenBSD 7.9 (GENERIC.MP) #449: Wed May 6 13:17:25 MDT 2026 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17052639232 (16262MB) avail mem = 16508112896 (15743MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (48 entries) bios0: vendor American Megatrends Inc. version "2.6" date 02/06/2024 bios0: Supermicro Super Server acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ acpi0: wakeup devices IP2P(S4) EHC1(S4) EHC2(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) BR2D(S4) BR3A(S4) BR3B(S4) BR3C(S4) BR3D(S4) RP01(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.05 MHz, 06-56-03, patch 0700001c cpu0: cpuid 1 edx=bfebfbff ecx=77fefbff cpu0: cpuid 6 eax=75 ecx=9 cpu0: cpuid 7.0 ebx=21cbfbb edx=9c000400 cpu0: cpuid a vers=3, gp=4, gpwidth=48, ff=3, ffwidth=48 cpu0: cpuid d.1 eax=1 cpu0: cpuid 80000001 edx=2c100800 ecx=121 cpu0: cpuid 80000007 edx=100 cpu0: MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 6MB 64b/line 12-way L3 cache cpu0: smt 0, core 0, package 0, type P mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 100MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.09 MHz, 06-56-03, patch 0700001c cpu1: smt 0, core 1, package 0, type P cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.09 MHz, 06-56-03, patch 0700001c cpu2: smt 0, core 2, package 0, type P cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.11 MHz, 06-56-03, patch 0700001c cpu3: smt 0, core 3, package 0, type P cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.20 MHz, 06-56-03, patch 0700001c cpu4: smt 1, core 0, package 0, type SP cpu5 at mainbus0: apid 3 (application processor) cpu5: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.31 MHz, 06-56-03, patch 0700001c cpu5: smt 1, core 1, package 0, type SP cpu6 at mainbus0: apid 5 (application processor) cpu6: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.41 MHz, 06-56-03, patch 0700001c cpu6: smt 1, core 2, package 0, type SP cpu7 at mainbus0: apid 7 (application processor) cpu7: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz, 2200.40 MHz, 06-56-03, patch 0700001c cpu7: smt 1, core 3, package 0, type SP ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0x80000000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 255 (UNC0) acpiprt1 at acpi0: bus 0 (PCI0) acpiprt2 at acpi0: bus 1 (BR1A) acpiprt3 at acpi0: bus 2 (BR1B) acpiprt4 at acpi0: bus 4 (BR2C) acpiprt5 at acpi0: bus 5 (BR3A) acpiprt6 at acpi0: bus 7 (RP01) acpiprt7 at acpi0: bus 8 (RP02) acpiprt8 at acpi0: bus 9 (RP04) acpiprt9 at acpi0: bus 10 (BR3A) acpiprt10 at acpi0: bus 11 (RP05) acpipci0 at acpi0 UNC0: 0x00000010 0x00000011 0x00000000 "ACPI0004" at acpi0 not configured "PNP0C33" at acpi0 not configured "PNP0C14" at acpi0 not configured acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000 "PNP0003" at acpi0 not configured "PNP0C14" at acpi0 not configured acpicmos0 at acpi0 com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo "IPI0001" at acpi0 not configured acpibtn0 at acpi0: PWRB acpicpu0 at acpi0: C1(1000@1 halt), PSS acpicpu1 at acpi0: C1(1000@1 halt), PSS acpicpu2 at acpi0: C1(1000@1 halt), PSS acpicpu3 at acpi0: C1(1000@1 halt), PSS acpicpu4 at acpi0: C1(1000@1 halt), PSS acpicpu5 at acpi0: C1(1000@1 halt), PSS acpicpu6 at acpi0: C1(1000@1 halt), PSS acpicpu7 at acpi0: C1(1000@1 halt), PSS ipmi at mainbus0 not configured cpu0: using VERW MDS workaround (except on vmm entry) cpu0: Enhanced SpeedStep 2200 MHz: speeds: 2200, 2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz pci0 at mainbus0 bus 255 "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 0 not configured "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 1 not configured "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 2 not configured "Intel Xeon-D QPI Debug" rev 0x03 at pci0 dev 11 function 3 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 0 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 1 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 2 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 3 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 0 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 4 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 5 not configured "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 6 not configured "Intel Xeon-D PCIE" rev 0x03 at pci0 dev 16 function 0 not configured "Intel E5 v4 R2PCIe Agent" rev 0x03 at pci0 dev 16 function 1 not configured "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 5 not configured "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 6 not configured "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 7 not configured "Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 0 not configured "Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 1 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 0 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 1 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 2 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 3 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 4 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 5 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 6 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 7 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 0 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 1 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 2 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 3 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 4 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 5 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 6 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 7 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 0 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 1 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 2 not configured "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 3 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 0 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 1 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 2 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 3 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 4 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 0 not configured "Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 2 not configured pci1 at mainbus0 bus 0 pchb0 at pci1 dev 0 function 0 "Intel Xeon-D Host" rev 0x03 ppb0 at pci1 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x03 pci2 at ppb0 bus 1 ppb1 at pci1 dev 1 function 1 "Intel Xeon-D PCIE" rev 0x03 pci3 at ppb1 bus 2 ppb2 at pci1 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x03 pci4 at ppb2 bus 3 "Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 0 not configured "Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 1 not configured "Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 2 not configured "Intel Xeon-D QuickData" rev 0x00 at pci4 dev 0 function 3 not configured ppb3 at pci1 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x03 pci5 at ppb3 bus 4 ix0 at pci5 dev 0 function 0 "Intel X552 SFP+" rev 0x00, msix, 4 queues, address 00:25:90:5d:ca:38 ix1 at pci5 dev 0 function 1 "Intel X552 SFP+" rev 0x00, msix, 4 queues, address 00:25:90:5d:ca:39 ppb4 at pci1 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x03: msi pci6 at ppb4 bus 5 ppb5 at pci1 dev 3 function 2 "Intel Xeon-D PCIE" rev 0x03 pci7 at ppb5 bus 6 mcx0 at pci7 dev 0 function 0 "Mellanox ConnectX-4 Lx" rev 0x00: FW 14.32.2004, msix, 4 queues, address ec:0d:9a:9e:87:5e mcx1 at pci7 dev 0 function 1 "Mellanox ConnectX-4 Lx" rev 0x00: FW 14.32.2004, msix, 4 queues, address ec:0d:9a:9e:87:5f "Intel Xeon-D Address Map" rev 0x03 at pci1 dev 5 function 0 not configured "Intel Xeon-D Hot Plug" rev 0x03 at pci1 dev 5 function 1 not configured "Intel Xeon-D RAS" rev 0x03 at pci1 dev 5 function 2 not configured xhci0 at pci1 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1 "Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 0 not configured "Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 1 not configured ehci0 at pci1 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18 usb1 at ehci0: USB revision 2.0 uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb6 at pci1 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi pci8 at ppb6 bus 7 em0 at pci8 dev 0 function 0 "Intel I210" rev 0x03: msi, address 00:25:90:5d:c9:98 ppb7 at pci1 dev 28 function 1 "Intel 8 Series PCIE" rev 0xd5: msi pci9 at ppb7 bus 8 em1 at pci9 dev 0 function 0 "Intel I210" rev 0x03: msi, address 00:25:90:5d:c9:99 ppb8 at pci1 dev 28 function 3 "Intel 8 Series PCIE" rev 0xd5: msi pci10 at ppb8 bus 9 ppb9 at pci10 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03 pci11 at ppb9 bus 10 vga1 at pci11 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb10 at pci1 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi pci12 at ppb10 bus 11 em2 at pci12 dev 0 function 0 "Intel I350" rev 0x01: msi, address 00:25:90:5d:c9:9a em3 at pci12 dev 0 function 1 "Intel I350" rev 0x01: msi, address 00:25:90:5d:c9:9b em4 at pci12 dev 0 function 2 "Intel I350" rev 0x01: msi, address 00:25:90:5d:c9:9c em5 at pci12 dev 0 function 3 "Intel I350" rev 0x01: msi, address 00:25:90:5d:c9:9d ehci1 at pci1 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18 usb2 at ehci1: USB revision 2.0 uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 pcib0 at pci1 dev 31 function 0 "Intel C224 LPC" rev 0x05 ahci0 at pci1 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3 ahci0: port 4: 6.0Gb/s scsibus1 at ahci0: 32 targets sd0 at scsibus1 targ 4 lun 0: naa.55cd2e414c94a064 sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin ichiic0 at pci1 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18 iic0 at ichiic0 "Intel 8 Series Thermal" rev 0x05 at pci1 dev 31 function 6 not configured isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 vmm0 at mainbus0: VMX/EPT uhub3 at uhub0 port 3 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 2 uhub4 at uhub3 port 2 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 3 uhub5 at uhub0 port 4 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 4 uhidev0 at uhub5 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 5 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd0 at ukbd0: console keyboard, using wsdisplay0 uhidev1 at uhub5 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 5 uhidev1: iclass 3/1 ums0 at uhidev1: 3 buttons, Z dir wsmouse0 at ums0 mux 0 uhub6 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2 uhub7 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on sd0a (cd367e465be8e57a.a) swap on sd0b dump on sd0b