From: Hrvoje Popovski Subject: Re: vio: Enable multiqueue To: Stefan Fritsch , tech@openbsd.org Date: Tue, 21 Jan 2025 20:03:33 +0100 On 21.1.2025. 19:26, Stefan Fritsch wrote: > Hi, > > Am 16.01.25 um 19:19 schrieb Hrvoje Popovski: >>>>> this diff finally enables multiqueue for vio(4). It goes on top of the >>>>> "virtio: Support unused virtqueues" diff from my previous mail. >>>>> >>>>> The distribution of of packets to the enabled queues is not optimal. To >>>>> improve this, one would need the optional RSS (receive-side scaling) >>>>> feature which is difficult to configure with libvirt/ >>>>> qemu and therefore >>>>> usually not available on hypervisors. Things may improve with future >>>>> libvirt versions. RSS support is not included in this diff. But even >>>>> without RSS, we have seen some nice performance gains. >>> >>> >>>> >>>> I'm hitting this diff with forwarding setup over ipsec for two days and >>>> doing ifconfig up/ >>>> down and hosts seems stable. Forwarding performance is >>>> the same as without this diff. >>>> >>>> I'm sending traffic from host connected to obsd1 vio2 then that traffic >>>> goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits >>>> from obsd2 vio3 to other host >>> >>> >>> Thanks for testing. Since the traffic distribution is done heuristically >>> by the hypervisor, it is often not optimal. I think it is particularily >>> bad for your case because the hypervisor will think that all ipsec >>> traffic belongs to one flow and put it into the same queue. >>> >>> I will try to improve it a bit, but in general things get better if you >>> communicate with many peers. >> >> Hi, >> >> it seems that even with plain forwarding all traffic on egress >> interfaces are going to one queue. On ingress interface interrupts are >> spread nicely. Maybe because of that forwarding performance is same as >> without multiqueue vio. > > Thanks again for the testing. > > I don't see this on my test setup. Could you please check the packet > stats in > > kstat vio*::txq: > > and maybe while the benchmark is running the cpu load distribution of > the softnet tasks in systat -> 5 > > Cheers, > Stefan Hi, I'm sending from host connected to vio2 to host connected to vio3 systat PID USER NAME CPU 20\ 40\ 60\ 80\ 100\ 94814 root softnet2 81.01 ########################### 50598 root softnet0 73.34 ######################## 81083 root softnet1 72.90 ######################## 68793 root softnet3 68.85 ###################### 9.47 ## vmstat irq76/vio2:0 0 0 irq77/vio2:1 9 0 irq78/vio2:2 6219210 10452 irq79/vio2:3 18515128 31117 irq80/vio2:4 18456557 31019 irq81/vio2:5 17052730 28660 irq82/vio3:0 0 0 irq83/vio3:1 9 0 irq84/vio3:2 279289 469 irq85/vio3:3 20 0 irq86/vio3:4 1 0 irq87/vio3:5 0 0 kstat obsd1# kstat vio*::txq: vio0:0:txq:0 packets: 368 packets bytes: 25773 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio0:0:txq:1 packets: 1910 packets bytes: 203875 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio0:0:txq:2 packets: 493 packets bytes: 33022 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio0:0:txq:3 packets: 19 packets bytes: 1853 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio1:0:txq:0 packets: 1 packets bytes: 42 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio1:0:txq:1 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio1:0:txq:2 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio1:0:txq:3 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio2:0:txq:0 packets: 1 packets bytes: 42 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio2:0:txq:1 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio2:0:txq:2 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio2:0:txq:3 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio3:0:txq:0 packets: 43609617 packets bytes: 2616576984 bytes qdrops: 82596761 packets errors: 0 packets qlen: 255 packets maxqlen: 255 packets oactive: true oactives: 267392 vio3:0:txq:1 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio3:0:txq:2 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 vio3:0:txq:3 packets: 0 packets bytes: 0 bytes qdrops: 0 packets errors: 0 packets qlen: 0 packets maxqlen: 255 packets oactive: false oactives: 0 obsd1# dmesg OpenBSD 7.6-current (GENERIC.MP) #513: Mon Jan 20 21:25:36 MST 2025 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4278005760 (4079MB) avail mem = 4124790784 (3933MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5280 (10 entries) bios0: vendor SeaBIOS version "rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org" date 04/01/2014 bios0: QEMU Standard PC (Q35 + ICH9, 2009) acpi0 at bios0: ACPI 3.0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP APIC SSDT HPET MCFG WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: QEMU Virtual CPU version 2.5+, 2095.62 MHz, 0f-6b-01 cpu0: cpuid 1 edx=178bfbfd ecx=82b82201 cpu0: cpuid 80000001 edx=20100800 ecx=1 cpu0: MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 1000MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: QEMU Virtual CPU version 2.5+, 2095.59 MHz, 0f-6b-01 cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: QEMU Virtual CPU version 2.5+, 2095.56 MHz, 0f-6b-01 cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: QEMU Virtual CPU version 2.5+, 2095.60 MHz, 0f-6b-01 cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins acpihpet0 at acpi0: 100000000 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xb0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000 "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com0: console acpicmos0 at acpi0 "ACPI0010" at acpi0 not configured "QEMUVGID" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) acpicpu2 at acpi0: C1(@1 halt!) acpicpu3 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00 vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02 wsdisplay at vga1 not configured uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16 uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17 uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18 ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi azalia0: no HD-Audio codecs ppb0 at pci0 dev 28 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci1 at ppb0 bus 1 ppb1 at pci0 dev 28 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci2 at ppb1 bus 2 ppb2 at pci0 dev 28 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci3 at ppb2 bus 3 ppb3 at pci0 dev 28 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci4 at ppb3 bus 4 uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16 uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17 uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18 ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 pci5 at ppb4 bus 5 ppb5 at pci5 dev 1 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci6 at ppb5 bus 6 virtio0 at pci6 dev 8 function 0 "Qumranet Virtio Console" rev 0x00 virtio0: no matching child driver; not configured virtio1 at pci6 dev 18 function 0 "Qumranet Virtio Network" rev 0x00 vio0 at virtio1: 4 queues, address bc:24:11:f4:52:a5 virtio1: msix virtio2 at pci6 dev 19 function 0 "Qumranet Virtio Network" rev 0x00 vio1 at virtio2: 4 queues, address bc:24:11:d2:04:82 virtio2: msix virtio3 at pci6 dev 20 function 0 "Qumranet Virtio Network" rev 0x00 vio2 at virtio3: 4 queues, address bc:24:11:53:94:bf virtio3: msix virtio4 at pci6 dev 21 function 0 "Qumranet Virtio Network" rev 0x00 vio3 at virtio4: 4 queues, address bc:24:11:39:e4:04 virtio4: msix ppb6 at pci5 dev 2 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci7 at ppb6 bus 7 ppb7 at pci5 dev 3 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci8 at ppb7 bus 8 ppb8 at pci5 dev 4 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci9 at ppb8 bus 9 virtio5 at pci9 dev 1 function 0 "Qumranet Virtio SCSI" rev 0x00 vioscsi0 at virtio5: qsize 256 virtio5: msix per-VQ scsibus1 at vioscsi0: 255 targets sd0 at scsibus1 targ 0 lun 0: sd0: 32768MB, 512 bytes/sector, 67108864 sectors, thin pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02 ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0 ahci0: port 1: 1.5Gb/s scsibus2 at ahci0: 32 targets cd0 at scsibus2 targ 1 lun 0: removable ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16 iic0 at ichiic0 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0 mux 1 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 uhidev0 at uhub1 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2 uhidev0: iclass 3/0 ums0 at uhidev0: 3 buttons, Z dir wsmouse1 at ums0 mux 0 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (d4d69482c09bd003.a) swap on sd0b dump on sd0b obsd1#