From: Hrvoje Popovski Subject: Re: vio: Enable multiqueue To: tech@openbsd.org Date: Thu, 9 Jan 2025 10:12:47 +0100 On 9.1.2025. 10:03, Hrvoje Popovski wrote: > On 7.1.2025. 9:26, Stefan Fritsch wrote: >> Hi, >> >> this diff finally enables multiqueue for vio(4). It goes on top of the >> "virtio: Support unused virtqueues" diff from my previous mail. >> >> The distribution of of packets to the enabled queues is not optimal. To >> improve this, one would need the optional RSS (receive-side scaling) >> feature which is difficult to configure with libvirt/qemu and therefore >> usually not available on hypervisors. Things may improve with future >> libvirt versions. RSS support is not included in this diff. But even >> without RSS, we have seen some nice performance gains. >> >> We use a single interrupt vector for every rx/tx queue pair. With config >> and control queue vectors, we need N+2 vectors for N queues. If >> multi-queue is not available, the old scheme is used with either one >> vector per virtqueue or one vector for all queues. >> >> * virtio: Add API to establish interrupts on specific cpus in child >> drivers. Also make virtio_pci_setup_msix return proper errno. >> >> * virtio_pci: Increase max number of MSIX vectors >> >> * vio: Configure multiple queues and allocate proper interrupts. >> >> >> I am not entirely happy with the API for establishing interrupts. But >> there are several variants how interrupts need to be handled (virtio pci >> with or without MSIX, virtio mmio) and I want to keep those details from >> the child drivers as far as possible. The way I have implemented it, only >> the child drivers that need to allocate per-cpu interrupts need to deal >> with the new API. If anyone has a better idea, I would be interested to >> hear it. >> >> comments? ok? > > > Hi, > > I'm hitting this diff with forwarding setup over ipsec for two days and > doing ifconfig up/down and hosts seems stable. Forwarding performance is > the same as without this diff. > > I'm sending traffic from host connected to obsd1 vio2 then that traffic > goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits > from obsd2 vio3 to other host this is Proxmox Virtual Environment 8.3.2 dmesg OpenBSD 7.6-current (GENERIC.MP) #2: Tue Jan 7 10:59:25 CET 2025 hrvoje@obsd1.netlab:/sys/arch/amd64/compile/GENERIC.MP real mem = 4278005760 (4079MB) avail mem = 4124807168 (3933MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5280 (10 entries) bios0: vendor SeaBIOS version "rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org" date 04/01/2014 bios0: QEMU Standard PC (Q35 + ICH9, 2009) acpi0 at bios0: ACPI 3.0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP APIC SSDT HPET MCFG WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: QEMU Virtual CPU version 2.5+, 2095.42 MHz, 0f-6b-01 cpu0: cpuid 1 edx=178bfbfd ecx=82b82201 cpu0: cpuid 80000001 edx=20100800 ecx=1 cpu0: MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 1000MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: QEMU Virtual CPU version 2.5+, 2095.36 MHz, 0f-6b-01 cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: QEMU Virtual CPU version 2.5+, 2095.51 MHz, 0f-6b-01 cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: QEMU Virtual CPU version 2.5+, 2095.34 MHz, 0f-6b-01 cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins acpihpet0 at acpi0: 100000000 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xb0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000 "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com0: console acpicmos0 at acpi0 "ACPI0010" at acpi0 not configured "QEMUVGID" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) acpicpu2 at acpi0: C1(@1 halt!) acpicpu3 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00 vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02 wsdisplay at vga1 not configured uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16 uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17 uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18 ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi azalia0: no HD-Audio codecs ppb0 at pci0 dev 28 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci1 at ppb0 bus 1 ppb1 at pci0 dev 28 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci2 at ppb1 bus 2 ppb2 at pci0 dev 28 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci3 at ppb2 bus 3 ppb3 at pci0 dev 28 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 16 pci4 at ppb3 bus 4 uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16 uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17 uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18 ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 pci5 at ppb4 bus 5 ppb5 at pci5 dev 1 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci6 at ppb5 bus 6 virtio0 at pci6 dev 18 function 0 "Qumranet Virtio Network" rev 0x00 vio0 at virtio0: 4 queues, address bc:24:11:f4:52:a5 virtio0: msix virtio1 at pci6 dev 19 function 0 "Qumranet Virtio Network" rev 0x00 vio1 at virtio1: 4 queues, address bc:24:11:d2:04:82 virtio1: msix virtio2 at pci6 dev 20 function 0 "Qumranet Virtio Network" rev 0x00 vio2 at virtio2: 4 queues, address bc:24:11:53:94:bf virtio2: msix virtio3 at pci6 dev 21 function 0 "Qumranet Virtio Network" rev 0x00 vio3 at virtio3: 4 queues, address bc:24:11:39:e4:04 virtio3: msix ppb6 at pci5 dev 2 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci7 at ppb6 bus 7 ppb7 at pci5 dev 3 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci8 at ppb7 bus 8 ppb8 at pci5 dev 4 function 0 "Red Hat Qemu PCI-PCI" rev 0x00 pci9 at ppb8 bus 9 virtio4 at pci9 dev 1 function 0 "Qumranet Virtio SCSI" rev 0x00 vioscsi0 at virtio4: qsize 256 virtio4: msix per-VQ scsibus1 at vioscsi0: 255 targets sd0 at scsibus1 targ 0 lun 0: sd0: 32768MB, 512 bytes/sector, 67108864 sectors, thin pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02 ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0 ahci0: port 1: 1.5Gb/s scsibus2 at ahci0: 32 targets cd0 at scsibus2 targ 1 lun 0: removable ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16 iic0 at ichiic0 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0 mux 1 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 uhidev0 at uhub1 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2 uhidev0: iclass 3/0 ums0 at uhidev0: 3 buttons, Z dir wsmouse1 at ums0 mux 0 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (d4d69482c09bd003.a) swap on sd0b dump on sd0b