On 9.1.2025. 10:03, Hrvoje Popovski wrote:
> On 7.1.2025. 9:26, Stefan Fritsch wrote:
>> Hi,
>>
>> this diff finally enables multiqueue for vio(4). It goes on top of the 
>> "virtio: Support unused virtqueues" diff from my previous mail.
>>
>> The distribution of of packets to the enabled queues is not optimal. To 
>> improve this, one would need the optional RSS (receive-side scaling) 
>> feature which is difficult to configure with libvirt/qemu and therefore 
>> usually not available on hypervisors. Things may improve with future 
>> libvirt versions. RSS support is not included in this diff. But even 
>> without RSS, we have seen some nice performance gains.
>>     
>> We use a single interrupt vector for every rx/tx queue pair. With config
>> and control queue vectors, we need N+2 vectors for N queues.  If
>> multi-queue is not available, the old scheme is used with either one
>> vector per virtqueue or one vector for all queues.
>>     
>> * virtio: Add API to establish interrupts on specific cpus in child
>>   drivers. Also make virtio_pci_setup_msix return proper errno.
>>     
>> * virtio_pci: Increase max number of MSIX vectors
>>     
>> * vio: Configure multiple queues and allocate proper interrupts.
>>
>>
>> I am not entirely happy with the API for establishing interrupts. But 
>> there are several variants how interrupts need to be handled (virtio pci 
>> with or without MSIX, virtio mmio) and I want to keep those details from 
>> the child drivers as far as possible. The way I have implemented it, only 
>> the child drivers that need to allocate per-cpu interrupts need to deal 
>> with the new API. If anyone has a better idea, I would be interested to 
>> hear it.
>>
>> comments? ok?
> 
> 
> Hi,
> 
> I'm hitting this diff with forwarding setup over ipsec for two days and
> doing ifconfig up/down and hosts seems stable. Forwarding performance is
> the same as without this diff.
> 
> I'm sending traffic from host connected to obsd1 vio2 then that traffic
> goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits
> from obsd2 vio3 to other host


this is Proxmox Virtual Environment 8.3.2

dmesg

OpenBSD 7.6-current (GENERIC.MP) #2: Tue Jan  7 10:59:25 CET 2025
    hrvoje@obsd1.netlab:/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278005760 (4079MB)
avail mem = 4124807168 (3933MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5280 (10 entries)
bios0: vendor SeaBIOS version
"rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org" date 04/01/2014
bios0: QEMU Standard PC (Q35 + ICH9, 2009)
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC SSDT HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.5+, 2095.42 MHz, 0f-6b-01
cpu0: cpuid 1
edx=178bfbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
ecx=82b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,HV>
cpu0: cpuid 80000001 edx=20100800<NXE,LONG> ecx=1<LAHF>
cpu0: MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB
64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: QEMU Virtual CPU version 2.5+, 2095.36 MHz, 0f-6b-01
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: QEMU Virtual CPU version 2.5+, 2095.51 MHz, 0f-6b-01
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: QEMU Virtual CPU version 2.5+, 2095.34 MHz, 0f-6b-01
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
"QEMUVGID" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
wsdisplay at vga1 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi
azalia0: no HD-Audio codecs
ppb0 at pci0 dev 28 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci2 at ppb1 bus 2
ppb2 at pci0 dev 28 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci3 at ppb2 bus 3
ppb3 at pci0 dev 28 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci4 at ppb3 bus 4
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci5 at ppb4 bus 5
ppb5 at pci5 dev 1 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci6 at ppb5 bus 6
virtio0 at pci6 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: 4 queues, address bc:24:11:f4:52:a5
virtio0: msix
virtio1 at pci6 dev 19 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: 4 queues, address bc:24:11:d2:04:82
virtio1: msix
virtio2 at pci6 dev 20 function 0 "Qumranet Virtio Network" rev 0x00
vio2 at virtio2: 4 queues, address bc:24:11:53:94:bf
virtio2: msix
virtio3 at pci6 dev 21 function 0 "Qumranet Virtio Network" rev 0x00
vio3 at virtio3: 4 queues, address bc:24:11:39:e4:04
virtio3: msix
ppb6 at pci5 dev 2 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci7 at ppb6 bus 7
ppb7 at pci5 dev 3 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci8 at ppb7 bus 8
ppb8 at pci5 dev 4 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci9 at ppb8 bus 9
virtio4 at pci9 dev 1 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio4: qsize 256
virtio4: msix per-VQ
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 32768MB, 512 bytes/sector, 67108864 sectors, thin
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
ahci0: port 1: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 1 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0
int 16
iic0 at ichiic0
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub1 port 1 configuration 1 interface 0 "QEMU QEMU USB
Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (d4d69482c09bd003.a) swap on sd0b dump on sd0b