Download raw body.
vio: Enable multiqueue
On 21.1.2025. 19:26, Stefan Fritsch wrote:
> Hi,
>
> Am 16.01.25 um 19:19 schrieb Hrvoje Popovski:
>>>>> this diff finally enables multiqueue for vio(4). It goes on top of the
>>>>> "virtio: Support unused virtqueues" diff from my previous mail.
>>>>>
>>>>> The distribution of of packets to the enabled queues is not optimal. To
>>>>> improve this, one would need the optional RSS (receive-side scaling)
>>>>> feature which is difficult to configure with libvirt/
>>>>> qemu and therefore
>>>>> usually not available on hypervisors. Things may improve with future
>>>>> libvirt versions. RSS support is not included in this diff. But even
>>>>> without RSS, we have seen some nice performance gains.
>>>
>>>
>>>>
>>>> I'm hitting this diff with forwarding setup over ipsec for two days and
>>>> doing ifconfig up/
>>>> down and hosts seems stable. Forwarding performance is
>>>> the same as without this diff.
>>>>
>>>> I'm sending traffic from host connected to obsd1 vio2 then that traffic
>>>> goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits
>>>> from obsd2 vio3 to other host
>>>
>>>
>>> Thanks for testing. Since the traffic distribution is done heuristically
>>> by the hypervisor, it is often not optimal. I think it is particularily
>>> bad for your case because the hypervisor will think that all ipsec
>>> traffic belongs to one flow and put it into the same queue.
>>>
>>> I will try to improve it a bit, but in general things get better if you
>>> communicate with many peers.
>>
>> Hi,
>>
>> it seems that even with plain forwarding all traffic on egress
>> interfaces are going to one queue. On ingress interface interrupts are
>> spread nicely. Maybe because of that forwarding performance is same as
>> without multiqueue vio.
>
> Thanks again for the testing.
>
> I don't see this on my test setup. Could you please check the packet
> stats in
>
> kstat vio*::txq:
>
> and maybe while the benchmark is running the cpu load distribution of
> the softnet tasks in systat -> 5
>
> Cheers,
> Stefan
Hi,
I'm sending from host connected to vio2 to host connected to vio3
systat
PID USER NAME CPU 20\ 40\ 60\ 80\ 100\
94814 root softnet2 81.01 ###########################
50598 root softnet0 73.34 ########################
81083 root softnet1 72.90 ########################
68793 root softnet3 68.85 ######################
<idle> 9.47 ##
vmstat
irq76/vio2:0 0 0
irq77/vio2:1 9 0
irq78/vio2:2 6219210 10452
irq79/vio2:3 18515128 31117
irq80/vio2:4 18456557 31019
irq81/vio2:5 17052730 28660
irq82/vio3:0 0 0
irq83/vio3:1 9 0
irq84/vio3:2 279289 469
irq85/vio3:3 20 0
irq86/vio3:4 1 0
irq87/vio3:5 0 0
kstat
obsd1# kstat vio*::txq:
vio0:0:txq:0
packets: 368 packets
bytes: 25773 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio0:0:txq:1
packets: 1910 packets
bytes: 203875 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio0:0:txq:2
packets: 493 packets
bytes: 33022 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio0:0:txq:3
packets: 19 packets
bytes: 1853 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio1:0:txq:0
packets: 1 packets
bytes: 42 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio1:0:txq:1
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio1:0:txq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio1:0:txq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio2:0:txq:0
packets: 1 packets
bytes: 42 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio2:0:txq:1
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio2:0:txq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio2:0:txq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio3:0:txq:0
packets: 43609617 packets
bytes: 2616576984 bytes
qdrops: 82596761 packets
errors: 0 packets
qlen: 255 packets
maxqlen: 255 packets
oactive: true
oactives: 267392
vio3:0:txq:1
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio3:0:txq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
vio3:0:txq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
oactives: 0
obsd1#
dmesg
OpenBSD 7.6-current (GENERIC.MP) #513: Mon Jan 20 21:25:36 MST 2025
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278005760 (4079MB)
avail mem = 4124790784 (3933MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5280 (10 entries)
bios0: vendor SeaBIOS version
"rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org" date 04/01/2014
bios0: QEMU Standard PC (Q35 + ICH9, 2009)
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC SSDT HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.5+, 2095.62 MHz, 0f-6b-01
cpu0: cpuid 1
edx=178bfbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
ecx=82b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,HV>
cpu0: cpuid 80000001 edx=20100800<NXE,LONG> ecx=1<LAHF>
cpu0: MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB
64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: QEMU Virtual CPU version 2.5+, 2095.59 MHz, 0f-6b-01
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: QEMU Virtual CPU version 2.5+, 2095.56 MHz, 0f-6b-01
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: QEMU Virtual CPU version 2.5+, 2095.60 MHz, 0f-6b-01
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
"QEMUVGID" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
wsdisplay at vga1 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi
azalia0: no HD-Audio codecs
ppb0 at pci0 dev 28 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci2 at ppb1 bus 2
ppb2 at pci0 dev 28 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci3 at ppb2 bus 3
ppb3 at pci0 dev 28 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 16
pci4 at ppb3 bus 4
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 16
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 17
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci5 at ppb4 bus 5
ppb5 at pci5 dev 1 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci6 at ppb5 bus 6
virtio0 at pci6 dev 8 function 0 "Qumranet Virtio Console" rev 0x00
virtio0: no matching child driver; not configured
virtio1 at pci6 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: 4 queues, address bc:24:11:f4:52:a5
virtio1: msix
virtio2 at pci6 dev 19 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio2: 4 queues, address bc:24:11:d2:04:82
virtio2: msix
virtio3 at pci6 dev 20 function 0 "Qumranet Virtio Network" rev 0x00
vio2 at virtio3: 4 queues, address bc:24:11:53:94:bf
virtio3: msix
virtio4 at pci6 dev 21 function 0 "Qumranet Virtio Network" rev 0x00
vio3 at virtio4: 4 queues, address bc:24:11:39:e4:04
virtio4: msix
ppb6 at pci5 dev 2 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci7 at ppb6 bus 7
ppb7 at pci5 dev 3 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci8 at ppb7 bus 8
ppb8 at pci5 dev 4 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci9 at ppb8 bus 9
virtio5 at pci9 dev 1 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio5: qsize 256
virtio5: msix per-VQ
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 32768MB, 512 bytes/sector, 67108864 sectors, thin
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
ahci0: port 1: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 1 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0
int 16
iic0 at ichiic0
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub1 port 1 configuration 1 interface 0 "QEMU QEMU USB
Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (d4d69482c09bd003.a) swap on sd0b dump on sd0b
obsd1#
vio: Enable multiqueue