From: Hrvoje Popovski Subject: Re: vio: Enable multiqueue To: Stefan Fritsch , tech@openbsd.org Date: Thu, 16 Jan 2025 19:19:35 +0100 On 14.1.2025. 13:36, Stefan Fritsch wrote: > Hi Hrvoje, > > On 09.01.25 10:03, Hrvoje Popovski wrote: > >>> this diff finally enables multiqueue for vio(4). It goes on top of the >>> "virtio: Support unused virtqueues" diff from my previous mail. >>> >>> The distribution of of packets to the enabled queues is not optimal. To >>> improve this, one would need the optional RSS (receive-side scaling) >>> feature which is difficult to configure with libvirt/qemu and therefore >>> usually not available on hypervisors. Things may improve with future >>> libvirt versions. RSS support is not included in this diff. But even >>> without RSS, we have seen some nice performance gains. > > >> >> I'm hitting this diff with forwarding setup over ipsec for two days and >> doing ifconfig up/down and hosts seems stable. Forwarding performance is >> the same as without this diff. >> >> I'm sending traffic from host connected to obsd1 vio2 then that traffic >> goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits >> from obsd2 vio3 to other host > > > Thanks for testing. Since the traffic distribution is done heuristically > by the hypervisor, it is often not optimal. I think it is particularily > bad for your case because the hypervisor will think that all ipsec > traffic belongs to one flow and put it into the same queue. > > I will try to improve it a bit, but in general things get better if you > communicate with many peers. Hi, it seems that even with plain forwarding all traffic on egress interfaces are going to one queue. On ingress interface interrupts are spread nicely. Maybe because of that forwarding performance is same as without multiqueue vio. I'm sending random 10.111/24 from host connected to vio2 to random 10.222/24 to host connected to vio3. netstat Destination Gateway Flags Refs Use Mtu Prio Iface 10.111/16 192.168.111.11 UGS 0 0 - 8 vio2 10.222/16 192.168.222.11 UGS 1 1957400238 - 8 vio3 obsd1# vmstat -iz interrupt total rate irq144/com0 484 0 irq48/acpi0 0 0 irq49/uhci0 0 0 irq50/uhci1 0 0 irq51/uhci2 0 0 irq52/ehci0 0 0 irq49/ppb0 0 0 irq49/ppb1 0 0 irq49/ppb2 0 0 irq49/ppb3 0 0 irq49/uhci3 0 0 irq50/uhci4 0 0 irq51/uhci5 0 0 irq52/ehci1 20 0 irq64/vio0:0 0 0 irq65/vio0:1 9 0 irq66/vio0:2 286 0 irq67/vio0:3 1467 0 irq68/vio0:4 543 0 irq69/vio0:5 841 0 irq70/vio1:0 0 0 irq71/vio1:1 9 0 irq72/vio1:2 1 0 irq73/vio1:3 1 0 irq74/vio1:4 0 0 irq75/vio1:5 1 0 irq76/vio2:0 0 0 irq77/vio2:1 9 0 irq78/vio2:2 91031269 37170 irq79/vio2:3 154334709 63019 irq80/vio2:4 152971269 62462 irq81/vio2:5 155102163 63332 irq82/vio3:0 0 0 irq83/vio3:1 9 0 irq84/vio3:2 1928679 787 irq85/vio3:3 4 0 irq86/vio3:4 1 0 irq87/vio3:5 0 0 irq53/vioscsi0:0 0 0 irq54/vioscsi0:1 0 0 irq55/vioscsi0:2 0 0 irq56/vioscsi0:3 45856 18 irq57/ahci0 0 0 irq49/ichiic0 0 0 irq145/pckbc0 0 0 irq146/pckbc0 0 0 irq0/clock 1358939 554 irq0/ipi 10597761 4327 Total 567374330 231675 Thank you for mq vio ....