From: Hrvoje Popovski <hrvoje@srce.hr>
Subject: Re: vio: Enable multiqueue
To: tech@openbsd.org
Date: Thu, 9 Jan 2025 10:03:28 +0100

On 7.1.2025. 9:26, Stefan Fritsch wrote:
> Hi,
> 
> this diff finally enables multiqueue for vio(4). It goes on top of the 
> "virtio: Support unused virtqueues" diff from my previous mail.
> 
> The distribution of of packets to the enabled queues is not optimal. To 
> improve this, one would need the optional RSS (receive-side scaling) 
> feature which is difficult to configure with libvirt/qemu and therefore 
> usually not available on hypervisors. Things may improve with future 
> libvirt versions. RSS support is not included in this diff. But even 
> without RSS, we have seen some nice performance gains.
>     
> We use a single interrupt vector for every rx/tx queue pair. With config
> and control queue vectors, we need N+2 vectors for N queues.  If
> multi-queue is not available, the old scheme is used with either one
> vector per virtqueue or one vector for all queues.
>     
> * virtio: Add API to establish interrupts on specific cpus in child
>   drivers. Also make virtio_pci_setup_msix return proper errno.
>     
> * virtio_pci: Increase max number of MSIX vectors
>     
> * vio: Configure multiple queues and allocate proper interrupts.
> 
> 
> I am not entirely happy with the API for establishing interrupts. But 
> there are several variants how interrupts need to be handled (virtio pci 
> with or without MSIX, virtio mmio) and I want to keep those details from 
> the child drivers as far as possible. The way I have implemented it, only 
> the child drivers that need to allocate per-cpu interrupts need to deal 
> with the new API. If anyone has a better idea, I would be interested to 
> hear it.
> 
> comments? ok?


Hi,

I'm hitting this diff with forwarding setup over ipsec for two days and
doing ifconfig up/down and hosts seems stable. Forwarding performance is
the same as without this diff.

I'm sending traffic from host connected to obsd1 vio2 then that traffic
goes over ipsec link between obsd1 vio1 - obsd2 vio1 and traffic exits
from obsd2 vio3 to other host


obsd1# vmstat -i

interrupt                       total     rate
irq144/com0                      2749        0
irq52/ehci1                        20        0
irq65/vio0:1                       15        0
irq66/vio0:2                    27709        0
irq67/vio0:3                    37917        0
irq68/vio0:4                      267        0
irq69/vio0:5                    17490        0
irq71/vio1:1                       81        0
irq72/vio1:2                   106539        1
irq73/vio1:3                       81        0
irq74/vio1:4                       21        0
irq77/vio2:1                       45        0
irq78/vio2:2               5444311772    68168
irq79/vio2:3               5529889506    69239
irq80/vio2:4               5550808699    69501
irq81/vio2:5               5560299463    69620
irq83/vio3:1                        1        0
irq56/vioscsi0:3                57609        0
irq0/clock                   44969768      563
irq0/ipi                       933325       11
Total                     22131463077   277107


interrupt                       total     rate
irq144/com0                      3453        0
irq52/ehci1                        20        0
irq54/vioblk0:1                 52021        0
irq65/vio0:1                       15        0
irq66/vio0:2                    26751        0
irq67/vio0:3                    36743        0
irq68/vio0:4                      161        0
irq69/vio0:5                    17215        0
irq71/vio1:1                      501        0
irq72/vio1:2                151685670     1898
irq73/vio1:3                159400783     1995
irq74/vio1:4                217874776     2727
irq75/vio1:5                221039736     2766
irq77/vio2:1                        1        0
irq83/vio3:1                        9        0
irq84/vio3:2                   101817        1
irq85/vio3:3                       22        0
irq0/clock                   42318639      529
irq0/ipi                    687262836     8602
Total                      1479821169    18522