From: Alexander Bluhm Subject: Re: spread 8 network interrupt over cpu and softnet To: Mark Kettenis Cc: tech@openbsd.org Date: Sat, 4 Oct 2025 21:55:46 +0200 On Fri, Oct 03, 2025 at 04:27:33PM +0200, Mark Kettenis wrote: > > Date: Sat, 27 Sep 2025 18:36:50 +0200 > > From: Alexander Bluhm > > > > Hi, > > > > Currently most network drivers use 8 queues and distribute interrupts. > > ix(4) is the only one that may allocate more than 8. ice(8) uses > > only 8 vectors, but allocates more interrupts. > > > > I would like to limit interrupts and queues to 8 for all drivers. > > This prevents running out of interrupt vectors so easily. > > Here's a nickel boy; go buy yourself a better computer! > > Joking aside, we shouldn't waste resources and I it is plausible that > using the maximum number of queues provided by the hardware isn' > optimal. Underlying problem is that amd64 installes interrupt vectors on all CPUs although it is only delivered to one CPU. After we have fixed that, we can reconsider the IF_MAX_VECTORS limit. But even then it makes sense to have some common limit for all drivers. With that we can find an optimal value later. > > > Currently we have 8 softnet threads which limits parallel processing > > anyway. We can tune these values later. > > Does it even make sense to have more rx queues than softnet threads? > Should there be a single limit for both? Number of softnet threads, number of interupts, and number of CPUs influence each other. But without proper testing, it is hard to say what the performance impact is. For that I want independent limits to make experiments on multple CPU machines. Than we can find values that are suitable for common use cases. I need distinct constants for my experiments. > > With this diff, forwarding througput increases from 45 to 50 GBit/sec > > on my 12 core machine from ice0 to ice1. I am using iperf3 TCP on > > Linux to measure it. I think slower performance happend because > > ice(4) was allocating more interrupts than it has vectors. Now > > everything is limited to 8. For other drivers I see no difference > > as they operate at line speed anyway. > > > > I think arbitrary numbers IXL_MAX_VECTORS, IGC_MAX_VECTORS and > > IGC_MAX_VECTORS could go away, but that would be per driver diffs. > > > > Second part of the diff spreads the interrupts of network devices > > equally over the softnet tasks. > > That is this line: > > > - ifq->ifq_softnet = net_tq(idx); > > + ifq->ifq_softnet = net_tq(ifp->if_index * IF_MAX_VECTORS + idx); > > Does this even do anything given that both IF_MAX_VECTOR and the > number of softneq threads is 8? Doesn't this simply reduce to: > > (ifp->if_index * 8 + idx) % 8 = idx Currently it does nothing, the 8 is eliminated. But I want to see what happens with 16 softnets. And I need an easy way to test that. > Maybe it is better to stagger the interrupts instead? Like > > ifq->ifq_softnet = net_tq(ifp->if_index + idx); That code was there before, but dlg@ removed it. As we want to distribute traffic with toeplitz hash, adding interface index is wrong. My idea is to use toeplitz over 8 softnets queues. But if we have more threads, distribute the next interface over the remaining threads. Currently both are 8 and this does not matter. But I want infrastructure to play with these numbers. If it does not work, we can adjust the algorithm later. bluhm > > Index: dev/pci/if_bnxt.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_bnxt.c,v > > diff -u -p -r1.56 if_bnxt.c > > --- dev/pci/if_bnxt.c 5 Sep 2025 09:58:24 -0000 1.56 > > +++ dev/pci/if_bnxt.c 25 Sep 2025 08:27:45 -0000 > > @@ -545,9 +545,11 @@ bnxt_attach(struct device *parent, struc > > nmsix = pci_intr_msix_count(pa); > > if (nmsix > 1) { > > sc->sc_ih = pci_intr_establish(sc->sc_pc, ih, > > - IPL_NET | IPL_MPSAFE, bnxt_admin_intr, sc, DEVNAME(sc)); > > - sc->sc_intrmap = intrmap_create(&sc->sc_dev, > > - nmsix - 1, BNXT_MAX_QUEUES, INTRMAP_POWEROF2); > > + IPL_NET | IPL_MPSAFE, bnxt_admin_intr, sc, > > + DEVNAME(sc)); > > + sc->sc_intrmap = intrmap_create(&sc->sc_dev, nmsix - 1, > > + MIN(BNXT_MAX_QUEUES, IF_MAX_VECTORS), > > + INTRMAP_POWEROF2); > > sc->sc_nqueues = intrmap_count(sc->sc_intrmap); > > KASSERT(sc->sc_nqueues > 0); > > KASSERT(powerof2(sc->sc_nqueues)); > > Index: dev/pci/if_ice.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ice.c,v > > diff -u -p -r1.59 if_ice.c > > --- dev/pci/if_ice.c 17 Sep 2025 12:54:19 -0000 1.59 > > +++ dev/pci/if_ice.c 27 Sep 2025 07:35:25 -0000 > > @@ -30501,7 +30501,8 @@ ice_attach_hook(struct device *self) > > sc->sc_nmsix = nmsix; > > nqueues_max = MIN(sc->isc_nrxqsets_max, sc->isc_ntxqsets_max); > > sc->sc_intrmap = intrmap_create(&sc->sc_dev, sc->sc_nmsix - 1, > > - nqueues_max, INTRMAP_POWEROF2); > > + MIN(MIN(nqueues_max, ICE_MAX_VECTORS), IF_MAX_VECTORS), > > + INTRMAP_POWEROF2); > > nqueues = intrmap_count(sc->sc_intrmap); > > KASSERT(nqueues > 0); > > KASSERT(powerof2(nqueues)); > > Index: dev/pci/if_igc.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_igc.c,v > > diff -u -p -r1.28 if_igc.c > > --- dev/pci/if_igc.c 24 Jun 2025 11:00:27 -0000 1.28 > > +++ dev/pci/if_igc.c 27 Sep 2025 07:29:57 -0000 > > @@ -724,8 +724,8 @@ igc_setup_msix(struct igc_softc *sc) > > /* Give one vector to events. */ > > nmsix--; > > > > - sc->sc_intrmap = intrmap_create(&sc->sc_dev, nmsix, IGC_MAX_VECTORS, > > - INTRMAP_POWEROF2); > > + sc->sc_intrmap = intrmap_create(&sc->sc_dev, nmsix, > > + MIN(IGC_MAX_VECTORS, IF_MAX_VECTORS), INTRMAP_POWEROF2); > > sc->sc_nqueues = intrmap_count(sc->sc_intrmap); > > } > > > > Index: dev/pci/if_ix.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ix.c,v > > diff -u -p -r1.221 if_ix.c > > --- dev/pci/if_ix.c 24 Jun 2025 11:02:03 -0000 1.221 > > +++ dev/pci/if_ix.c 25 Sep 2025 08:27:45 -0000 > > @@ -1854,7 +1854,8 @@ ixgbe_setup_msix(struct ix_softc *sc) > > /* XXX the number of queues is limited to what we can keep stats on */ > > maxq = (sc->hw.mac.type == ixgbe_mac_82598EB) ? 8 : 16; > > > > - sc->sc_intrmap = intrmap_create(&sc->dev, nmsix, maxq, 0); > > + sc->sc_intrmap = intrmap_create(&sc->dev, nmsix, > > + MIN(maxq, IF_MAX_VECTORS), 0); > > sc->num_queues = intrmap_count(sc->sc_intrmap); > > } > > > > Index: dev/pci/if_ixl.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ixl.c,v > > diff -u -p -r1.109 if_ixl.c > > --- dev/pci/if_ixl.c 17 Sep 2025 12:54:19 -0000 1.109 > > +++ dev/pci/if_ixl.c 27 Sep 2025 07:29:57 -0000 > > @@ -1782,8 +1782,9 @@ ixl_attach(struct device *parent, struct > > if (nmsix > 1) { /* we used 1 (the 0th) for the adminq */ > > nmsix--; > > > > - sc->sc_intrmap = intrmap_create(&sc->sc_dev, > > - nmsix, IXL_MAX_VECTORS, INTRMAP_POWEROF2); > > + sc->sc_intrmap = intrmap_create(&sc->sc_dev, nmsix, > > + MIN(IXL_MAX_VECTORS, IF_MAX_VECTORS), > > + INTRMAP_POWEROF2); > > nqueues = intrmap_count(sc->sc_intrmap); > > KASSERT(nqueues > 0); > > KASSERT(powerof2(nqueues)); > > Index: net/if.h > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/net/if.h,v > > diff -u -p -r1.221 if.h > > --- net/if.h 9 Sep 2025 09:16:18 -0000 1.221 > > +++ net/if.h 25 Sep 2025 08:27:45 -0000 > > @@ -526,6 +526,9 @@ struct if_sffpage { > > #include > > > > #ifdef _KERNEL > > + > > +#define IF_MAX_VECTORS 8 > > + > > struct socket; > > struct ifnet; > > struct ifq_ops; > > Index: net/ifq.c > > =================================================================== > > RCS file: /data/mirror/openbsd/cvs/src/sys/net/ifq.c,v > > diff -u -p -r1.62 ifq.c > > --- net/ifq.c 28 Jul 2025 05:25:44 -0000 1.62 > > +++ net/ifq.c 25 Sep 2025 08:27:45 -0000 > > @@ -255,7 +255,7 @@ void > > ifq_init(struct ifqueue *ifq, struct ifnet *ifp, unsigned int idx) > > { > > ifq->ifq_if = ifp; > > - ifq->ifq_softnet = net_tq(idx); > > + ifq->ifq_softnet = net_tq(ifp->if_index * IF_MAX_VECTORS + idx); > > ifq->ifq_softc = NULL; > > > > mtx_init(&ifq->ifq_mtx, IPL_NET); > > > > >