On Mon, Jan 01, 2024 at 07:12:31PM +0100, Mark Kettenis wrote:
> On amd64, we only have 240 interrupt vectors, which means they're a
> limited.  And unless I'm terribly confused, we're not using 48 of
> those because of the way we've assigned the priority levels.  And with
> more and more devices supporting multiple interrupt vectors, we can
> put these to good use.  So this diff rearranges the levels to make the
> additonal vectors available for use.  Since it most likely network
> devices that need these, I leave a gap between between IPL_NET and
> IPL_SOFTTTY.  That way, it will take longer for IPL_NET interrupts to
> spill over into the IPL_SOFTTTY range and mess up the interrupt
> priorities.
> 
> Now I vagelue remember something about trying something like this
> before.  So I want to make sure this doesn't cause any problems.
> Therefore I'd appreciate it if people can test this, especially on
> machines with lots of network devices.

Together with the multiqueue diff for em(4) from jmatthew@ the interrupt
error of em18 just changed.  I guess it got some more interrupts
allocates, but still not enough.  Thus, also on this machine the
interrupt issue improves.

Thanks,
Jan

--- old.log	Tue Jan  2 20:39:48 2024
+++ new.log	Tue Jan  2 20:51:10 2024
@@ -1,7 +1,7 @@
-OpenBSD 7.4-current (GENERIC.MP) #8: Tue Jan  2 20:32:25 CET 2024
+OpenBSD 7.4-current (GENERIC.MP) #9: Tue Jan  2 20:45:55 CET 2024
     root@ot19.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
 real mem = 34306859008 (32717MB)
-avail mem = 33246859264 (31706MB)
+avail mem = 33246846976 (31706MB)
 random: good seed from bootblocks
 mpath0 at root
 scsibus0 at mpath0: 256 targets

... skip cpu attachment ...

@@ -271,8 +271,7 @@
 em17 at pci17 dev 0 function 0 "Intel 82576" rev 0x01, msix, 8 queues, address 00:1b:21:da:f6:ec
 em18 at pci17 dev 0 function 1 "Intel 82576" rev 0x01failed to allocate interrupt slot for PIC msix pin -2138504960
 : couldn't establish queue interrupt 0
-failed to allocate interrupt slot for PIC msi pin -2138504960
-: couldn't establish interrupt at msi
+: msi, address 00:1b:21:da:f6:ed
 "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 0 not configured
 "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 1 not configured
 "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 2 not configured

> Index: arch/amd64/include/intrdefs.h
> ===================================================================
> RCS file: /cvs/src/sys/arch/amd64/include/intrdefs.h,v
> retrieving revision 1.22
> diff -u -p -r1.22 intrdefs.h
> --- arch/amd64/include/intrdefs.h	31 Aug 2021 17:40:59 -0000	1.22
> +++ arch/amd64/include/intrdefs.h	1 Jan 2024 17:54:53 -0000
> @@ -19,10 +19,10 @@
>   *
>   */
>  #define	IPL_NONE	0x0	/* nothing */
> -#define	IPL_SOFTCLOCK	0x4	/* timeouts */
> -#define	IPL_SOFTNET	0x5	/* protocol stacks */
> -#define	IPL_BIO		0x6	/* block I/O */
> -#define	IPL_NET		0x7	/* network */
> +#define	IPL_SOFTCLOCK	0x1	/* timeouts */
> +#define	IPL_SOFTNET	0x2	/* protocol stacks */
> +#define	IPL_BIO		0x3	/* block I/O */
> +#define	IPL_NET		0x4	/* network */
>  #define	IPL_SOFTTTY	0x8	/* delayed terminal handling */
>  #define	IPL_TTY		0x9	/* terminal */
>  #define	IPL_VM		0xa	/* memory allocation */
>