From: Jan Klemkow Subject: Re: Adjust amd64 IPLs To: Mark Kettenis Cc: tech@openbsd.org Date: Tue, 2 Jan 2024 21:27:30 +0100 On Mon, Jan 01, 2024 at 07:12:31PM +0100, Mark Kettenis wrote: > On amd64, we only have 240 interrupt vectors, which means they're a > limited. And unless I'm terribly confused, we're not using 48 of > those because of the way we've assigned the priority levels. And with > more and more devices supporting multiple interrupt vectors, we can > put these to good use. So this diff rearranges the levels to make the > additonal vectors available for use. Since it most likely network > devices that need these, I leave a gap between between IPL_NET and > IPL_SOFTTTY. That way, it will take longer for IPL_NET interrupts to > spill over into the IPL_SOFTTTY range and mess up the interrupt > priorities. > > Now I vagelue remember something about trying something like this > before. So I want to make sure this doesn't cause any problems. > Therefore I'd appreciate it if people can test this, especially on > machines with lots of network devices. Together with the multiqueue diff for em(4) from jmatthew@ the interrupt error of em18 just changed. I guess it got some more interrupts allocates, but still not enough. Thus, also on this machine the interrupt issue improves. Thanks, Jan --- old.log Tue Jan 2 20:39:48 2024 +++ new.log Tue Jan 2 20:51:10 2024 @@ -1,7 +1,7 @@ -OpenBSD 7.4-current (GENERIC.MP) #8: Tue Jan 2 20:32:25 CET 2024 +OpenBSD 7.4-current (GENERIC.MP) #9: Tue Jan 2 20:45:55 CET 2024 root@ot19.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34306859008 (32717MB) -avail mem = 33246859264 (31706MB) +avail mem = 33246846976 (31706MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets ... skip cpu attachment ... @@ -271,8 +271,7 @@ em17 at pci17 dev 0 function 0 "Intel 82576" rev 0x01, msix, 8 queues, address 00:1b:21:da:f6:ec em18 at pci17 dev 0 function 1 "Intel 82576" rev 0x01failed to allocate interrupt slot for PIC msix pin -2138504960 : couldn't establish queue interrupt 0 -failed to allocate interrupt slot for PIC msi pin -2138504960 -: couldn't establish interrupt at msi +: msi, address 00:1b:21:da:f6:ed "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 0 not configured "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 1 not configured "Intel E5 v2 I/OAT" rev 0x04 at pci9 dev 4 function 2 not configured > Index: arch/amd64/include/intrdefs.h > =================================================================== > RCS file: /cvs/src/sys/arch/amd64/include/intrdefs.h,v > retrieving revision 1.22 > diff -u -p -r1.22 intrdefs.h > --- arch/amd64/include/intrdefs.h 31 Aug 2021 17:40:59 -0000 1.22 > +++ arch/amd64/include/intrdefs.h 1 Jan 2024 17:54:53 -0000 > @@ -19,10 +19,10 @@ > * > */ > #define IPL_NONE 0x0 /* nothing */ > -#define IPL_SOFTCLOCK 0x4 /* timeouts */ > -#define IPL_SOFTNET 0x5 /* protocol stacks */ > -#define IPL_BIO 0x6 /* block I/O */ > -#define IPL_NET 0x7 /* network */ > +#define IPL_SOFTCLOCK 0x1 /* timeouts */ > +#define IPL_SOFTNET 0x2 /* protocol stacks */ > +#define IPL_BIO 0x3 /* block I/O */ > +#define IPL_NET 0x4 /* network */ > #define IPL_SOFTTTY 0x8 /* delayed terminal handling */ > #define IPL_TTY 0x9 /* terminal */ > #define IPL_VM 0xa /* memory allocation */ >