From: Scott Cheloha <scottcheloha@gmail.com>
Subject: Re: kernel: disable hardclock() on secondary CPUs
To: Mark Kettenis <mark.kettenis@xs4all.nl>
Cc: tech@openbsd.org
Date: Mon, 12 Feb 2024 14:07:20 -0600

On Mon, Feb 12, 2024 at 08:47:27PM +0100, Mark Kettenis wrote:
> > Date: Mon, 12 Feb 2024 11:35:20 -0600
> > From: Scott Cheloha <scottcheloha@gmail.com>
> > 
> > On Fri, Feb 09, 2024 at 11:57:00AM -0600, Scott Cheloha wrote:
> > > Now that the dt(4) probe entry points are out of the hardclock() we
> > > can disable it on secondary CPUs.  There is no useful work left for
> > > them to do there.
> > > 
> > > This can wait a few days, just in case we need to back out the dt(4)
> > > change.
> > > 
> > > Thoughts?
> > 
> > Ping.
> 
> ok kettenis@
> 
> Does this mean that we'll start running less interrupts on secondary
> CPUs now?

Yes.  Every secondary CPU makes HZ fewer hardclock() calls per second.
However, roundrobin() was synchronized with the hardclock() and still
runs HZ/10 times per second, so e.g. in systat(1) you will see only
(HZ - HZ/10) fewer clock interrupts per secondary CPU.

So, on my 8-core laptop, the clock interrupt rate drops from ~1600 to
~960.

This change may surprise people who eyeball the clock interrupt rate
in the vmstat view when checking whether a given system looks normal.

Index: kern_clockintr.c
===================================================================
RCS file: /cvs/src/sys/kern/kern_clockintr.c,v
diff -u -p -r1.66 kern_clockintr.c
--- kern_clockintr.c	9 Feb 2024 16:52:58 -0000	1.66
+++ kern_clockintr.c	12 Feb 2024 17:35:03 -0000
@@ -63,7 +63,7 @@ clockintr_cpu_init(const struct intrcloc
 		clockqueue_intrclock_install(cq, ic);
 
 	/* TODO: Remove this from struct clockintr_queue. */
-	if (cq->cq_hardclock.cl_expiration == 0) {
+	if (CPU_IS_PRIMARY(ci) && cq->cq_hardclock.cl_expiration == 0) {
 		clockintr_bind(&cq->cq_hardclock, ci, clockintr_hardclock,
 		    NULL);
 	}
@@ -99,12 +99,6 @@ clockintr_cpu_init(const struct intrcloc
 			clockintr_schedule(&cq->cq_hardclock, 0);
 		else
 			clockintr_advance(&cq->cq_hardclock, hardclock_period);
-	} else {
-		if (cq->cq_hardclock.cl_expiration == 0) {
-			clockintr_stagger(&cq->cq_hardclock, hardclock_period,
-			     multiplier, MAXCPUS);
-		}
-		clockintr_advance(&cq->cq_hardclock, hardclock_period);
 	}
 
 	/*
Index: kern_clock.c
===================================================================
RCS file: /cvs/src/sys/kern/kern_clock.c,v
diff -u -p -r1.122 kern_clock.c
--- kern_clock.c	9 Feb 2024 17:42:18 -0000	1.122
+++ kern_clock.c	12 Feb 2024 17:35:03 -0000
@@ -140,13 +140,6 @@ initclocks(void)
 void
 hardclock(struct clockframe *frame)
 {
-	/*
-	 * If we are not the primary CPU, we're not allowed to do
-	 * any more work.
-	 */
-	if (CPU_IS_PRIMARY(curcpu()) == 0)
-		return;
-
 	tc_ticktock();
 	ticks++;
 	jiffies++;