From: Martin Pieuchot <mpi@grenadille.net>
Subject: Reduce KERNEL_LOCK() contention in uvn_detach()
To: tech@openbsd.org
Date: Sat, 19 Oct 2024 15:16:03 +0200

The reaper is currently running mostly without grabbing the KERNEL_LOCK().
The contending exception is the teardown path of mmaped files: uvn_detach().

We can easily improve this case by grabbing the KERNEL_LOCK() only if UVM
has released its last reference on the vnode.

Here it's ok to interleave the KERNEL_LOCK() and a rwlock because the
KERNEL_LOCK() is always released when failing to grab a rwlock.  The
same trick has been (is?) used in the network stack with the NET_LOCK().

This shifts the contention when using multiple reaper paths as tested by
claudio@.

ok?

Index: uvm/uvm_vnode.c
===================================================================
RCS file: /cvs/src/sys/uvm/uvm_vnode.c,v
diff -u -p -r1.133 uvm_vnode.c
--- uvm/uvm_vnode.c	24 Jul 2024 12:16:21 -0000	1.133
+++ uvm/uvm_vnode.c	8 Oct 2024 09:14:47 -0000
@@ -306,15 +306,14 @@ uvn_detach(struct uvm_object *uobj)
 	struct vnode *vp;
 	int oldflags;
 
-	KERNEL_LOCK();
 	rw_enter(uobj->vmobjlock, RW_WRITE);
 	uobj->uo_refs--;			/* drop ref! */
 	if (uobj->uo_refs) {			/* still more refs */
 		rw_exit(uobj->vmobjlock);
-		KERNEL_UNLOCK();
 		return;
 	}
 
+	KERNEL_LOCK();
 	/* get other pointers ... */
 	uvn = (struct uvm_vnode *) uobj;
 	vp = uvn->u_vnode;