From: Mark Kettenis <mark.kettenis@xs4all.nl>
Subject: Re: Reduce KERNEL_LOCK() contention in uvn_detach()
To: Martin Pieuchot <mpi@grenadille.net>
Cc: tech@openbsd.org
Date: Tue, 22 Oct 2024 20:56:16 +0200

> Date: Sat, 19 Oct 2024 15:16:03 +0200
> From: Martin Pieuchot <mpi@grenadille.net>
> 
> The reaper is currently running mostly without grabbing the KERNEL_LOCK().
> The contending exception is the teardown path of mmaped files: uvn_detach().
> 
> We can easily improve this case by grabbing the KERNEL_LOCK() only if UVM
> has released its last reference on the vnode.
> 
> Here it's ok to interleave the KERNEL_LOCK() and a rwlock because the
> KERNEL_LOCK() is always released when failing to grab a rwlock.  The
> same trick has been (is?) used in the network stack with the NET_LOCK().
> 
> This shifts the contention when using multiple reaper paths as tested by
> claudio@.
> 
> ok?

ok kettenis@

> Index: uvm/uvm_vnode.c
> ===================================================================
> RCS file: /cvs/src/sys/uvm/uvm_vnode.c,v
> diff -u -p -r1.133 uvm_vnode.c
> --- uvm/uvm_vnode.c	24 Jul 2024 12:16:21 -0000	1.133
> +++ uvm/uvm_vnode.c	8 Oct 2024 09:14:47 -0000
> @@ -306,15 +306,14 @@ uvn_detach(struct uvm_object *uobj)
>  	struct vnode *vp;
>  	int oldflags;
>  
> -	KERNEL_LOCK();
>  	rw_enter(uobj->vmobjlock, RW_WRITE);
>  	uobj->uo_refs--;			/* drop ref! */
>  	if (uobj->uo_refs) {			/* still more refs */
>  		rw_exit(uobj->vmobjlock);
> -		KERNEL_UNLOCK();
>  		return;
>  	}
>  
> +	KERNEL_LOCK();
>  	/* get other pointers ... */
>  	uvn = (struct uvm_vnode *) uobj;
>  	vp = uvn->u_vnode;
> 
>