From: Christian Schulte Subject: Re: vfs: cap maxvnodes autogrow from bcstats.numbufs To: tech@openbsd.org Date: Fri, 29 May 2026 04:35:03 +0200 Am 29.05.2026 um 00:31 schrieb Mateusz Guzik: > On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote: >> I am seeing a serious performance issue on OpenBSD on a hosting server >> with many files and 128 GB RAM. >> >> After running a large backup scan, for example with restic or rsync, the >> kernel cache grows very large. That alone would not be a problem, but >> after such a scan normal file access becomes much slower. >> >> This is especially visible with PHP CMS workloads, where applications >> perform more filesystem I/O and touch many files during a single >> request. Simple websites load about 3-4 times slower, while larger PHP >> CMS-based sites can become tens of times slower after the backup scan. >> >> The server uses fast NVMe storage. In this workload, a very large >> vnode/buffer cache appears to hurt performance more than it helps. >> >> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode(): >> >> ``` >> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs >> : maxvnodes; >> ``` >> >> Because of this, maxvnodes can grow to match bcstats.numbufs and is >> never reduced afterwards. After a large backup scan this results in a >> very large vnode limit, and the system keeps a huge amount of >> vnode/buffer cache state. >> >> As a local test, I disabled this automatic maxvnodes growth. With this >> change the kernel respects the configured kern.maxvnodes behavior much >> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays >> around 11854, which matches the expected 2x behavior from vntblinit(). >> >> After applying this patch, the slowdown disappears on my workload. PHP >> CMS sites return to normal response times even after large restic/rsync >> backup scans. >> >> I do not claim that simply removing this code is the best final fix. It >> is only a local workaround that clearly improves this workload. Maybe a >> better solution would be to limit this autogrow, make it shrinkable, or >> expose a tunable to control the maximum automatic vnode growth caused by >> buffer cache size. >> >> I can provide more details, measurements, sysctl output, or test >> alternative patches if needed. >> >> System details: >> >> * OpenBSD version: >> * Architecture: amd64 >> * RAM: 128 GB >> * Storage: NVMe >> * Workload: hosting server, many small files, many PHP CMS installations >> * Backup tools tested: restic, rsync >> * kern.maxvnodes: 5926 >> * kern.numvnodes after patch: about 11854 > > You never specified what vnode count you see without the change nor how > many CPUs are present on the box. > > First thing to do is to grab CPU profile to figure out where the time is > spent. > > You can do it with btrace. > > First, make sure you have this in sysctl.conf: > kern.allowdt=1 > > Afterwards on the working state: > btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre > > ... and visit some sites, preferably in a repeatable manner (run a > benchmark like hey or wrk2 against them perhaps?) > > "break" the state, then: > btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post > > ... and again the same bench. > > Absent said profiling info one can only speculate. If I had to do it, my > money is on the RB tree in the namecache being the problem (combined > with the other things). All path lookups are globally serialized on the > kernel lock, each name comparison is very expensive due to how memcmp is > implemented and after a full fs walk there is presumably way more > entries to traverse than usual. Pair that with waiting on other CPUs to > finish and I can easily see this being the problem. > > Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2 > > But ultimately the real problem is the RB tree. > If it's that memcmp, this patch would remove it completely from the namecache lookup.