From: Robert Subject: Re: vfs: cap maxvnodes autogrow from bcstats.numbufs To: Mateusz Guzik , beck@obtuse.com Cc: tech@openbsd.org Date: Tue, 15 Sep 2026 00:02:50 +0200 Sorry for the delayed follow-up. This is a production hosting server, so I cannot reboot it or freely experiment with the kernel very often. For the last few months I was running my locally patched kernel, with the maxvnodes autogrow from bcstats.numbufs disabled. That kernel has been stable in production and, more importantly, I did not observe the cache growth/performance degradation described in my original mail. I have now upgraded the machine to OpenBSD 7.8 and am running the stock kernel again, so I was able to reproduce the original behaviour and collect some numbers. On the stock kernel, after the cache has grown: kern.bufcachepercent=5 kern.maxvnodes=3486664 kern.numvnodes=3486664 `systat iostat` shows: 23238785 total pages 27533 dma pages 1840 dirty pages 258 delwri bufs With 4096 byte pages, 23238785 pages is about 88.65 GiB. `vmstat -s` at the same time shows: 4096 bytes per page 32301971 pages managed 1175670 pages free 3965351 pages active 1109207 pages inactive 1189156 pages wired 43 number of times the pagedaemon woke up 0 number of times the pagedaemon scanned for free pages 0 pages freed by pagedaemon `top` at the same time: 762 processes: 756 idle, 6 on processor up 0 days 09:01:40 24 CPUs: 2.9% user, 0.0% nice, 2.5% sys, 8.0% spin, 0.1% intr, 86.5% idle Memory: Real: 14G/116G act/tot Free: 7155M Cache: 92G Swap: 0K/8197M `top` after reboot: 590 processes: 589 idle, 1 on processor up 0 days 00:03:38 24 CPUs: 1.2% user, 0.0% nice, 1.8% sys, 1.2% spin, 0.0% intr, 95.8% idle Memory: Real: 4391M/15G act/tot Free: 108G Cache: 6060M Swap: 0K/8197M So this happens even with: kern.bufcachepercent=5 For comparison, with my change disabling this in `getnewvnode()`: maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs : maxvnodes; I was seeing approximately: kern.maxvnodes=5926 kern.numvnodes=11854 and I could not reproduce the severe post-backup slowdown. I don't claim that the vnode count itself is necessarily the direct cause of the slowdown, as opposed to some related buffer cache or namecache behaviour. What I can say is that on this workload the correlation is very strong: on the stock kernel a large restic/rsync scan causes the vnode/cache state to grow enormously and filesystem-heavy applications become much slower. With the maxvnodes autogrow disabled, I did not observe this behaviour during several months of production use. In particular, the change prevents the cache growth in RAM which I see with the stock kernel, along with the associated performance degradation. Robert On 29/05/2026 00:31, Mateusz Guzik wrote: > On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote: >> I am seeing a serious performance issue on OpenBSD on a hosting server >> with many files and 128 GB RAM. >> >> After running a large backup scan, for example with restic or rsync, the >> kernel cache grows very large. That alone would not be a problem, but >> after such a scan normal file access becomes much slower. >> >> This is especially visible with PHP CMS workloads, where applications >> perform more filesystem I/O and touch many files during a single >> request. Simple websites load about 3-4 times slower, while larger PHP >> CMS-based sites can become tens of times slower after the backup scan. >> >> The server uses fast NVMe storage. In this workload, a very large >> vnode/buffer cache appears to hurt performance more than it helps. >> >> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode(): >> >> ``` >> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs >> : maxvnodes; >> ``` >> >> Because of this, maxvnodes can grow to match bcstats.numbufs and is >> never reduced afterwards. After a large backup scan this results in a >> very large vnode limit, and the system keeps a huge amount of >> vnode/buffer cache state. >> >> As a local test, I disabled this automatic maxvnodes growth. With this >> change the kernel respects the configured kern.maxvnodes behavior much >> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays >> around 11854, which matches the expected 2x behavior from vntblinit(). >> >> After applying this patch, the slowdown disappears on my workload. PHP >> CMS sites return to normal response times even after large restic/rsync >> backup scans. >> >> I do not claim that simply removing this code is the best final fix. It >> is only a local workaround that clearly improves this workload. Maybe a >> better solution would be to limit this autogrow, make it shrinkable, or >> expose a tunable to control the maximum automatic vnode growth caused by >> buffer cache size. >> >> I can provide more details, measurements, sysctl output, or test >> alternative patches if needed. >> >> System details: >> >> * OpenBSD version: >> * Architecture: amd64 >> * RAM: 128 GB >> * Storage: NVMe >> * Workload: hosting server, many small files, many PHP CMS installations >> * Backup tools tested: restic, rsync >> * kern.maxvnodes: 5926 >> * kern.numvnodes after patch: about 11854 > > You never specified what vnode count you see without the change nor how > many CPUs are present on the box. > > First thing to do is to grab CPU profile to figure out where the time is > spent. > > You can do it with btrace. > > First, make sure you have this in sysctl.conf: > kern.allowdt=1 > > Afterwards on the working state: > btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre > > ... and visit some sites, preferably in a repeatable manner (run a > benchmark like hey or wrk2 against them perhaps?) > > "break" the state, then: > btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post > > ... and again the same bench. > > Absent said profiling info one can only speculate. If I had to do it, my > money is on the RB tree in the namecache being the problem (combined > with the other things). All path lookups are globally serialized on the > kernel lock, each name comparison is very expensive due to how memcmp is > implemented and after a full fs walk there is presumably way more > entries to traverse than usual. Pair that with waiting on other CPUs to > finish and I can easily see this being the problem. > > Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2 > > But ultimately the real problem is the RB tree.