From: Christian Schulte Subject: Re: vfs: cap maxvnodes autogrow from bcstats.numbufs To: tech@openbsd.org Date: Fri, 29 May 2026 04:37:35 +0200 Am 29.05.2026 um 04:35 schrieb Christian Schulte: > Am 29.05.2026 um 00:31 schrieb Mateusz Guzik: >> On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote: >>> I am seeing a serious performance issue on OpenBSD on a hosting server >>> with many files and 128 GB RAM. >>> >>> After running a large backup scan, for example with restic or rsync, the >>> kernel cache grows very large. That alone would not be a problem, but >>> after such a scan normal file access becomes much slower. >>> >>> This is especially visible with PHP CMS workloads, where applications >>> perform more filesystem I/O and touch many files during a single >>> request. Simple websites load about 3-4 times slower, while larger PHP >>> CMS-based sites can become tens of times slower after the backup scan. >>> >>> The server uses fast NVMe storage. In this workload, a very large >>> vnode/buffer cache appears to hurt performance more than it helps. >>> >>> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode(): >>> >>> ``` >>> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs >>> : maxvnodes; >>> ``` >>> >>> Because of this, maxvnodes can grow to match bcstats.numbufs and is >>> never reduced afterwards. After a large backup scan this results in a >>> very large vnode limit, and the system keeps a huge amount of >>> vnode/buffer cache state. >>> >>> As a local test, I disabled this automatic maxvnodes growth. With this >>> change the kernel respects the configured kern.maxvnodes behavior much >>> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays >>> around 11854, which matches the expected 2x behavior from vntblinit(). >>> >>> After applying this patch, the slowdown disappears on my workload. PHP >>> CMS sites return to normal response times even after large restic/rsync >>> backup scans. >>> >>> I do not claim that simply removing this code is the best final fix. It >>> is only a local workaround that clearly improves this workload. Maybe a >>> better solution would be to limit this autogrow, make it shrinkable, or >>> expose a tunable to control the maximum automatic vnode growth caused by >>> buffer cache size. >>> >>> I can provide more details, measurements, sysctl output, or test >>> alternative patches if needed. >>> >>> System details: >>> >>> * OpenBSD version: >>> * Architecture: amd64 >>> * RAM: 128 GB >>> * Storage: NVMe >>> * Workload: hosting server, many small files, many PHP CMS installations >>> * Backup tools tested: restic, rsync >>> * kern.maxvnodes: 5926 >>> * kern.numvnodes after patch: about 11854 >> >> You never specified what vnode count you see without the change nor how >> many CPUs are present on the box. >> >> First thing to do is to grab CPU profile to figure out where the time is >> spent. >> >> You can do it with btrace. >> >> First, make sure you have this in sysctl.conf: >> kern.allowdt=1 >> >> Afterwards on the working state: >> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre >> >> ... and visit some sites, preferably in a repeatable manner (run a >> benchmark like hey or wrk2 against them perhaps?) >> >> "break" the state, then: >> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post >> >> ... and again the same bench. >> >> Absent said profiling info one can only speculate. If I had to do it, my >> money is on the RB tree in the namecache being the problem (combined >> with the other things). All path lookups are globally serialized on the >> kernel lock, each name comparison is very expensive due to how memcmp is >> implemented and after a full fs walk there is presumably way more >> entries to traverse than usual. Pair that with waiting on other CPUs to >> finish and I can easily see this being the problem. >> >> Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2 >> >> But ultimately the real problem is the RB tree. >> > > If it's that memcmp, this patch would remove it completely from the > namecache lookup. > > > Ah sorry. That was about memcpy...