On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote:
> I am seeing a serious performance issue on OpenBSD on a hosting server 
> with many files and 128 GB RAM.
> 
> After running a large backup scan, for example with restic or rsync, the 
> kernel cache grows very large. That alone would not be a problem, but 
> after such a scan normal file access becomes much slower.
> 
> This is especially visible with PHP CMS workloads, where applications 
> perform more filesystem I/O and touch many files during a single 
> request. Simple websites load about 3-4 times slower, while larger PHP 
> CMS-based sites can become tens of times slower after the backup scan.
> 
> The server uses fast NVMe storage. In this workload, a very large 
> vnode/buffer cache appears to hurt performance more than it helps.
> 
> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode():
> 
> ```
> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs
>      : maxvnodes;
> ```
> 
> Because of this, maxvnodes can grow to match bcstats.numbufs and is 
> never reduced afterwards. After a large backup scan this results in a 
> very large vnode limit, and the system keeps a huge amount of 
> vnode/buffer cache state.
> 
> As a local test, I disabled this automatic maxvnodes growth. With this 
> change the kernel respects the configured kern.maxvnodes behavior much 
> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays 
> around 11854, which matches the expected 2x behavior from vntblinit().
> 
> After applying this patch, the slowdown disappears on my workload. PHP 
> CMS sites return to normal response times even after large restic/rsync 
> backup scans.
> 
> I do not claim that simply removing this code is the best final fix. It 
> is only a local workaround that clearly improves this workload. Maybe a 
> better solution would be to limit this autogrow, make it shrinkable, or 
> expose a tunable to control the maximum automatic vnode growth caused by 
> buffer cache size.
> 
> I can provide more details, measurements, sysctl output, or test 
> alternative patches if needed.
> 
> System details:
> 
> * OpenBSD version:
> * Architecture: amd64
> * RAM: 128 GB
> * Storage: NVMe
> * Workload: hosting server, many small files, many PHP CMS installations
> * Backup tools tested: restic, rsync
> * kern.maxvnodes: 5926
> * kern.numvnodes after patch: about 11854

You never specified what vnode count you see without the change nor how
many CPUs are present on the box.

First thing to do is to grab CPU profile to figure out where the time is
spent.

You can do it with btrace.

First, make sure you have this in sysctl.conf:
kern.allowdt=1

Afterwards on the working state:
btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre

... and visit some sites, preferably in a repeatable manner (run a
benchmark like hey or wrk2 against them perhaps?)

"break" the state, then:
btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post

... and again the same bench.

Absent said profiling info one can only speculate. If I had to do it, my
money is on the RB tree in the namecache being the problem (combined
with the other things). All path lookups are globally serialized on the
kernel lock, each name comparison is very expensive due to how memcmp is
implemented and after a full fs walk there is presumably way more
entries to traverse than usual. Pair that with waiting on other CPUs to
finish and I can easily see this being the problem.

Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2

But ultimately the real problem is the RB tree.