Sorry for the delayed follow-up.

This is a production hosting server, so I cannot reboot it or freely 
experiment with the kernel very often. For the last few months I was 
running my locally patched kernel, with the maxvnodes autogrow from 
bcstats.numbufs disabled.

That kernel has been stable in production and, more importantly, I did 
not observe the cache growth/performance degradation described in my 
original mail.

I have now upgraded the machine to OpenBSD 7.8 and am running the stock 
kernel again, so I was able to reproduce the original behaviour and 
collect some numbers.

On the stock kernel, after the cache has grown:

kern.bufcachepercent=5
kern.maxvnodes=3486664
kern.numvnodes=3486664

`systat iostat` shows:

23238785 total pages
27533 dma pages
1840 dirty pages
258 delwri bufs

With 4096 byte pages, 23238785 pages is about 88.65 GiB.

`vmstat -s` at the same time shows:

4096 bytes per page
32301971 pages managed
1175670 pages free
3965351 pages active
1109207 pages inactive
1189156 pages wired

43 number of times the pagedaemon woke up
0 number of times the pagedaemon scanned for free pages
0 pages freed by pagedaemon

`top` at the same time:

762 processes: 756 idle, 6 on processor  up 0 days 09:01:40
24  CPUs:  2.9% user,  0.0% nice,  2.5% sys,  8.0% spin,  0.1% intr, 
86.5% idle
Memory: Real: 14G/116G act/tot Free: 7155M Cache: 92G Swap: 0K/8197M

`top` after reboot:

590 processes: 589 idle, 1 on processor  up 0 days 00:03:38
24  CPUs:  1.2% user,  0.0% nice,  1.8% sys,  1.2% spin,  0.0% intr, 
95.8% idle
Memory: Real: 4391M/15G act/tot Free: 108G Cache: 6060M Swap: 0K/8197M

So this happens even with:

kern.bufcachepercent=5

For comparison, with my change disabling this in `getnewvnode()`:

maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs : maxvnodes;

I was seeing approximately:

kern.maxvnodes=5926
kern.numvnodes=11854

and I could not reproduce the severe post-backup slowdown.

I don't claim that the vnode count itself is necessarily the direct 
cause of the slowdown, as opposed to some related buffer cache or 
namecache behaviour. What I can say is that on this workload the 
correlation is very strong: on the stock kernel a large restic/rsync 
scan causes the vnode/cache state to grow enormously and 
filesystem-heavy applications become much slower.

With the maxvnodes autogrow disabled, I did not observe this behaviour 
during several months of production use. In particular, the change 
prevents the cache growth in RAM which I see with the stock kernel, 
along with the associated performance degradation.

Robert


On 29/05/2026 00:31, Mateusz Guzik wrote:
> On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote:
>> I am seeing a serious performance issue on OpenBSD on a hosting server
>> with many files and 128 GB RAM.
>>
>> After running a large backup scan, for example with restic or rsync, the
>> kernel cache grows very large. That alone would not be a problem, but
>> after such a scan normal file access becomes much slower.
>>
>> This is especially visible with PHP CMS workloads, where applications
>> perform more filesystem I/O and touch many files during a single
>> request. Simple websites load about 3-4 times slower, while larger PHP
>> CMS-based sites can become tens of times slower after the backup scan.
>>
>> The server uses fast NVMe storage. In this workload, a very large
>> vnode/buffer cache appears to hurt performance more than it helps.
>>
>> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode():
>>
>> ```
>> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs
>>       : maxvnodes;
>> ```
>>
>> Because of this, maxvnodes can grow to match bcstats.numbufs and is
>> never reduced afterwards. After a large backup scan this results in a
>> very large vnode limit, and the system keeps a huge amount of
>> vnode/buffer cache state.
>>
>> As a local test, I disabled this automatic maxvnodes growth. With this
>> change the kernel respects the configured kern.maxvnodes behavior much
>> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays
>> around 11854, which matches the expected 2x behavior from vntblinit().
>>
>> After applying this patch, the slowdown disappears on my workload. PHP
>> CMS sites return to normal response times even after large restic/rsync
>> backup scans.
>>
>> I do not claim that simply removing this code is the best final fix. It
>> is only a local workaround that clearly improves this workload. Maybe a
>> better solution would be to limit this autogrow, make it shrinkable, or
>> expose a tunable to control the maximum automatic vnode growth caused by
>> buffer cache size.
>>
>> I can provide more details, measurements, sysctl output, or test
>> alternative patches if needed.
>>
>> System details:
>>
>> * OpenBSD version:
>> * Architecture: amd64
>> * RAM: 128 GB
>> * Storage: NVMe
>> * Workload: hosting server, many small files, many PHP CMS installations
>> * Backup tools tested: restic, rsync
>> * kern.maxvnodes: 5926
>> * kern.numvnodes after patch: about 11854
> 
> You never specified what vnode count you see without the change nor how
> many CPUs are present on the box.
> 
> First thing to do is to grab CPU profile to figure out where the time is
> spent.
> 
> You can do it with btrace.
> 
> First, make sure you have this in sysctl.conf:
> kern.allowdt=1
> 
> Afterwards on the working state:
> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre
> 
> ... and visit some sites, preferably in a repeatable manner (run a
> benchmark like hey or wrk2 against them perhaps?)
> 
> "break" the state, then:
> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post
> 
> ... and again the same bench.
> 
> Absent said profiling info one can only speculate. If I had to do it, my
> money is on the RB tree in the namecache being the problem (combined
> with the other things). All path lookups are globally serialized on the
> kernel lock, each name comparison is very expensive due to how memcmp is
> implemented and after a full fs walk there is presumably way more
> entries to traverse than usual. Pair that with waiting on other CPUs to
> finish and I can easily see this being the problem.
> 
> Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2
> 
> But ultimately the real problem is the RB tree.