Am 29.05.2026 um 00:31 schrieb Mateusz Guzik:
> On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote:
>> I am seeing a serious performance issue on OpenBSD on a hosting server 
>> with many files and 128 GB RAM.
>>
>> After running a large backup scan, for example with restic or rsync, the 
>> kernel cache grows very large. That alone would not be a problem, but 
>> after such a scan normal file access becomes much slower.
>>
>> This is especially visible with PHP CMS workloads, where applications 
>> perform more filesystem I/O and touch many files during a single 
>> request. Simple websites load about 3-4 times slower, while larger PHP 
>> CMS-based sites can become tens of times slower after the backup scan.
>>
>> The server uses fast NVMe storage. In this workload, a very large 
>> vnode/buffer cache appears to hurt performance more than it helps.
>>
>> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode():
>>
>> ```
>> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs
>>      : maxvnodes;
>> ```
>>
>> Because of this, maxvnodes can grow to match bcstats.numbufs and is 
>> never reduced afterwards. After a large backup scan this results in a 
>> very large vnode limit, and the system keeps a huge amount of 
>> vnode/buffer cache state.
>>
>> As a local test, I disabled this automatic maxvnodes growth. With this 
>> change the kernel respects the configured kern.maxvnodes behavior much 
>> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays 
>> around 11854, which matches the expected 2x behavior from vntblinit().
>>
>> After applying this patch, the slowdown disappears on my workload. PHP 
>> CMS sites return to normal response times even after large restic/rsync 
>> backup scans.
>>
>> I do not claim that simply removing this code is the best final fix. It 
>> is only a local workaround that clearly improves this workload. Maybe a 
>> better solution would be to limit this autogrow, make it shrinkable, or 
>> expose a tunable to control the maximum automatic vnode growth caused by 
>> buffer cache size.
>>
>> I can provide more details, measurements, sysctl output, or test 
>> alternative patches if needed.
>>
>> System details:
>>
>> * OpenBSD version:
>> * Architecture: amd64
>> * RAM: 128 GB
>> * Storage: NVMe
>> * Workload: hosting server, many small files, many PHP CMS installations
>> * Backup tools tested: restic, rsync
>> * kern.maxvnodes: 5926
>> * kern.numvnodes after patch: about 11854
> 
> You never specified what vnode count you see without the change nor how
> many CPUs are present on the box.
> 
> First thing to do is to grab CPU profile to figure out where the time is
> spent.
> 
> You can do it with btrace.
> 
> First, make sure you have this in sysctl.conf:
> kern.allowdt=1
> 
> Afterwards on the working state:
> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre
> 
> ... and visit some sites, preferably in a repeatable manner (run a
> benchmark like hey or wrk2 against them perhaps?)
> 
> "break" the state, then:
> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post
> 
> ... and again the same bench.
> 
> Absent said profiling info one can only speculate. If I had to do it, my
> money is on the RB tree in the namecache being the problem (combined
> with the other things). All path lookups are globally serialized on the
> kernel lock, each name comparison is very expensive due to how memcmp is
> implemented and after a full fs walk there is presumably way more
> entries to traverse than usual. Pair that with waiting on other CPUs to
> finish and I can easily see this being the problem.
> 
> Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2
> 
> But ultimately the real problem is the RB tree.
> 

If it's that memcmp, this patch would remove it completely from the
namecache lookup.

<https://marc.info/?l=openbsd-tech&m=177543192301688&w=2>