Am 29.05.2026 um 04:35 schrieb Christian Schulte:
> Am 29.05.2026 um 00:31 schrieb Mateusz Guzik:
>> On Wed, May 27, 2026 at 03:14:02PM +0000, Robert wrote:
>>> I am seeing a serious performance issue on OpenBSD on a hosting server 
>>> with many files and 128 GB RAM.
>>>
>>> After running a large backup scan, for example with restic or rsync, the 
>>> kernel cache grows very large. That alone would not be a problem, but 
>>> after such a scan normal file access becomes much slower.
>>>
>>> This is especially visible with PHP CMS workloads, where applications 
>>> perform more filesystem I/O and touch many files during a single 
>>> request. Simple websites load about 3-4 times slower, while larger PHP 
>>> CMS-based sites can become tens of times slower after the backup scan.
>>>
>>> The server uses fast NVMe storage. In this workload, a very large 
>>> vnode/buffer cache appears to hurt performance more than it helps.
>>>
>>> I traced the issue to sys/kern/vfs_subr.c, in getnewvnode():
>>>
>>> ```
>>> maxvnodes = maxvnodes < bcstats.numbufs ? bcstats.numbufs
>>>      : maxvnodes;
>>> ```
>>>
>>> Because of this, maxvnodes can grow to match bcstats.numbufs and is 
>>> never reduced afterwards. After a large backup scan this results in a 
>>> very large vnode limit, and the system keeps a huge amount of 
>>> vnode/buffer cache state.
>>>
>>> As a local test, I disabled this automatic maxvnodes growth. With this 
>>> change the kernel respects the configured kern.maxvnodes behavior much 
>>> better. In my case kern.maxvnodes is 5926 and kern.numvnodes stays 
>>> around 11854, which matches the expected 2x behavior from vntblinit().
>>>
>>> After applying this patch, the slowdown disappears on my workload. PHP 
>>> CMS sites return to normal response times even after large restic/rsync 
>>> backup scans.
>>>
>>> I do not claim that simply removing this code is the best final fix. It 
>>> is only a local workaround that clearly improves this workload. Maybe a 
>>> better solution would be to limit this autogrow, make it shrinkable, or 
>>> expose a tunable to control the maximum automatic vnode growth caused by 
>>> buffer cache size.
>>>
>>> I can provide more details, measurements, sysctl output, or test 
>>> alternative patches if needed.
>>>
>>> System details:
>>>
>>> * OpenBSD version:
>>> * Architecture: amd64
>>> * RAM: 128 GB
>>> * Storage: NVMe
>>> * Workload: hosting server, many small files, many PHP CMS installations
>>> * Backup tools tested: restic, rsync
>>> * kern.maxvnodes: 5926
>>> * kern.numvnodes after patch: about 11854
>>
>> You never specified what vnode count you see without the change nor how
>> many CPUs are present on the box.
>>
>> First thing to do is to grab CPU profile to figure out where the time is
>> spent.
>>
>> You can do it with btrace.
>>
>> First, make sure you have this in sysctl.conf:
>> kern.allowdt=1
>>
>> Afterwards on the working state:
>> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > pre
>>
>> ... and visit some sites, preferably in a repeatable manner (run a
>> benchmark like hey or wrk2 against them perhaps?)
>>
>> "break" the state, then:
>> btrace -e 'profile:hz:99 { @[kstack] = count(); }' > post
>>
>> ... and again the same bench.
>>
>> Absent said profiling info one can only speculate. If I had to do it, my
>> money is on the RB tree in the namecache being the problem (combined
>> with the other things). All path lookups are globally serialized on the
>> kernel lock, each name comparison is very expensive due to how memcmp is
>> implemented and after a full fs walk there is presumably way more
>> entries to traverse than usual. Pair that with waiting on other CPUs to
>> finish and I can easily see this being the problem.
>>
>> Some of it can be alleviated with a faster memcmp, see https://marc.info/?l=openbsd-tech&m=173284208231267&w=2
>>
>> But ultimately the real problem is the RB tree.
>>
> 
> If it's that memcmp, this patch would remove it completely from the
> namecache lookup.
> 
> <https://marc.info/?l=openbsd-tech&m=177543192301688&w=2>
> 

Ah sorry. That was about memcpy...