> On 3 Feb 2025, at 20:25, Kirill A. Korinsky <kirill@korins.ky> wrote:
> 
> On Mon, 03 Feb 2025 11:44:42 +0100,
> Vitaliy Makkoveev <mvs@openbsd.org> wrote:
>> 
>> First split them by filt_sor*() and filt_soe*() functions for
>> `soread_filtops' and `soexcept_filtops' respectively. The
>> filt_soexcept() path is like filt_sowrite() and the `sb_mtx'
>> serialization is pretty enough, but filt_soread() is more complicated
>> for the connection oriented sockets. It is not yet clean is the
>> concurrent SO_ACCEPTCONN transition safe for select(2) and poll(2), so
>> keep socket lock together with `sb_mtx' lock for that case.
>> 
>> Also get rid of sofilt_*lock() functions. They were temporary made to
>> take `sb_mtx' mutex with shared netlock for inet sockets case or with
>> socket lock for other cases. Now shared netlock is not enough and should
>> be taken together with `so_lock' rwlock. We have special solock_shared()
>> function for that purpose, so use it directly in corresponding
>> filt_sor*() handlers.
>> 
>> Kirill, could you run sys/netinet/tcpthread/ and sys/kern/sosplice on
>> your arm64 machine?
>> 
> 

Thanks for testing. The regress/sys/kern/sosplice was fixed with
the ifq [1] diff I committed today. You already tested it by my private
request. This diff should appear in the most recent snap. The diff from
this thread only adds some performance. 

1. http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/ifq.c.diff?r1=1.55&r2=1.56

> Sure. I've used current snapshot as baseline.
> 
> 1. regress/sys/netinet/tcpthread
> 
> Your diff improves tcpthread in term of sys column time. Significant.
> 
> your diff
> 
>        ==== run-default ====
>        time  ./tcpthread
>        count: connect 171, accept 170, send 118, recv 278, close 170, drop 0
>               10.03 real        24.54 user         5.41 sys
> 
>        ==== run-sock100 ====
>        ulimit -n 500; time  ./tcpthread -n100
>        count: connect 174558, accept 9961, send 286661, recv 65474, close 45805, drop 0
>            0m10.04s real     0m09.19s user     0m14.41s system
> 
>        ==== run-splice ====
>        time  ./tcpthread -n10 -S2
>        count: connect 170624, splice 0, unsplice 0, max 0, idle 0, accept 0, send 0, recv 85097, close 85527, drop 0
>               10.12 real        23.98 user         7.31 sys
> 
>        ==== run-unsplice ====
>        time  ./tcpthread -n10 -S2 -U1
>        count: connect 142843, splice 0, unsplice 0, max 0, idle 0, accept 0, send 0, recv 71286, close 71547, drop 0
>               10.12 real        33.44 user         6.74 sys
> 
>        ==== run-max ====
>        time  ./tcpthread -n10 -S4 -M100 -s2 -r2
>        count: connect 250205, splice 0, unsplice 0, max 0, idle 0, accept 0, send 4138, recv 124616, close 121443, drop 0
>               10.13 real        38.94 user        11.47 sys
> 
>        ==== run-idle ====
>        time  ./tcpthread -n10 -S4 -I100 -s2 -r2
>        count: connect 325965, splice 0, unsplice 0, max 0, idle 0, accept 0, send 325972, recv 0, close 0, drop 0
>               10.13 real        45.97 user         7.95 sys
> 
>        ==== run-drop ====
>        time doas  ./tcpthread -n10 -D3 -o0
>        count: connect 39142, accept 0, send 39492, recv 0, close 0, drop 0
>               10.12 real        49.39 user         0.82 sys
> 
>        ==== run-splicedrop ====
>        time doas  ./tcpthread -n10 -S2 -D3 -o0
>        count: connect 61005, splice 0, unsplice 0, max 0, idle 0, accept 0, send 61727, recv 0, close 0, drop 0
>               10.12 real        48.96 user         1.42 sys
> 
> and the snapshot:
> 
>        ==== run-default ====
>        time  ./tcpthread
>        count: connect 199, accept 198, send 164, recv 355, close 198, drop 0
>               10.04 real        24.73 user         5.17 sys
> 
>        ==== run-sock100 ====
>        ulimit -n 500; time  ./tcpthread -n100
>        count: connect 12079, accept 12079, send 6561, recv 18550, close 12029, drop 0
>            0m10.43s real     0m21.34s user     0m10.07s system
> 
>        ==== run-splice ====
>        time  ./tcpthread -n10 -S2
>        count: connect 39071, splice 129, unsplice 0, max 0, idle 0, accept 1222, send 626064, recv 474297, close 44484, drop 0
>               10.04 real         9.38 user        10.21 sys
> 
>        ==== run-unsplice ====
>        time  ./tcpthread -n10 -S2 -U1
>        count: connect 321093, splice 12, unsplice 749, max 0, idle 0, accept 10, send 52780, recv 11, close 276654, drop 0
>               10.04 real        26.03 user        17.15 sys
> 
>        ==== run-max ====
>        time  ./tcpthread -n10 -S4 -M100 -s2 -r2
>        count: connect 20, splice 6, unsplice 0, max 0, idle 0, accept 9, send 5, recv 12, close 13, drop 0
>               10.10 real        31.41 user        18.78 sys
> 
>        ==== run-idle ====
>        time  ./tcpthread -n10 -S4 -I100 -s2 -r2
>        count: connect 298272, splice 349, unsplice 0, max 0, idle 7, accept 506, send 43897, recv 28, close 273104, drop 0
>               10.04 real        15.28 user        37.83 sys
> 
>        ==== run-drop ====
>        time doas  ./tcpthread -n10 -D3 -o0
>        count: connect 37, accept 10, send 319, recv 6, close 0, drop 10
>               10.02 real         1.92 user        42.07 sys
> 
>        ==== run-splicedrop ====
>        time doas  ./tcpthread -n10 -S2 -D3 -o0
>        count: connect 1821, splice 427, unsplice 0, max 0, idle 0, accept 516, send 53614, recv 1135, close 0, drop 499
>               10.04 real        41.18 user         8.23 sys
> 
> 
> 2 regress/sys/kern/sosplice 
> 
> On snapshot, when I run it as doas make, it fails on:
> 
>        ==== run-udp-splice ====
>        pkill -x tcpbench
>        *** Error 1 in target 'run-udp-splice' (ignored)
>        ./splice -u -4 12345 127.0.0.1 12346 &  rpid=$! ;  tcpbench -u -4 -t 7 -s -p 12346 &  sleep 1 ;  tcpbench -u -4 -t 5 127.0.0.1 ;  kill $rpid ;  pkill -x tcpbench
>        splice: setsockopt rcvbuf: No buffer space available
>        tcpbench: write: Connection refused
>        --- 127.0.0.1 tcpbench statistics ---
>        94208 bytes sent over 0.000 seconds
>        bandwidth min/avg/max/std-dev = inf/0.000/0.000/0.000 Mbps
>        *** Error 1 in perf (Makefile:30 'run-udp-splice')
>        FAILED
>        *** Error 1 in perf (<bsd.regress.mk>:105 'regress': if make -C /usr/src/regress/sys/kern/sosplice/perf run-udp-splice; then  echo -n "SUCCE...)
>        *** Error 2 in /usr/src/regress/sys/kern/sosplice (<bsd.subdir.mk>:48 'all': @for entry in  error tcp udp loop scapy perf; do  set -e; if te...)
> 
> and with your diff it runs till the end.
> 
> As far as I understood the code, it is reads well, but I not think that my
> OK here means anything. Anyway, you have it.
> 
> -- 
> wbr, Kirill