From: Florian Obser Subject: Re: UDP send in parallel To: Alexander Bluhm Cc: tech@openbsd.org Date: Mon, 08 Jan 2024 11:25:18 +0100 On 2024-01-05 23:06 +01, Alexander Bluhm wrote: > Hi, > > Sending UDP packets via datagram socket is MP safe now. Same applies > to raw IPv4 and IPv6, and divert sockets. Switch sosend() from > exclusive net lock to shared net lock in combination with per socket > lock. TCP and GRE still use exclusive net lock. > > Please test this diff if you have applications that run multithreaded > and send lots of UDP packets. So I ran dnsbench and things do not look good. current: 1 client(s) for 120s Queries per second: 25202.641044 2 client(s) for 120s Queries per second: 26331.337927 4 client(s) for 120s Queries per second: 25651.039089 8 client(s) for 120s Queries per second: 26005.807446 10 client(s) for 120s Queries per second: 25824.787767 1 client(s) for 120s with DO bit set Queries per second: 25573.938322 2 client(s) for 120s with DO bit set Queries per second: 25383.326988 4 client(s) for 120s with DO bit set Queries per second: 25284.565750 8 client(s) for 120s with DO bit set Queries per second: 24415.760138 10 client(s) for 120s with DO bit set Queries per second: 25475.886520 With the diff: 1 client(s) for 120s Queries per second: 1620.002447 2 client(s) for 120s Queries per second: 4526.044116 4 client(s) for 120s Queries per second: 7079.784528 8 client(s) for 120s Queries per second: 9210.908147 10 client(s) for 120s Queries per second: 14360.313588 1 client(s) for 120s with DO bit set Queries per second: 5997.335569 2 client(s) for 120s with DO bit set Queries per second: 7326.934034 4 client(s) for 120s with DO bit set Queries per second: 2622.538847 8 client(s) for 120s with DO bit set Queries per second: 8985.788030 10 client(s) for 120s with DO bit set Queries per second: 2591.206131 I ran dnsbench on a different machine. The target is a random amd64 kvm VM I have lying around with 8 cores. I'm starting 8 nsd servers. I'm adding the full log, nsd.conf and some scripts: current: ======== 1 client(s) for 120s Statistics: Queries sent: 3026586 Queries completed: 3024320 (99.93%) Queries lost: 2266 (0.07%) Response codes: NOERROR 3024320 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000122 Queries per second: 25202.641044 Average Latency (s): 0.000258 (min 0.000054, max 3.222699) Latency StdDev (s): 0.002885 2 client(s) for 120s Statistics: Queries sent: 3161990 Queries completed: 3159763 (99.93%) Queries lost: 2227 (0.07%) Response codes: NOERROR 3159763 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000093 Queries per second: 26331.337927 Average Latency (s): 0.000289 (min 0.000052, max 0.374885) Latency StdDev (s): 0.001140 4 client(s) for 120s Statistics: Queries sent: 3080387 Queries completed: 3078129 (99.93%) Queries lost: 2258 (0.07%) Response codes: NOERROR 3078129 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000168 Queries per second: 25651.039089 Average Latency (s): 0.000274 (min 0.000052, max 0.233587) Latency StdDev (s): 0.000842 8 client(s) for 120s Statistics: Queries sent: 3122931 Queries completed: 3120699 (99.93%) Queries lost: 2232 (0.07%) Response codes: NOERROR 3120699 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000081 Queries per second: 26005.807446 Average Latency (s): 0.000281 (min 0.000051, max 0.625339) Latency StdDev (s): 0.001165 10 client(s) for 120s Statistics: Queries sent: 3101371 Queries completed: 3099152 (99.93%) Queries lost: 2219 (0.07%) Response codes: NOERROR 3099152 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.006872 Queries per second: 25824.787767 Average Latency (s): 0.000292 (min 0.000053, max 1.417694) Latency StdDev (s): 0.001889 1 client(s) for 120s with DO bit set Statistics: Queries sent: 3071148 Queries completed: 3068876 (99.93%) Queries lost: 2272 (0.07%) Response codes: NOERROR 3068876 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000133 Queries per second: 25573.938322 Average Latency (s): 0.000251 (min 0.000042, max 0.801752) Latency StdDev (s): 0.001281 2 client(s) for 120s with DO bit set Statistics: Queries sent: 3048263 Queries completed: 3046000 (99.93%) Queries lost: 2263 (0.07%) Response codes: NOERROR 3046000 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000030 Queries per second: 25383.326988 Average Latency (s): 0.000280 (min 0.000055, max 0.366167) Latency StdDev (s): 0.000907 4 client(s) for 120s with DO bit set Statistics: Queries sent: 3036399 Queries completed: 3034151 (99.93%) Queries lost: 2248 (0.07%) Response codes: NOERROR 3034151 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000123 Queries per second: 25284.565750 Average Latency (s): 0.000288 (min 0.000051, max 0.536886) Latency StdDev (s): 0.001050 8 client(s) for 120s with DO bit set Statistics: Queries sent: 2932157 Queries completed: 2929894 (99.92%) Queries lost: 2263 (0.08%) Response codes: NOERROR 2929894 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000114 Queries per second: 24415.760138 Average Latency (s): 0.000277 (min 0.000052, max 2.126242) Latency StdDev (s): 0.002415 10 client(s) for 120s with DO bit set Statistics: Queries sent: 3059310 Queries completed: 3057110 (99.93%) Queries lost: 2200 (0.07%) Response codes: NOERROR 3057110 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000142 Queries per second: 25475.886520 Average Latency (s): 0.000314 (min 0.000058, max 1.122629) Latency StdDev (s): 0.001273 ------------------------------------------------------------------------ with the diff ============= 1 client(s) for 120s Statistics: Queries sent: 196737 Queries completed: 194415 (98.82%) Queries lost: 2322 (1.18%) Response codes: NOERROR 194415 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.009078 Queries per second: 1620.002447 Average Latency (s): 0.002164 (min 0.000060, max 4.914079) Latency StdDev (s): 0.093896 2 client(s) for 120s Statistics: Queries sent: 545388 Queries completed: 543126 (99.59%) Queries lost: 2262 (0.41%) Response codes: NOERROR 543126 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000156 Queries per second: 4526.044116 Average Latency (s): 0.001541 (min 0.000060, max 4.913738) Latency StdDev (s): 0.069487 4 client(s) for 120s Statistics: Queries sent: 851846 Queries completed: 849575 (99.73%) Queries lost: 2271 (0.27%) Response codes: NOERROR 849575 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000121 Queries per second: 7079.784528 Average Latency (s): 0.000986 (min 0.000055, max 4.909502) Latency StdDev (s): 0.050539 8 client(s) for 120s Statistics: Queries sent: 1107606 Queries completed: 1105310 (99.79%) Queries lost: 2296 (0.21%) Response codes: NOERROR 1105310 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000111 Queries per second: 9210.908147 Average Latency (s): 0.000709 (min 0.000056, max 4.911321) Latency StdDev (s): 0.035660 10 client(s) for 120s Statistics: Queries sent: 1725511 Queries completed: 1723240 (99.87%) Queries lost: 2271 (0.13%) Response codes: NOERROR 1723240 (100.00%) Average packet size: request 30, response 348 Run time (s): 120.000165 Queries per second: 14360.313588 Average Latency (s): 0.000469 (min 0.000059, max 4.910076) Latency StdDev (s): 0.025829 1 client(s) for 120s with DO bit set Statistics: Queries sent: 722037 Queries completed: 719681 (99.67%) Queries lost: 2356 (0.33%) Response codes: NOERROR 719681 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000122 Queries per second: 5997.335569 Average Latency (s): 0.000454 (min 0.000054, max 4.915879) Latency StdDev (s): 0.027013 2 client(s) for 120s with DO bit set Statistics: Queries sent: 881534 Queries completed: 879233 (99.74%) Queries lost: 2301 (0.26%) Response codes: NOERROR 879233 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000125 Queries per second: 7326.934034 Average Latency (s): 0.000693 (min 0.000059, max 4.999180) Latency StdDev (s): 0.041681 4 client(s) for 120s with DO bit set Statistics: Queries sent: 317082 Queries completed: 314705 (99.25%) Queries lost: 2377 (0.75%) Response codes: NOERROR 314705 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000129 Queries per second: 2622.538847 Average Latency (s): 0.000791 (min 0.000057, max 4.816053) Latency StdDev (s): 0.031375 8 client(s) for 120s with DO bit set Statistics: Queries sent: 1080608 Queries completed: 1078300 (99.79%) Queries lost: 2308 (0.21%) Response codes: NOERROR 1078300 (100.00%) Average packet size: request 41, response 708 Run time (s): 120.000605 Queries per second: 8985.788030 Average Latency (s): 0.000602 (min 0.000055, max 4.913466) Latency StdDev (s): 0.034349 10 client(s) for 120s with DO bit set Statistics: Queries sent: 313334 Queries completed: 310945 (99.24%) Queries lost: 2389 (0.76%) Response codes: NOERROR 310945 (100.00%) Average packet size: request 41, response 709 Run time (s): 120.000102 Queries per second: 2591.206131 Average Latency (s): 0.000515 (min 0.000065, max 4.206721) Latency StdDev (s): 0.013964 ------------------------------------------------------------------------ $ cat gen_input.sh #! /bin/sh dig +noall +answer +onesoa @k.root-servers.net AXFR . | awk '$4 !~ /NSEC|RRSIG|ZONEMD/ {print $1,$4}' | sort -u ------------------------------------------------------------------------ $ cat run.sh #! /bin/sh RUNTIME=120 for NUMCLIENTS in 1 2 4 8 10; do echo ${NUMCLIENTS} 'client(s)' for ${RUNTIME}s dnsperf -c ${NUMCLIENTS} -l ${RUNTIME} -d dnsperf.input -s $1 | sed -n '/^Statistics/,$p' done for NUMCLIENTS in 1 2 4 8 10; do echo ${NUMCLIENTS} 'client(s)' for ${RUNTIME}s with DO bit set dnsperf -D -c ${NUMCLIENTS} -l ${RUNTIME} -d dnsperf.input -s $1 | sed -n '/^Statistics/,$p' done ------------------------------------------------------------------------ nsd.conf: server: server-count: 8 hide-version: yes verbosity: 1 database: "" # disable database ip-address: xxx remote-control: control-enable: yes control-interface: /var/run/nsd.sock zone: name: "." request-xfr: 192.228.79.201 NOKEY # b.root-servers.net request-xfr: 192.33.4.12 NOKEY # c.root-servers.net request-xfr: 192.5.5.241 NOKEY # f.root-servers.net request-xfr: 192.112.36.4 NOKEY # g.root-servers.net request-xfr: 193.0.14.129 NOKEY # k.root-servers.net request-xfr: 192.0.47.132 NOKEY # xfr.cjr.dns.icann.org request-xfr: 192.0.32.132 NOKEY # xfr.lax.dns.icann.org request-xfr: 2001:500:84::b NOKEY # b.root-servers.net request-xfr: 2001:500:2f::f NOKEY # f.root-servers.net request-xfr: 2001:7fd::1 NOKEY # k.root-servers.net request-xfr: 2620:0:2830:202::132 NOKEY # xfr.cjr.dns.icann.org request-xfr: 2620:0:2d0:202::132 NOKEY # xfr.lax.dns.icann.org -- In my defence, I have been left unsupervised.