From: Otto Moerbeek <otto@drijf.net>
Subject: Re: UDP send in parallel
To: Alexander Bluhm <alexander.bluhm@gmx.net>, tech@openbsd.org
Date: Mon, 8 Jan 2024 13:02:24 +0100

On Mon, Jan 08, 2024 at 11:25:18AM +0100, Florian Obser wrote:

> On 2024-01-05 23:06 +01, Alexander Bluhm <alexander.bluhm@gmx.net> wrote:
> > Hi,
> >
> > Sending UDP packets via datagram socket is MP safe now.  Same applies
> > to raw IPv4 and IPv6, and divert sockets.  Switch sosend() from
> > exclusive net lock to shared net lock in combination with per socket
> > lock.  TCP and GRE still use exclusive net lock.
> >
> > Please test this diff if you have applications that run multithreaded
> > and send lots of UDP packets.
> 
> So I ran dnsbench and things do not look good.

I ran my tests wit pdns_recursor and dnsperf using 10 clients on
another machine (which is running debian). That might explain why I
did not spot a performance regression.

	-Otto
> 
> current:
> 1 client(s) for 120s
>   Queries per second:   25202.641044
> 2 client(s) for 120s
>   Queries per second:   26331.337927
> 4 client(s) for 120s
>   Queries per second:   25651.039089
> 8 client(s) for 120s
>   Queries per second:   26005.807446
> 10 client(s) for 120s
>   Queries per second:   25824.787767
> 1 client(s) for 120s with DO bit set
>   Queries per second:   25573.938322
> 2 client(s) for 120s with DO bit set
>   Queries per second:   25383.326988
> 4 client(s) for 120s with DO bit set
>   Queries per second:   25284.565750
> 8 client(s) for 120s with DO bit set
>   Queries per second:   24415.760138
> 10 client(s) for 120s with DO bit set
>   Queries per second:   25475.886520
> 
> With the diff:
> 1 client(s) for 120s
>   Queries per second:   1620.002447
> 2 client(s) for 120s
>   Queries per second:   4526.044116
> 4 client(s) for 120s
>   Queries per second:   7079.784528
> 8 client(s) for 120s
>   Queries per second:   9210.908147
> 10 client(s) for 120s
>   Queries per second:   14360.313588
> 1 client(s) for 120s with DO bit set
>   Queries per second:   5997.335569
> 2 client(s) for 120s with DO bit set
>   Queries per second:   7326.934034
> 4 client(s) for 120s with DO bit set
>   Queries per second:   2622.538847
> 8 client(s) for 120s with DO bit set
>   Queries per second:   8985.788030
> 10 client(s) for 120s with DO bit set
>   Queries per second:   2591.206131
> 
> I ran dnsbench on a different machine. The target is a random amd64 kvm
> VM I have lying around with 8 cores. I'm starting 8 nsd servers.
> 
> I'm adding the full log, nsd.conf and some scripts:
> 
> current:
> ========
> 1 client(s) for 120s
> Statistics:
> 
>   Queries sent:         3026586
>   Queries completed:    3024320 (99.93%)
>   Queries lost:         2266 (0.07%)
> 
>   Response codes:       NOERROR 3024320 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000122
>   Queries per second:   25202.641044
> 
>   Average Latency (s):  0.000258 (min 0.000054, max 3.222699)
>   Latency StdDev (s):   0.002885
> 
> 2 client(s) for 120s
> Statistics:
> 
>   Queries sent:         3161990
>   Queries completed:    3159763 (99.93%)
>   Queries lost:         2227 (0.07%)
> 
>   Response codes:       NOERROR 3159763 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000093
>   Queries per second:   26331.337927
> 
>   Average Latency (s):  0.000289 (min 0.000052, max 0.374885)
>   Latency StdDev (s):   0.001140
> 
> 4 client(s) for 120s
> Statistics:
> 
>   Queries sent:         3080387
>   Queries completed:    3078129 (99.93%)
>   Queries lost:         2258 (0.07%)
> 
>   Response codes:       NOERROR 3078129 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000168
>   Queries per second:   25651.039089
> 
>   Average Latency (s):  0.000274 (min 0.000052, max 0.233587)
>   Latency StdDev (s):   0.000842
> 
> 8 client(s) for 120s
> Statistics:
> 
>   Queries sent:         3122931
>   Queries completed:    3120699 (99.93%)
>   Queries lost:         2232 (0.07%)
> 
>   Response codes:       NOERROR 3120699 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000081
>   Queries per second:   26005.807446
> 
>   Average Latency (s):  0.000281 (min 0.000051, max 0.625339)
>   Latency StdDev (s):   0.001165
> 
> 10 client(s) for 120s
> Statistics:
> 
>   Queries sent:         3101371
>   Queries completed:    3099152 (99.93%)
>   Queries lost:         2219 (0.07%)
> 
>   Response codes:       NOERROR 3099152 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.006872
>   Queries per second:   25824.787767
> 
>   Average Latency (s):  0.000292 (min 0.000053, max 1.417694)
>   Latency StdDev (s):   0.001889
> 
> 1 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         3071148
>   Queries completed:    3068876 (99.93%)
>   Queries lost:         2272 (0.07%)
> 
>   Response codes:       NOERROR 3068876 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000133
>   Queries per second:   25573.938322
> 
>   Average Latency (s):  0.000251 (min 0.000042, max 0.801752)
>   Latency StdDev (s):   0.001281
> 
> 2 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         3048263
>   Queries completed:    3046000 (99.93%)
>   Queries lost:         2263 (0.07%)
> 
>   Response codes:       NOERROR 3046000 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000030
>   Queries per second:   25383.326988
> 
>   Average Latency (s):  0.000280 (min 0.000055, max 0.366167)
>   Latency StdDev (s):   0.000907
> 
> 4 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         3036399
>   Queries completed:    3034151 (99.93%)
>   Queries lost:         2248 (0.07%)
> 
>   Response codes:       NOERROR 3034151 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000123
>   Queries per second:   25284.565750
> 
>   Average Latency (s):  0.000288 (min 0.000051, max 0.536886)
>   Latency StdDev (s):   0.001050
> 
> 8 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         2932157
>   Queries completed:    2929894 (99.92%)
>   Queries lost:         2263 (0.08%)
> 
>   Response codes:       NOERROR 2929894 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000114
>   Queries per second:   24415.760138
> 
>   Average Latency (s):  0.000277 (min 0.000052, max 2.126242)
>   Latency StdDev (s):   0.002415
> 
> 10 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         3059310
>   Queries completed:    3057110 (99.93%)
>   Queries lost:         2200 (0.07%)
> 
>   Response codes:       NOERROR 3057110 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000142
>   Queries per second:   25475.886520
> 
>   Average Latency (s):  0.000314 (min 0.000058, max 1.122629)
>   Latency StdDev (s):   0.001273
> 
> ------------------------------------------------------------------------
> 
> with the diff
> =============
> 
> 1 client(s) for 120s
> Statistics:
> 
>   Queries sent:         196737
>   Queries completed:    194415 (98.82%)
>   Queries lost:         2322 (1.18%)
> 
>   Response codes:       NOERROR 194415 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.009078
>   Queries per second:   1620.002447
> 
>   Average Latency (s):  0.002164 (min 0.000060, max 4.914079)
>   Latency StdDev (s):   0.093896
> 
> 2 client(s) for 120s
> Statistics:
> 
>   Queries sent:         545388
>   Queries completed:    543126 (99.59%)
>   Queries lost:         2262 (0.41%)
> 
>   Response codes:       NOERROR 543126 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000156
>   Queries per second:   4526.044116
> 
>   Average Latency (s):  0.001541 (min 0.000060, max 4.913738)
>   Latency StdDev (s):   0.069487
> 
> 4 client(s) for 120s
> Statistics:
> 
>   Queries sent:         851846
>   Queries completed:    849575 (99.73%)
>   Queries lost:         2271 (0.27%)
> 
>   Response codes:       NOERROR 849575 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000121
>   Queries per second:   7079.784528
> 
>   Average Latency (s):  0.000986 (min 0.000055, max 4.909502)
>   Latency StdDev (s):   0.050539
> 
> 8 client(s) for 120s
> Statistics:
> 
>   Queries sent:         1107606
>   Queries completed:    1105310 (99.79%)
>   Queries lost:         2296 (0.21%)
> 
>   Response codes:       NOERROR 1105310 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000111
>   Queries per second:   9210.908147
> 
>   Average Latency (s):  0.000709 (min 0.000056, max 4.911321)
>   Latency StdDev (s):   0.035660
> 
> 10 client(s) for 120s
> Statistics:
> 
>   Queries sent:         1725511
>   Queries completed:    1723240 (99.87%)
>   Queries lost:         2271 (0.13%)
> 
>   Response codes:       NOERROR 1723240 (100.00%)
>   Average packet size:  request 30, response 348
>   Run time (s):         120.000165
>   Queries per second:   14360.313588
> 
>   Average Latency (s):  0.000469 (min 0.000059, max 4.910076)
>   Latency StdDev (s):   0.025829
> 
> 1 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         722037
>   Queries completed:    719681 (99.67%)
>   Queries lost:         2356 (0.33%)
> 
>   Response codes:       NOERROR 719681 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000122
>   Queries per second:   5997.335569
> 
>   Average Latency (s):  0.000454 (min 0.000054, max 4.915879)
>   Latency StdDev (s):   0.027013
> 
> 2 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         881534
>   Queries completed:    879233 (99.74%)
>   Queries lost:         2301 (0.26%)
> 
>   Response codes:       NOERROR 879233 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000125
>   Queries per second:   7326.934034
> 
>   Average Latency (s):  0.000693 (min 0.000059, max 4.999180)
>   Latency StdDev (s):   0.041681
> 
> 4 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         317082
>   Queries completed:    314705 (99.25%)
>   Queries lost:         2377 (0.75%)
> 
>   Response codes:       NOERROR 314705 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000129
>   Queries per second:   2622.538847
> 
>   Average Latency (s):  0.000791 (min 0.000057, max 4.816053)
>   Latency StdDev (s):   0.031375
> 
> 8 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         1080608
>   Queries completed:    1078300 (99.79%)
>   Queries lost:         2308 (0.21%)
> 
>   Response codes:       NOERROR 1078300 (100.00%)
>   Average packet size:  request 41, response 708
>   Run time (s):         120.000605
>   Queries per second:   8985.788030
> 
>   Average Latency (s):  0.000602 (min 0.000055, max 4.913466)
>   Latency StdDev (s):   0.034349
> 
> 10 client(s) for 120s with DO bit set
> Statistics:
> 
>   Queries sent:         313334
>   Queries completed:    310945 (99.24%)
>   Queries lost:         2389 (0.76%)
> 
>   Response codes:       NOERROR 310945 (100.00%)
>   Average packet size:  request 41, response 709
>   Run time (s):         120.000102
>   Queries per second:   2591.206131
> 
>   Average Latency (s):  0.000515 (min 0.000065, max 4.206721)
>   Latency StdDev (s):   0.013964
> 
> 
> ------------------------------------------------------------------------
> 
> $ cat gen_input.sh
> #! /bin/sh
> 
> dig +noall +answer +onesoa @k.root-servers.net AXFR .  |
>     awk '$4 !~ /NSEC|RRSIG|ZONEMD/ {print $1,$4}' | sort -u
> 
> ------------------------------------------------------------------------
> 
> $ cat run.sh
> #! /bin/sh
> 
> RUNTIME=120
> 
> for NUMCLIENTS in 1 2 4 8 10; do
> 	echo ${NUMCLIENTS} 'client(s)' for ${RUNTIME}s
> 	dnsperf -c ${NUMCLIENTS} -l ${RUNTIME} -d dnsperf.input -s $1 |
> 	    sed -n '/^Statistics/,$p'
> done
> 
> for NUMCLIENTS in 1 2 4 8 10; do
> 	echo ${NUMCLIENTS} 'client(s)' for ${RUNTIME}s with DO bit set
> 	dnsperf -D -c ${NUMCLIENTS} -l ${RUNTIME} -d dnsperf.input -s $1 |
> 	    sed -n '/^Statistics/,$p'
> done
> 
> 
> ------------------------------------------------------------------------
> 
> nsd.conf:
> 
> server:
> 	server-count: 8
> 	hide-version: yes
> 	verbosity: 1
> 	database: "" # disable database
> 	ip-address: xxx
> remote-control:
> 	control-enable: yes
> 	control-interface: /var/run/nsd.sock
> zone:
> 	name: "."
> 	request-xfr: 192.228.79.201 NOKEY # b.root-servers.net
> 	request-xfr: 192.33.4.12 NOKEY    # c.root-servers.net
> 	request-xfr: 192.5.5.241 NOKEY    # f.root-servers.net
> 	request-xfr: 192.112.36.4 NOKEY   # g.root-servers.net
> 	request-xfr: 193.0.14.129 NOKEY   # k.root-servers.net
> 	request-xfr: 192.0.47.132 NOKEY   # xfr.cjr.dns.icann.org
> 	request-xfr: 192.0.32.132 NOKEY   # xfr.lax.dns.icann.org
> 	request-xfr: 2001:500:84::b NOKEY # b.root-servers.net
> 	request-xfr: 2001:500:2f::f NOKEY # f.root-servers.net
> 	request-xfr: 2001:7fd::1 NOKEY    # k.root-servers.net
> 	request-xfr: 2620:0:2830:202::132 NOKEY  # xfr.cjr.dns.icann.org
> 	request-xfr: 2620:0:2d0:202::132 NOKEY  # xfr.lax.dns.icann.org
> 
> 
> -- 
> In my defence, I have been left unsupervised.
>