Download raw body.
SoftLRO for ixl(4), bnxt(4) and em(4)
From: Jan Klemkow <jan@openbsd.org>
Subject: Re: SoftLRO for ixl(4), bnxt(4) and em(4)
Date: Fri, 15 Nov 2024 11:30:08 +0100
> On Thu, Nov 07, 2024 at 11:30:26AM GMT, David Gwynne wrote:
>> On Thu, Nov 07, 2024 at 01:10:10AM +0100, Jan Klemkow wrote:
>> > This diff introduces a software solution for TCP Large Receive Offload
>> > (SoftLRO) for network interfaces don't hat hardware support for it.
>> > This is needes at least for newer Intel interfaces as their
>> > documentation said that LRO a.k.a. Receive Side Coalescing (RSC) has to
>> > be done by software.
>> > This diff coalesces TCP segments during the receive interrupt before
>> > queueing them. Thus, our TCP/IP stack has to process less packet
>> > headers per amount of received data.
>> >
>> > I measured receiving performance with Intel XXV710 25 GbE interfaces.
>> > It increased from 6 Gbit/s to 23 Gbit/s.
>> >
>> > Even if we saturate em(4) without any of these technique its also part
>> > this diff. I'm interested if this diff helps to reach 1 Gbit/s on old
>> > or slow hardware.
>> >
>> > I also add bnxt(4) to this diff to increase test coverage. If you want
>> > to tests this implementation with your favorite interface, just replace
>> > the ml_enqueue() call with the new tcp_softlro_enqueue() (as seen
>> > below). It should work with all kind network interfaces.
>> >
>> > Any comments and tests reports are welcome.
>>
>> nice.
>>
>> i would argue this should be ether_softlro_enqueue and put in
>> if_ethersubr.c because it's specific to ethernet interfaces. we don't
>> really have any other type of interface that bundles reception of
>> packets that we can take advantage of like this, and internally it
>> assumes it's pulling ethernet packets apart.
>>
>> aside from that, just a few comments on the code.
>
> I adapted your comments in the diff below.
Hi, I tested SoftLRO patch, and as you said, I saw that TCP receive
performance improved. However, packet forwarding performance has decreased.
I used ipgen (*1), a packet-forwarding benchmark tool developed by IIJ.
This tool measures the highest transfer rate without packet loss.
The document says that benchmarking depends on RFC2544.
*1: https://github.com/iij/ipgen
Before applying SoftLRO, I built the OpenBSD current by the following commit.
https://github.com/OpenBSD/src/commit/e874061e6eeada0b555c78c1394fda050afee12d
My test machine specs:
CPU: core i7 11700
Mem: 32GB
NIC: Intel 710 SFP+ 2 port (ixl)
My test environment:
ipgen tx ---------> ixl0 OpenBSD ixl1 -------> ipgen rx
The ipgen host is a FreeBSD machine. The tx and rx sides are the same process
on the same machine.
ipgen result:
rfc2544 tolerable error rate: 0.0000%
rfc2544 trial duration: 10 sec
rfc2544 pps resolution: 0.0000%
rfc2544 interval: 0 sec
rfc2544 warming duration: 1 sec
framesize|0G 1G 2G 3G 4G 5G 6G 7G 8G 9G 10Gbps
---------+----+----+----+----+----+----+----+----+----+----+
64 |#################### 3809.52Mbps, 7440476/14880952pps, 50.00%
128 |##### 861.49Mbps, 841302/ 8445945pps, 9.96%
512 |################## 3459.16Mbps, 844522/ 2349624pps, 35.94%
1024 |################################### 6915.74Mbps, 844207/ 1197318pps, 70.51%
1280 |################################################## 9846.15Mbps, 961538/ 961538pps, 100.00%
1408 |################################################## 9859.94Mbps, 875350/ 875350pps, 100.00%
1518 |################################################## 9869.95Mbps, 812743/ 812743pps, 100.00%
framesize|0 |1m |2m |3m |4m |5m |6m |7m |8m |9m |10m |11m |12m |13m |14m |15m pps
---------+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
64 |##################################### 7440476/14880952pps, 50.00%
128 |#### 841302/ 8445945pps, 9.96%
512 |#### 844522/ 2349624pps, 35.94%
1024 |#### 844207/ 1197318pps, 70.51%
1280 |#### 961538/ 961538pps, 100.00%
1408 |#### 875350/ 875350pps, 100.00%
1518 |#### 812743/ 812743pps, 100.00%
This result is very good performance.
After applying SoftLRO, ipgen result:
rfc2544 tolerable error rate: 0.0000%
rfc2544 trial duration: 10 sec
rfc2544 pps resolution: 0.0000%
rfc2544 interval: 0 sec
rfc2544 warming duration: 1 sec
framesize|0G 1G 2G 3G 4G 5G 6G 7G 8G 9G 10Gbps
---------+----+----+----+----+----+----+----+----+----+----+
64 |# 121.93Mbps, 238135/14880952pps, 1.60%
128 |######################################## 7987.88Mbps, 7800660/ 8445945pps, 92.36%
512 |##### 892.86Mbps, 217984/ 2349624pps, 9.28%
1024 |########## 1902.07Mbps, 232186/ 1197318pps, 19.39%
1280 |############ 2324.60Mbps, 227012/ 961538pps, 23.61%
1408 |############# 2531.55Mbps, 224747/ 875350pps, 25.68%
1518 |########### 2081.96Mbps, 171439/ 812743pps, 21.09%
framesize|0 |1m |2m |3m |4m |5m |6m |7m |8m |9m |10m |11m |12m |13m |14m |15m pps
---------+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
64 |# 238135/14880952pps, 1.60%
128 |####################################### 7800660/ 8445945pps, 92.36%
512 |# 217984/ 2349624pps, 9.28%
1024 |# 232186/ 1197318pps, 19.39%
1280 |# 227012/ 961538pps, 23.61%
1408 |# 224747/ 875350pps, 25.68%
1518 | 171439/ 812743pps, 21.09%
The ipgen benchmark uses UDP packets. I'm guessing it's the bad
combination with SoftLRO. SoftLRO works only for TCP packets.
I guess checking each packet is just overhead in this case.
I also tested TCP packet forwarding. The ipgen has TCP testing
mode but it cannot use for OpenBSD tesging. It generates the same
sequence number of packets. OpenBSD will never forward such illegal
packets.
I used iperf3 on the FreeBSD machine. My test environment:
iperf3 -----> ixl0 OpenBSD ixl1 -----> iperf3 -s
The `iperf3 -s` process is in a vnet jail, where the protocol stack
is logically separated from the host OS.
Before applying SoftLRO:
% iperf3 -c 192.168.200.10
Connecting to host 192.168.200.10, port 5201
[ 5] local 192.168.100.10 port 60502 connected to 192.168.200.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 346 MBytes 2.87 Gbits/sec 339 1019 KBytes
[ 5] 1.01-2.01 sec 855 MBytes 7.16 Gbits/sec 0 1.61 MBytes
[ 5] 2.01-3.00 sec 932 MBytes 7.91 Gbits/sec 0 1.61 MBytes
[ 5] 3.00-4.05 sec 992 MBytes 7.91 Gbits/sec 0 1.61 MBytes
[ 5] 4.05-5.00 sec 896 MBytes 7.92 Gbits/sec 0 1.61 MBytes
[ 5] 5.00-6.00 sec 944 MBytes 7.91 Gbits/sec 0 1.61 MBytes
[ 5] 6.00-7.00 sec 943 MBytes 7.92 Gbits/sec 0 1.61 MBytes
[ 5] 7.00-8.00 sec 945 MBytes 7.91 Gbits/sec 0 1.61 MBytes
[ 5] 8.00-9.00 sec 944 MBytes 7.92 Gbits/sec 0 1.61 MBytes
[ 5] 9.00-10.00 sec 942 MBytes 7.91 Gbits/sec 0 1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.53 GBytes 7.33 Gbits/sec 339 sender
[ 5] 0.00-10.00 sec 8.53 GBytes 7.33 Gbits/sec receiver
iperf Done.
Reverse direction:
% iperf3 -c 192.168.200.10 -R
Connecting to host 192.168.200.10, port 5201
Reverse mode, remote host 192.168.200.10 is sending
[ 5] local 192.168.100.10 port 63875 connected to 192.168.200.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 805 MBytes 6.71 Gbits/sec
[ 5] 1.01-2.02 sec 776 MBytes 6.45 Gbits/sec
[ 5] 2.02-3.02 sec 765 MBytes 6.42 Gbits/sec
[ 5] 3.02-4.00 sec 753 MBytes 6.42 Gbits/sec
[ 5] 4.00-5.00 sec 768 MBytes 6.42 Gbits/sec
[ 5] 5.00-6.03 sec 785 MBytes 6.40 Gbits/sec
[ 5] 6.03-7.00 sec 741 MBytes 6.42 Gbits/sec
[ 5] 7.00-8.01 sec 770 MBytes 6.43 Gbits/sec
[ 5] 8.01-9.01 sec 767 MBytes 6.42 Gbits/sec
[ 5] 9.01-10.03 sec 782 MBytes 6.43 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.03 sec 7.53 GBytes 6.45 Gbits/sec 0 sender
[ 5] 0.00-10.03 sec 7.53 GBytes 6.45 Gbits/sec receiver
iperf Done.
After applying SoftLRO:
% iperf3 -c 192.168.200.10
Connecting to host 192.168.200.10, port 5201
[ 5] local 192.168.100.10 port 43789 connected to 192.168.200.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 352 MBytes 2.92 Gbits/sec 88 1.02 MBytes
[ 5] 1.01-2.00 sec 701 MBytes 5.93 Gbits/sec 199 1.20 MBytes
[ 5] 2.00-3.01 sec 789 MBytes 6.55 Gbits/sec 0 1.61 MBytes
[ 5] 3.01-4.00 sec 550 MBytes 4.67 Gbits/sec 0 1.61 MBytes
[ 5] 4.00-5.04 sec 488 MBytes 3.93 Gbits/sec 0 1.61 MBytes
[ 5] 5.04-6.01 sec 437 MBytes 3.79 Gbits/sec 0 1.61 MBytes
[ 5] 6.01-7.04 sec 459 MBytes 3.74 Gbits/sec 0 1.61 MBytes
[ 5] 7.04-8.00 sec 430 MBytes 3.75 Gbits/sec 0 1.61 MBytes
[ 5] 8.00-9.00 sec 447 MBytes 3.75 Gbits/sec 0 1.61 MBytes
[ 5] 9.00-10.00 sec 443 MBytes 3.72 Gbits/sec 0 1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.98 GBytes 4.27 Gbits/sec 287 sender
[ 5] 0.00-10.00 sec 4.98 GBytes 4.27 Gbits/sec receiver
iperf Done.
Reverse Direction:
% iperf3 -c 192.168.200.10 -R
Connecting to host 192.168.200.10, port 5201
Reverse mode, remote host 192.168.200.10 is sending
[ 5] local 192.168.100.10 port 50264 connected to 192.168.200.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec
[ 5] 1.00-2.01 sec 861 MBytes 7.13 Gbits/sec
[ 5] 2.01-3.04 sec 830 MBytes 6.76 Gbits/sec
[ 5] 3.04-4.01 sec 562 MBytes 4.88 Gbits/sec
[ 5] 4.01-5.06 sec 494 MBytes 3.93 Gbits/sec
[ 5] 5.06-6.01 sec 402 MBytes 3.57 Gbits/sec
[ 5] 6.01-7.00 sec 422 MBytes 3.55 Gbits/sec
[ 5] 7.00-8.00 sec 422 MBytes 3.55 Gbits/sec
[ 5] 8.00-9.01 sec 422 MBytes 3.53 Gbits/sec
[ 5] 9.01-10.00 sec 420 MBytes 3.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.05 GBytes 4.34 Gbits/sec 270 sender
[ 5] 0.00-10.00 sec 5.05 GBytes 4.34 Gbits/sec receiver
iperf Done.
I'm not sure why TCP packet forwarding performance decreases.
I think SoftLRO will work on TCP packet forwarding.
--
Yuichiro NAITO (naito.yuichiro@gmail.com)
SoftLRO for ixl(4), bnxt(4) and em(4)