From: Alexander Bluhm <alexander.bluhm@gmx.net>
Subject: TSO with small packets
To: tech@openbsd.org
Date: Fri, 9 Feb 2024 11:37:32 +0100

Hi,

After a lot of debugging and testing with Hrvoje and mglocker@, we
have found the cause for occasional watchdog timeouts with em TSO
diff.

In this setup TCP packets are routed from ix(4) with LRO to em(4)
with TSO.  It happens that ix hardware coalesces packets that are
in total smaller than MTU.  In real live this is rare as either you
have large TCP packets in a row, or small TCP packets from time to
time.

Small packets that went though LRO have the TSO bit set.  But TSO
is not necessary as only few small TCP packets have been reassembled.
This confuses em hardware.  So clear TSO flag during interface
output path in that case.

This is only relevant when forwarding to hardware TSO.  Our stack
does not generate such packets and can handle them.  Diff below
prevents them to reach hardware TSO.

ok?

bluhm

Index: netinet/tcp_output.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/sys/netinet/tcp_output.c,v
diff -u -p -r1.141 tcp_output.c
--- netinet/tcp_output.c	26 Nov 2023 22:08:10 -0000	1.141
+++ netinet/tcp_output.c	8 Feb 2024 21:17:50 -0000
@@ -1360,6 +1360,12 @@ tcp_if_output_tso(struct ifnet *ifp, str
 	/* caller must fail later or fragment */
 	if (!ISSET((*mp)->m_pkthdr.csum_flags, M_TCP_TSO))
 		return 0;
+	/* send without hardware TSO if interface can handle packet size */
+	if ((*mp)->m_pkthdr.len <= mtu) {
+		CLR((*mp)->m_pkthdr.csum_flags, M_TCP_TSO);
+		return 0;
+	}
+	/* fragment or fail if interface cannot handle size after chopping */
 	if ((*mp)->m_pkthdr.ph_mss > mtu) {
 		CLR((*mp)->m_pkthdr.csum_flags, M_TCP_TSO);
 		return 0;