From: Visa Hankala Subject: Re: SoftLRO diff for cnmac/octeon To: "Kirill A. Korinsky" Cc: Janne Johansson , tech@openbsd.org Date: Tue, 7 Apr 2026 15:50:22 +0000 On Tue, Mar 31, 2026 at 05:26:21PM +0200, Kirill A. Korinsky wrote: > On Wed, 04 Feb 2026 20:30:13 +0100, > Janne Johansson wrote: > > > > > > The SMALL_KERNEL diff is at > > > > > > > > https://s3.sto3.safedc.net/jj-public-files/small-kernel-cnmac-softlro.diff > > > > > > I can't boot kernel with this patch, it fails as: > > > > > > starting network > > > > > > Trap cause = 2 Frame 0x980000000fd97878 > > > Trap PC 0xffffffff8119f2a4 RA 0xffffffff8119f5ec fault 0x0 > > > cnmac_recv_mbuf+0x134 (77e0000d9f08a,1,980000000fd97a00,980000000fd979fc) ra 0xffffffff8119f5ec sp 0x980000000fd979d0, sz 32 > > > > > > > Thanks for testing, I'll rig something up with wg and see if I can > > provoke the error with it. > > > > Here a bit reworked version of your last diff. > > On my ER-4 with this diff I see: > > % iperf3-darwin -t 30 -c 172.31.3.1 > Connecting to host 172.31.3.1, port 5201 > [ 5] local 172.31.3.11 port 49306 connected to 172.31.3.1 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd RTT > [ 5] 0.00-1.00 sec 84.8 MBytes 710 Mbits/sec 5792 2.61 MBytes 7ms > [ 5] 1.00-2.00 sec 102 MBytes 862 Mbits/sec 7240 1004 KBytes 9ms > [ 5] 2.00-3.00 sec 106 MBytes 890 Mbits/sec 0 1.06 MBytes 9ms > [ 5] 3.00-4.00 sec 101 MBytes 844 Mbits/sec 0 1.16 MBytes 12ms > [ 5] 4.00-5.00 sec 105 MBytes 878 Mbits/sec 0 1.27 MBytes 9ms > [ 5] 5.00-6.00 sec 99.8 MBytes 837 Mbits/sec 8688 1020 KBytes 8ms > [ 5] 6.00-7.00 sec 105 MBytes 879 Mbits/sec 0 1.10 MBytes 10ms > [ 5] 7.00-8.00 sec 104 MBytes 869 Mbits/sec 0 1.18 MBytes 13ms > [ 5] 8.00-9.00 sec 106 MBytes 886 Mbits/sec 0 1.24 MBytes 5ms > [ 5] 9.00-10.00 sec 105 MBytes 883 Mbits/sec 0 1.28 MBytes 9ms > [ 5] 10.00-11.00 sec 108 MBytes 903 Mbits/sec 0 1.31 MBytes 13ms > [ 5] 11.00-12.00 sec 105 MBytes 878 Mbits/sec 10136 1011 KBytes 8ms > [ 5] 12.00-13.00 sec 103 MBytes 864 Mbits/sec 0 1.10 MBytes 10ms > [ 5] 13.00-14.00 sec 103 MBytes 862 Mbits/sec 0 1.18 MBytes 8ms > [ 5] 14.00-15.00 sec 102 MBytes 851 Mbits/sec 0 1.24 MBytes 11ms > [ 5] 15.00-16.00 sec 99.3 MBytes 835 Mbits/sec 0 1.28 MBytes 7ms > [ 5] 16.00-17.00 sec 104 MBytes 868 Mbits/sec 0 1.31 MBytes 5ms > [ 5] 17.00-18.00 sec 95.1 MBytes 800 Mbits/sec 1448 549 KBytes 4ms > [ 5] 18.00-19.00 sec 90.2 MBytes 756 Mbits/sec 0 694 KBytes 7ms > [ 5] 19.00-20.00 sec 93.1 MBytes 781 Mbits/sec 0 867 KBytes 10ms > [ 5] 20.00-21.00 sec 97.2 MBytes 815 Mbits/sec 0 1015 KBytes 10ms > [ 5] 21.00-22.00 sec 107 MBytes 902 Mbits/sec 0 1.13 MBytes 11ms > [ 5] 22.00-23.00 sec 101 MBytes 847 Mbits/sec 0 1.25 MBytes 10ms > [ 5] 23.00-24.00 sec 91.3 MBytes 767 Mbits/sec 2896 557 KBytes 2ms > [ 5] 24.00-25.00 sec 82.6 MBytes 691 Mbits/sec 0 660 KBytes 7ms > [ 5] 25.00-26.00 sec 90.8 MBytes 763 Mbits/sec 0 783 KBytes 1ms > [ 5] 26.00-27.00 sec 99.4 MBytes 832 Mbits/sec 0 949 KBytes 9ms > [ 5] 27.00-28.00 sec 97.8 MBytes 822 Mbits/sec 0 1.06 MBytes 1ms > [ 5] 28.00-29.01 sec 105 MBytes 878 Mbits/sec 0 1.19 MBytes 11ms > [ 5] 29.01-30.00 sec 109 MBytes 916 Mbits/sec 0 1.31 MBytes 13ms > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-30.00 sec 2.93 GBytes 839 Mbits/sec 36200 sender > [ 5] 0.00-30.16 sec 2.93 GBytes 834 Mbits/sec receiver > > iperf Done. > > So far it seems stable enough to actually use it. > > The diff: > > Index: sys/arch/octeon/dev/if_cnmac.c > =================================================================== > RCS file: /home/cvs/src/sys/arch/octeon/dev/if_cnmac.c,v > diff -u -p -r1.86 if_cnmac.c > --- sys/arch/octeon/dev/if_cnmac.c 20 May 2024 23:13:33 -0000 1.86 > +++ sys/arch/octeon/dev/if_cnmac.c 31 Mar 2026 14:32:09 -0000 > @@ -55,6 +55,11 @@ > #include > #include > #include > +#ifndef SMALL_KERNEL > +#include > +#include > +#include > +#endif > > #if NBPFILTER > 0 > #include > @@ -307,6 +312,9 @@ cnmac_attach(struct device *parent, stru > ifp->if_softc = sc; > ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; > ifp->if_xflags = IFXF_MPSAFE; > +#ifndef SMALL_KERNEL > + ifp->if_xflags |= IFXF_LRO; > +#endif > ifp->if_ioctl = cnmac_ioctl; > ifp->if_qstart = cnmac_start; > ifp->if_watchdog = cnmac_watchdog; > @@ -315,6 +323,9 @@ cnmac_attach(struct device *parent, stru > > ifp->if_capabilities = IFCAP_VLAN_MTU | IFCAP_CSUM_TCPv4 | > IFCAP_CSUM_UDPv4 | IFCAP_CSUM_TCPv6 | IFCAP_CSUM_UDPv6; > +#ifndef SMALL_KERNEL > + ifp->if_capabilities |= IFCAP_LRO; > +#endif > > cn30xxgmx_set_filter(sc->sc_gmx_port); > > @@ -1214,6 +1225,9 @@ cnmac_recv(struct cnmac_softc *sc, uint6 > struct mbuf *m; > uint64_t word2; > int nmbuf = 0; > +#ifndef SMALL_KERNEL > + struct ether_extracted ext; > +#endif Please move this definition next to ether_extract_headers() inside the if block, to reduce #ifndef'ing. > > word2 = work[2]; > > @@ -1246,7 +1260,19 @@ cnmac_recv(struct cnmac_softc *sc, uint6 > M_TCP_CSUM_IN_OK | M_UDP_CSUM_IN_OK; > } > > - ml_enqueue(ml, m); > +#ifndef SMALL_KERNEL > + if (__predict_true(ISSET(ifp->if_xflags, IFXF_LRO)) && > + __predict_true(!ISSET(word2, PIP_WQE_WORD2_IP_NI)) && I would leave these branch hints out as they mainly add visual noise here. Also, the predict true hint does not seem correct with IFXF_LRO because it is a valid admin choice to not use LRO. With the above tweaks, OK visa@ after release. Tested with EdgeRouter Lite (CN5020) and EdgeRouter Pro (CN6120). > + ISSET(word2, PIP_WQE_WORD2_IP_TU) && > + !ISSET(word2, PIP_WQE_WORD2_IP_FR | PIP_WQE_WORD2_IP_LE)) { > + ether_extract_headers(m, &ext); > + if (ext.tcp != NULL) > + tcp_softlro_glue(ml, m, ifp); > + else > + ml_enqueue(ml, m); > + } else > +#endif > + ml_enqueue(ml, m); > > return nmbuf; > > > > -- > wbr, Kirill >