Index | Thread | Search

From:
YASUOKA Masahiko <yasuoka@openbsd.org>
Subject:
Re: bge/bnx/iavf/igc/ix/ixl/ngbe/pcn: ifq_restart() fix
To:
stsp@stsp.name
Cc:
tech@openbsd.org
Date:
Wed, 25 Jun 2025 10:33:52 +0900

Download raw body.

Thread
  • YASUOKA Masahiko:

    bge/bnx/iavf/igc/ix/ixl/ngbe/pcn: ifq_restart() fix

  • Dmitry Bogdan:

    bge/bnx/iavf/igc/ix/ixl/ngbe/pcn: ifq_restart() fix

  • I'd like to know the problem clearly for the future debug.
    
    On Fri, 20 Jun 2025 12:12:14 +0200
    Stefan Sperling <stsp@stsp.name> wrote:
    > A bug has been fixed by yasuaok@ in vmx(4) where the driver was
    > calling ifq_restart() without actually having made any space on
    > a full Tx ring. Calling ifq_restart() in this case can lead to
    > a condition where the interface gets stuck in OACTIVE until the
    > interface is reset with ifconfig.
    
    On vmx(4), if_vmx.c,v 1.90 move the chunk
    
                   if (free <= NTXSEGS) {
                           ifq_set_oactive(ifq);
    
    to the middle of the loop in vmxnet3_start().  So it was stuck without
    OACTIVE.  tx could stop following conditions.
    
    - start() isn't called because the queue length >= the max queue length
    - tx ring is empty
    (I put the whole scenario at the end of the message)
    
    > I could trigger the same bug on ice(4) with iperf as follows:
    >   for i in `seq 5`; do (iperf -l0 -t 0 -c 2001:db8::1 -u &) ; done
    > This needs another machine which runs iperf -u -s (and -V if using IPv6).
    
    ice was stuck with OACITVE.  Kevin reported ix was stuck with OACTIVE.
    
    These drivers must be stuck by a different scenario than vmx because
    they set OACTIVE at the beginning of the loop in drv_start().
    
    A scenario I imagine is:
    
      1. tx full in drv_start()
         - set oactive
      2. interrupt happen.  drv_txeof() is called
         - no tx is consumed
         - ifq_restart() but 1. is not finished so the task is queued in
           ifq_serialize()
      3. returned from 1. ifq_restart() queued in 2. is called.
         - clear oactive
         - drv_start() is called
         - set oactive again
         - (do nothing for tx ring)
      4. interrput happen.  drv_txeof() is called
         - all tx are consumed
         - ifq_restart() but if 3. is not finished, the task is *not*
           queued (since 3. is also ifq_restart())
    
      -> tx stucked
        - the tx ring is empty
        - oactive is set
          - then drv_start() is not called (tx is kept empty)
    
    If you know another or the real scenario, I'd like to know that.
    
    * * *
    The vmx scenario:
    
      1. a tx queue becomes oactive
      2. txtintr is called, but no ring is consumed
      3. the oactive is cleared at the bottom of txintr()
      4. start() also be called
      5. txintr is called, all tx ring are consumed
    
    In the period between 1. and 3, the start() is never called because
    the queue is oactive.  In the period between 3 and 5, calling start()
    (including 4.) doesn't put a new packet because tx ring is still full.
    If many packets are sent out and the queue length reaches the maxqlen
    until 5, the problem happens.  Because there is no packet on the tx
    ring  and a new packets will be dropped without calling start() since
    the queue is full.
    
    
  • YASUOKA Masahiko:

    bge/bnx/iavf/igc/ix/ixl/ngbe/pcn: ifq_restart() fix

  • Dmitry Bogdan:

    bge/bnx/iavf/igc/ix/ixl/ngbe/pcn: ifq_restart() fix