From: Mark Kettenis <mark.kettenis@xs4all.nl>
Subject: Re: sys/octeon: fix PCIe config tag layout
To: Kirill A. Korinsky <kirill@korins.ky>
Cc: tech@openbsd.org, visa@openbsd.org
Date: Mon, 20 Apr 2026 22:51:23 +0200

> Date: Mon, 20 Apr 2026 22:38:55 +0200
> From: Kirill A. Korinsky <kirill@korins.ky>
> 
> tech@, visa@,
> 
> Octeon PCIe config-space MMIO uses a 12-bit register field; function,
> device, and bus begin at bits 12, 15, and 20.
> 
> octpcie_make_tag() and octpcie_decompose_tag() used the conventional PCI
> tag layout instead, so config accesses to non-zero device or function
> numbers used the wrong MMIO offset.
> 
> On SRX300 this breaks enumeration of the second Broadcom switch function
> at 0:0:1, which reads back garbage until the tag layout is corrected.
> 
> FreeBSD had Cavium’s SDK header:
> https://github.com/freebsd/freebsd-src/blob/release/13.5.0/sys/contrib/octeon-sdk/cvmx-pcie.h#L89-L93
> where it states:
> 
>     89          uint64_t    bus             : 8;    /* Target bus number sent in the ID in the request. */
>     90          uint64_t    dev             : 5;    /* Target device number sent in the ID in the request. Note that Dev must be
>     91                                                  zero for type 0 configuration requests. */
>     92          uint64_t    func            : 3;    /* Target function number sent in the ID in the request. */
>     93          uint64_t    reg             : 12;   /* Selects a register in the configuration space of the target. */
> 
> which I read as:
>  - reg in bits 11:0
>  - func in bits 14:12
>  - dev in bits 19:15
>  - bus in bits 27:20
> 
> On SRX300 it changes dmesg as:
> 
>  @@ -59,6 +59,7 @@
>   octpcie0 at iobus0: 3 ports
>   pci0 at octpcie0 bus 0
>   vendor "Broadcom", unknown product 0x8343 (class network subclass ethernet, rev 0x01) at pci0 dev 0 function 0 not configured
>  +vendor "Broadcom", unknown product 0x8343 (class network subclass ethernet, rev 0x01) at pci0 dev 0 function 1 not configured
>   octpcie0 port 1: link timeout
>   octpcie0 port 2: reset timeout
>   umass0 at uhub1 port 1 configuration 1 interface 0 "Swissbit USB Flash Module" rev 2.00/1.00 addr 2
> 
> On ER4 I don see any change in dmesg.
> 
> Ok?

That is the stande PCIe ECAM layout.  In principle the tags are just
tags and how we encode/decode bus, device and function doesn't matter.
But octpcie_conf_read() and octpcie_conf_write() use the tag as an
offset.

Doing it this way makes sense.

ok kettenis@

> Index: sys/arch/octeon/dev/octpcie.c
> ===================================================================
> RCS file: /home/cvs/src/sys/arch/octeon/dev/octpcie.c,v
> diff -u -p -r1.2 octpcie.c
> --- sys/arch/octeon/dev/octpcie.c	22 Sep 2019 04:43:24 -0000	1.2
> +++ sys/arch/octeon/dev/octpcie.c	20 Apr 2026 11:04:57 -0000
> @@ -723,18 +723,18 @@ octpcie_bus_maxdevs(void *v, int busno)
>  pcitag_t
>  octpcie_make_tag(void *unused, int b, int d, int f)
>  {
> -	return (b << 16) | (d << 11) | (f << 8);
> +	return (b << 20) | (d << 15) | (f << 12);
>  }
>  
>  void
>  octpcie_decompose_tag(void *unused, pcitag_t tag, int *bp, int *dp, int *fp)
>  {
>  	if (bp != NULL)
> -		*bp = (tag >> 16) & 0xff;
> +		*bp = (tag >> 20) & 0xff;
>  	if (dp != NULL)
> -		*dp = (tag >> 11) & 0x1f;
> +		*dp = (tag >> 15) & 0x1f;
>  	if (fp != NULL)
> -		*fp = (tag >> 8) & 0x7;
> +		*fp = (tag >> 12) & 0x7;
>  }
>  
>  int
> 
> -- 
> wbr, Kirill
> 
>