From: Mark Kettenis <mark.kettenis@xs4all.nl>
Subject: Re: mbuf dma 64 bit
To: Alexander Bluhm <bluhm@openbsd.org>
Cc: tech@openbsd.org, beck@obtuse.com
Date: Thu, 05 Mar 2026 13:32:40 +0100

> Date: Wed, 4 Mar 2026 23:02:54 +0100
> From: Alexander Bluhm <bluhm@openbsd.org>

Hey,

> On Fri, Feb 13, 2026 at 11:37:27AM +0100, Mark Kettenis wrote:
> > Getting the driver bits in will help the bounce buffers.  And I don't
> > think the rest of this diff should be considered before we have bounce
> > buffers.
> 
> Driver bits are in.  The idea of my diff is to mark network drivers
> that can do 64 bit mbufs.  If all interfaces during attach have
> this feature, we allocate mbufs in high memory.
> 
> Problematic are hotplug PCI, CardBus and Thunderbolt busses which
> attach devices that do not support 64 bit DMA.  Machines running
> i386 or with less than 2 GB RAM are not affected.  USB ethernet or
> USB wifi hotplug is not affected.
> 
> What options do we have to get this in?
> 1. Ignore these machines, devices won't work.

Where "won't work" means you'll get a panic or maybe even random
memory corruption.  I'd really don't like that.

> 2. Detect the problem during ifconfig up and print an error.
>    Maybe it works after a reboot if the device is attached during boot.

That is indeed an option.

> 3. Wait for mbuf bounce buffer diff for unsupported devices.

The bounce buffer implementation is generic, not just for mbufs.  But
it only works if drivers use bus_dmamap_sync(9) correctly.  A lot of
drivers do, but it'll take some time to get them all fixed.

> 4. Wait until someone implements IOMMU.  But this is not
>    available for all machines.

Well, acpidmar(4) works on both Intel and AMD machines.  It needs a
few more tweaks to make it better, and more testing.

> Question is how relevant are problematic setups for our user base.
> Can we risk to do 1 or 2?
> 
> Is the mbuf bounce buffer implementation nearly finished and we can
> commit it and do 3?

Yes, at least for amd64.  It has been in snaps for a few days but was removed again.  I fixed an iwx(4) bug that was uncovered during those few days.

But we need a plan.  It is March so we don't have a lot of time left
before the next release.  We should be focussing on fixing bugs, not
introducing new ones.  So I think all this work on getting around DMA
limits (including bounce buffers and IOMMU stuff) is post release
stuff now.  But we should be ready to move forward as soon as that's
done.  I'm happy to commit the bounce buffer diff quickly after
release if I get some ok's for it.

However, I agree with Theo that we should focus on the buffer cache
simplification first.  I asked some questions about that to Bob on how
to handle other architectures with DMA limits, but sadly didn't get an
answer. So Bob, you can consider this a ping for that.

Cheers,

Mark

> Index: arch/amd64/amd64/autoconf.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/arch/amd64/amd64/autoconf.c,v
> diff -u -p -r1.59 autoconf.c
> --- arch/amd64/amd64/autoconf.c	12 Nov 2025 10:00:27 -0000	1.59
> +++ arch/amd64/amd64/autoconf.c	4 Mar 2026 14:50:42 -0000
> @@ -108,6 +108,23 @@ unmap_startup(void)
>  	} while (p < (vaddr_t)endboot);
>  }
>  
> +void
> +mbuf_dma_64bit_enable(void)
> +{
> +	struct ifnet *ifp;
> +
> +	TAILQ_FOREACH(ifp, &ifnetlist, if_list) {
> +		if (!ISSET(ifp->if_xflags, IFXF_MBUF_64BIT)) {
> +			printf("%s: restrict all mbufs to low memory\n",
> +			    ifp->if_xname);
> +			return;
> +		}
> +	}
> +
> +	printf("enable mbufs in high memory\n");
> +	m_pool_noconstraints();
> +}
> +
>  /*
>   * Determine i/o configuration for a machine.
>   */
> @@ -123,6 +140,8 @@ cpu_configure(void)
>  		panic("configure: mainbus not configured");
>  
>  	intr_printconfig();
> +
> +	mbuf_dma_64bit_enable();
>  
>  #if NIOAPIC > 0
>  	lapic_set_lvt();
> Index: dev/pci/if_bnxt.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_bnxt.c,v
> diff -u -p -r1.67 if_bnxt.c
> --- dev/pci/if_bnxt.c	19 Feb 2026 10:15:36 -0000	1.67
> +++ dev/pci/if_bnxt.c	4 Mar 2026 14:47:53 -0000
> @@ -654,7 +654,7 @@ bnxt_attach(struct device *parent, struc
>  	strlcpy(ifp->if_xname, DEVNAME(sc), IFNAMSIZ);
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_MULTICAST | IFF_SIMPLEX;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = bnxt_ioctl;
>  	ifp->if_qstart = bnxt_start;
>  	ifp->if_watchdog = bnxt_watchdog;
> Index: dev/pci/if_em.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_em.c,v
> diff -u -p -r1.380 if_em.c
> --- dev/pci/if_em.c	4 Mar 2026 14:15:36 -0000	1.380
> +++ dev/pci/if_em.c	4 Mar 2026 14:47:53 -0000
> @@ -1994,6 +1994,8 @@ em_setup_interface(struct em_softc *sc)
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
>  	ifp->if_xflags = IFXF_MPSAFE;
> +	if (ISSET(sc->sc_dmaflags, BUS_DMA_64BIT))
> +		ifp->if_xflags |= IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = em_ioctl;
>  	ifp->if_qstart = em_start;
>  	ifp->if_watchdog = em_watchdog;
> Index: dev/pci/if_ice.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ice.c,v
> diff -u -p -r1.68 if_ice.c
> --- dev/pci/if_ice.c	24 Feb 2026 20:14:29 -0000	1.68
> +++ dev/pci/if_ice.c	4 Mar 2026 14:47:53 -0000
> @@ -30657,7 +30657,7 @@ ice_attach_hook(struct device *self)
>  	ifp->if_softc = sc;
>  	strlcpy(ifp->if_xname, sc->sc_dev.dv_xname, IFNAMSIZ);
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = ice_ioctl;
>  	ifp->if_qstart = ice_start;
>  	ifp->if_watchdog = ice_watchdog;
> Index: dev/pci/if_igc.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_igc.c,v
> diff -u -p -r1.31 if_igc.c
> --- dev/pci/if_igc.c	24 Feb 2026 23:01:10 -0000	1.31
> +++ dev/pci/if_igc.c	4 Mar 2026 14:47:53 -0000
> @@ -796,7 +796,7 @@ igc_setup_interface(struct igc_softc *sc
>  	ifp->if_softc = sc;
>  	strlcpy(ifp->if_xname, DEVNAME(sc), IFNAMSIZ);
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = igc_ioctl;
>  	ifp->if_qstart = igc_start;
>  	ifp->if_watchdog = igc_watchdog;
> Index: dev/pci/if_ix.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ix.c,v
> diff -u -p -r1.223 if_ix.c
> --- dev/pci/if_ix.c	26 Feb 2026 23:38:10 -0000	1.223
> +++ dev/pci/if_ix.c	4 Mar 2026 14:47:53 -0000
> @@ -1929,7 +1929,7 @@ ixgbe_setup_interface(struct ix_softc *s
>  	strlcpy(ifp->if_xname, sc->dev.dv_xname, IFNAMSIZ);
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = ixgbe_ioctl;
>  	ifp->if_qstart = ixgbe_start;
>  	ifp->if_timer = 0;
> Index: dev/pci/if_ixl.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ixl.c,v
> diff -u -p -r1.116 if_ixl.c
> --- dev/pci/if_ixl.c	25 Feb 2026 23:40:49 -0000	1.116
> +++ dev/pci/if_ixl.c	4 Mar 2026 14:47:53 -0000
> @@ -1914,7 +1914,7 @@ ixl_attach(struct device *parent, struct
>  
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = ixl_ioctl;
>  	ifp->if_qstart = ixl_start;
>  	ifp->if_watchdog = ixl_watchdog;
> Index: dev/pci/if_ixv.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_ixv.c,v
> diff -u -p -r1.2 if_ixv.c
> --- dev/pci/if_ixv.c	14 Jul 2025 23:49:08 -0000	1.2
> +++ dev/pci/if_ixv.c	4 Mar 2026 14:47:53 -0000
> @@ -768,7 +768,7 @@ ixv_setup_interface(struct device *dev, 
>  	strlcpy(ifp->if_xname, sc->dev.dv_xname, IFNAMSIZ);
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = ixv_ioctl;
>  	ifp->if_qstart = ixgbe_start;
>  	ifp->if_timer = 0;
> Index: dev/pci/if_mcx.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pci/if_mcx.c,v
> diff -u -p -r1.121 if_mcx.c
> --- dev/pci/if_mcx.c	19 Nov 2025 04:58:04 -0000	1.121
> +++ dev/pci/if_mcx.c	4 Mar 2026 14:47:53 -0000
> @@ -2953,7 +2953,7 @@ mcx_attach(struct device *parent, struct
>  	strlcpy(ifp->if_xname, DEVNAME(sc), IFNAMSIZ);
>  	ifp->if_softc = sc;
>  	ifp->if_flags = IFF_BROADCAST | IFF_MULTICAST | IFF_SIMPLEX;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  	ifp->if_ioctl = mcx_ioctl;
>  	ifp->if_qstart = mcx_start;
>  	ifp->if_watchdog = mcx_watchdog;
> Index: dev/pv/if_vio.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/dev/pv/if_vio.c,v
> diff -u -p -r1.78 if_vio.c
> --- dev/pv/if_vio.c	15 Jan 2026 09:06:19 -0000	1.78
> +++ dev/pv/if_vio.c	4 Mar 2026 14:47:53 -0000
> @@ -750,7 +750,7 @@ negotiate:
>  
>  	ifp->if_capabilities = 0;
>  	ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
> -	ifp->if_xflags = IFXF_MPSAFE;
> +	ifp->if_xflags = IFXF_MPSAFE | IFXF_MBUF_64BIT;
>  #if NVLAN > 0
>  	ifp->if_capabilities |= IFCAP_VLAN_MTU;
>  	ifp->if_capabilities |= IFCAP_VLAN_HWOFFLOAD;
> Index: kern/uipc_mbuf.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/kern/uipc_mbuf.c,v
> diff -u -p -r1.304 uipc_mbuf.c
> --- kern/uipc_mbuf.c	5 Feb 2026 03:26:00 -0000	1.304
> +++ kern/uipc_mbuf.c	4 Mar 2026 14:50:42 -0000
> @@ -1480,6 +1480,17 @@ m_pool_init(struct pool *pp, u_int size,
>  	pool_set_constraints(pp, &kp_dma_contig);
>  }
>  
> +void
> +m_pool_noconstraints(void)
> +{
> +	int i;
> +
> +	pool_set_constraints(&mbpool, &kp_mbuf_contig);
> +
> +	for (i = 0; i < nitems(mclsizes); i++)
> +		pool_set_constraints(&mclpools[i], &kp_mbuf_contig);
> +}
> +
>  u_int
>  m_pool_used(void)
>  {
> Index: net/if.h
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/net/if.h,v
> diff -u -p -r1.223 if.h
> --- net/if.h	9 Dec 2025 03:33:06 -0000	1.223
> +++ net/if.h	4 Mar 2026 14:47:53 -0000
> @@ -232,6 +232,7 @@ struct if_status_description {
>  #define	IFXF_AUTOCONF4		0x80	/* [N] v4 autoconf (aka dhcp) enabled */
>  #define	IFXF_MONITOR		0x100	/* [N] only used for bpf */
>  #define	IFXF_LRO		0x200	/* [N] TCP large recv offload */
> +#define	IFXF_MBUF_64BIT		0x400	/* [I] mbuf with 64 bit DMA supported */
>  
>  #define	IFXF_CANTCHANGE \
>  	(IFXF_MPSAFE|IFXF_CLONED)
> Index: sys/mbuf.h
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/sys/mbuf.h,v
> diff -u -p -r1.269 mbuf.h
> --- sys/mbuf.h	5 Feb 2026 03:26:00 -0000	1.269
> +++ sys/mbuf.h	4 Mar 2026 14:50:42 -0000
> @@ -441,6 +441,7 @@ void	m_align(struct mbuf *, int);
>  struct mbuf *m_clget(struct mbuf *, int, u_int);
>  void	m_extref(struct mbuf *, struct mbuf *);
>  void	m_pool_init(struct pool *, u_int, u_int, const char *);
> +void	m_pool_noconstraints(void);
>  u_int	m_pool_used(void);
>  void	m_extfree_pool(caddr_t, u_int, void *);
>  void	m_adj(struct mbuf *, int);
> Index: uvm/uvm_extern.h
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/uvm/uvm_extern.h,v
> diff -u -p -r1.187 uvm_extern.h
> --- uvm/uvm_extern.h	13 Nov 2025 10:55:51 -0000	1.187
> +++ uvm/uvm_extern.h	4 Mar 2026 14:50:42 -0000
> @@ -358,6 +358,7 @@ extern const struct kmem_pa_mode kp_zero
>  extern const struct kmem_pa_mode kp_dma;
>  extern const struct kmem_pa_mode kp_dma_contig;
>  extern const struct kmem_pa_mode kp_dma_zero;
> +extern const struct kmem_pa_mode kp_mbuf_contig;
>  extern const struct kmem_pa_mode kp_pageable;
>  extern const struct kmem_pa_mode kp_none;
>  
> Index: uvm/uvm_km.c
> ===================================================================
> RCS file: /data/mirror/openbsd/cvs/src/sys/uvm/uvm_km.c,v
> diff -u -p -r1.159 uvm_km.c
> --- uvm/uvm_km.c	13 Nov 2025 10:55:51 -0000	1.159
> +++ uvm/uvm_km.c	4 Mar 2026 14:50:42 -0000
> @@ -745,6 +745,11 @@ const struct kmem_pa_mode kp_dma_zero = 
>  	.kp_zero = 1
>  };
>  
> +const struct kmem_pa_mode kp_mbuf_contig = {
> +	.kp_constraint = &no_constraint,
> +	.kp_maxseg = 1
> +};
> +
>  const struct kmem_pa_mode kp_zero = {
>  	.kp_constraint = &no_constraint,
>  	.kp_zero = 1
> 
>