From: Patrick Wildt <patrick@blueri.se>
Subject: Re: smmu(4) on QC Laptops
To: Mark Kettenis <mark.kettenis@xs4all.nl>
Cc: tech@openbsd.org, tobhe@openbsd.org, mlarkin@openbsd.org
Date: Sat, 24 May 2025 07:21:52 +0200

On Fri, May 23, 2025 at 07:58:29PM +0200, Mark Kettenis wrote:
> > Date: Fri, 23 May 2025 08:46:16 +0200
> > From: Patrick Wildt <patrick@blueri.se>
> > Cc: kettenis@openbsd.org, tobhe@openbsd.org, mlarkin@openbsd.org
> > Content-Type: text/plain; charset=us-ascii
> > Content-Disposition: inline
> > 
> > Hi,
> > 
> > Nearly three years ago we disabled smmu(4) for QC laptops because they
> > just rebooted instantly.  I've finally gone to the bottom of why this
> > happens.
> > 
> > On servers we usually boot up with most peripherals not doing any kind
> > of DMA.  A serial console is kind of the only thing we depend on, which
> > usually doesn't use DMA for simplicity.  Everything else, like NVMe or
> > USB will get turned on later.  For those use cases it's been perfectly
> > fine to just turn the SMMU on with strict filtering, because only once
> > the driver configures puts in a job DMA transactions will occur.
> > 
> > On the QC laptops the framebuffer goes through an SMMU.  That's one of
> > the reasons we need to keep the streams alive and actively put them into
> > a bypass context, which we do!  As soon as a driver attaches, we move
> > the stream over into a strict translation context.
> > 
> > Unfortunately the changes I did were not completely sufficient.
> > 
> > (1) The IOMMU enforcement happens before the driver gets a chance to
> >     stop current work.
> > 
> > simplebus(4) creates an IOMMU-bound DMA tag before it attempts to attach
> > the children.  This leads do two interesting things:
> > 
> > (a) If a simplebus(4) has an IOMMU assigned, it is passed an IOMMU-bound
> >     DMA tag.  All children will get subjected to it.
> > 
> > smmu1: establishing sid 0x423
> > simplebus1 at simplebus0: "geniqup"
> > "serial" at simplebus1 not configured
> > smmu1: establishing sid 0x123
> > simplebus2 at simplebus0: "geniqup"
> > 
> > (b) If there's no driver to be found, the IOMMU-bound DMA tag is created
> >     anyway.
> > 
> > smmu1: establishing sid 0x1c00
> > smmu1: took over 8/1c00/2 for sid 1c00
> > "display-subsystem" at simplebus0 not configured
> > 
> > The good thing is that (a) is not a problem in this case as we do not
> > have any active transactions such mappings, as far as I can see on the
> > X1E.  But the framebuffer is behind the display-subsystem which is now
> > subjected to strict IOMMU configuration, making us need to fix (b).
> > 
> > One idea I have is that we keep all the setup code where it is, at
> > IOMMU-bound DMA tag creation time, but only move the stream to the
> > strict context block once someone(TM) creates a DMA map using this
> > tag.
> > 
> > Is that a reasonable thing to do?  I don't think anyone would be doing
> > DMA without creating a DMA map first, right?
> 
> Correct.
> 
> I'm not sure how this is going to work if we want to write a driver
> for the display controller.  Presumably we'd create IOMMU mappings for
> the initial framebuffer.  But to do so, we'd need to create a DMA map
> first, and when we do that...  game over?
> 
> So maybe we need to do the takeover when we do the first DMA map load,
> after we've entered the mappings into the page table?  But that only
> works if we have a single mapping for the framebuffer.

I was hoping that we'd have code that first turns off the transactions,
making us "go blind" until we re-enable it with the new address.  But
yes, that would require the disable sequence to happen before the map
is created.  We could do it on load, which is kind of a hot-path.  I
was hoping for now we put it into map create and tackle this once or
if we get around to writing this driver.

> On the Apple systems, the framebuffer is also behind an IOMMU.  There
> we have some code that creates some initial mappings based on
> information from the device tree.
> 
> > (2) Re-use of SMRs doesn't take mask in account.
> > 
> > In (1b) you can see that the display subsystem is using sid 0x1c00 with
> > mask 0x2, matching to the following SMR:
> > 
> > smmu1: SMR[8] = 0x1c00/0x2
> > 
> > We'd like to re-use that if possible, but because of the strict mask
> > checking that wasn't possible.  So I'd like to take the mask into
> > consideration when "taking over" streams.  This is only done on QC!
> 
> Sorry, you'll have to explain to me what what the SMRs do...

Think work slightly different depending on SMMU version/feature flags,
but for this machine it works an IOMMU setup like this:

You configure a context block.  The context block is basically the IOMMU
group that owns the pagetable.  To assign devices to a context block,
you setup a "Stream Mapping Register", e.g.:

 * Allocate + setup context block 3 (e.g. IOMMU group 3)
   * This is the actual pagetable and translation setup
 * Allocate a stream map entry (can be any, e.g. 5)
   * The array has a fixed length, we need to use one slot to
     point device to context block.
 * Write(SMR(5), DeviceId + Mask + Valid)
   * For each DMA transaction, the SMMUs stream mapping function will
     iterate the stream map registers to find an enty where:
     StreamId & Mask == DeviceId & Mask
   * In this example, after writing SMR(5), slot 5 should match
 * Write(S2CR(5) = 3)
   * This points stream map 5 to context block 3
   * In this example, after the HW's lookup function returns 5, it will
     look into S2CR(5) to find which context block it belongs to

On this laptop, the SMRs basically tell you what devices are currently
enabled to pass through the IOMMU.  We update the SMRs to point all
currenetly allowed devices to a "bypass domain", so that existing
traffic goes into a 1:1 translation.

One we want attach a driver and want to make a device use stricter IOMMU
mappings, we setup a context block and then switch the device from the
bypass IOMMU group/context block to the strict IOMMU group/context
block.

In reality we're not actually changing the SMR (that's just the filter
rule), we're updating the S2CR to point the stream to a different group.

But if we don't "take over" the existing SMR mapping, the code will
allocate a new SMR register, and then the IOMMU will complain about a
match conflict because an incoming transaction matches two SMR entries.

Let me know if I worded this too confusingly!

> > Please give this a go on machines where `dmesg | grep ^smmu' gives
> > output, and especially on QC machines with X1E, or SC8280XP/ThinkPad
> > x13s.  Please send me some dmesgs, as I have added some debug code
> > that I'd like to see run on other machines. :)
> 
> Here's the Ampere eMAG machine.  ACPI so some of your changes don't
> matter, but it still works with your diff.
> 

Thanks! :)

Cheers,
Patrick

> 
> OpenBSD 7.7-current (GENERIC.MP) #1: Fri May 23 19:28:38 CEST 2025
>     kettenis@liszt.sibelius.xs4all.nl:/home/kettenis/src/smmu/sys/arch/arm64/compile/GENERIC.MP
> real mem  = 137121468416 (130769MB)
> avail mem = 132743782400 (126594MB)
> random: good seed from bootblocks
> mainbus0 at root: ACPI
> psci0 at mainbus0: PSCI 1.1, SMCCC 1.0
> efi0 at mainbus0: UEFI 2.7
> efi0: American Megatrends rev 0x5000d
> smbios0 at efi0: SMBIOS 3.2.0
> smbios0: vendor LENOVO version "hve104r-1.15" date 02/26/2021
> smbios0: Lenovo HR330A            7X33CTO1WW
> cpu0 at mainbus0 mpidr 0: Applied Micro X-Gene r3p2
> cpu0: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu0: 256KB 64b/line 32-way L2 cache
> cpu0: CRC32,SHA2,SHA1,AES+PMULL,ASID16
> cpu1 at mainbus0 mpidr 1: Applied Micro X-Gene r3p2
> cpu1: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu1: 256KB 64b/line 32-way L2 cache
> cpu2 at mainbus0 mpidr 100: Applied Micro X-Gene r3p2
> cpu2: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu2: 256KB 64b/line 32-way L2 cache
> cpu3 at mainbus0 mpidr 101: Applied Micro X-Gene r3p2
> cpu3: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu3: 256KB 64b/line 32-way L2 cache
> cpu4 at mainbus0 mpidr 200: Applied Micro X-Gene r3p2
> cpu4: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu4: 256KB 64b/line 32-way L2 cache
> cpu5 at mainbus0 mpidr 201: Applied Micro X-Gene r3p2
> cpu5: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu5: 256KB 64b/line 32-way L2 cache
> cpu6 at mainbus0 mpidr 300: Applied Micro X-Gene r3p2
> cpu6: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu6: 256KB 64b/line 32-way L2 cache
> cpu7 at mainbus0 mpidr 301: Applied Micro X-Gene r3p2
> cpu7: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu7: 256KB 64b/line 32-way L2 cache
> cpu8 at mainbus0 mpidr 400: Applied Micro X-Gene r3p2
> cpu8: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu8: 256KB 64b/line 32-way L2 cache
> cpu9 at mainbus0 mpidr 401: Applied Micro X-Gene r3p2
> cpu9: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu9: 256KB 64b/line 32-way L2 cache
> cpu10 at mainbus0 mpidr 500: Applied Micro X-Gene r3p2
> cpu10: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu10: 256KB 64b/line 32-way L2 cache
> cpu11 at mainbus0 mpidr 501: Applied Micro X-Gene r3p2
> cpu11: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu11: 256KB 64b/line 32-way L2 cache
> cpu12 at mainbus0 mpidr 600: Applied Micro X-Gene r3p2
> cpu12: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu12: 256KB 64b/line 32-way L2 cache
> cpu13 at mainbus0 mpidr 601: Applied Micro X-Gene r3p2
> cpu13: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu13: 256KB 64b/line 32-way L2 cache
> cpu14 at mainbus0 mpidr 700: Applied Micro X-Gene r3p2
> cpu14: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu14: 256KB 64b/line 32-way L2 cache
> cpu15 at mainbus0 mpidr 701: Applied Micro X-Gene r3p2
> cpu15: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu15: 256KB 64b/line 32-way L2 cache
> cpu16 at mainbus0 mpidr 800: Applied Micro X-Gene r3p2
> cpu16: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu16: 256KB 64b/line 32-way L2 cache
> cpu17 at mainbus0 mpidr 801: Applied Micro X-Gene r3p2
> cpu17: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu17: 256KB 64b/line 32-way L2 cache
> cpu18 at mainbus0 mpidr 900: Applied Micro X-Gene r3p2
> cpu18: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu18: 256KB 64b/line 32-way L2 cache
> cpu19 at mainbus0 mpidr 901: Applied Micro X-Gene r3p2
> cpu19: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu19: 256KB 64b/line 32-way L2 cache
> cpu20 at mainbus0 mpidr a00: Applied Micro X-Gene r3p2
> cpu20: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu20: 256KB 64b/line 32-way L2 cache
> cpu21 at mainbus0 mpidr a01: Applied Micro X-Gene r3p2
> cpu21: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu21: 256KB 64b/line 32-way L2 cache
> cpu22 at mainbus0 mpidr b00: Applied Micro X-Gene r3p2
> cpu22: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu22: 256KB 64b/line 32-way L2 cache
> cpu23 at mainbus0 mpidr b01: Applied Micro X-Gene r3p2
> cpu23: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu23: 256KB 64b/line 32-way L2 cache
> cpu24 at mainbus0 mpidr c00: Applied Micro X-Gene r3p2
> cpu24: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu24: 256KB 64b/line 32-way L2 cache
> cpu25 at mainbus0 mpidr c01: Applied Micro X-Gene r3p2
> cpu25: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu25: 256KB 64b/line 32-way L2 cache
> cpu26 at mainbus0 mpidr d00: Applied Micro X-Gene r3p2
> cpu26: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu26: 256KB 64b/line 32-way L2 cache
> cpu27 at mainbus0 mpidr d01: Applied Micro X-Gene r3p2
> cpu27: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu27: 256KB 64b/line 32-way L2 cache
> cpu28 at mainbus0 mpidr e00: Applied Micro X-Gene r3p2
> cpu28: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu28: 256KB 64b/line 32-way L2 cache
> cpu29 at mainbus0 mpidr e01: Applied Micro X-Gene r3p2
> cpu29: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu29: 256KB 64b/line 32-way L2 cache
> cpu30 at mainbus0 mpidr f00: Applied Micro X-Gene r3p2
> cpu30: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu30: 256KB 64b/line 32-way L2 cache
> cpu31 at mainbus0 mpidr f01: Applied Micro X-Gene r3p2
> cpu31: 32KB 64b/line 8-way L1 PIPT I-cache, 32KB 64b/line 8-way L1 D-cache
> cpu31: 256KB 64b/line 32-way L2 cache
> apm0 at mainbus0
> agintc0 at mainbus0 shift 4:4 nirq 544 nredist 32 ipi: 0, 1, 2: "interrupt-controller"
> agintcmsi0 at agintc0
> agtimer0 at mainbus0: 40000 kHz
> acpi0 at mainbus0: ACPI 6.1
> acpi0: sleep states
> acpi0: tables DSDT FACP FIDT DBG2 GTDT IORT MCFG SSDT SPMI APIC PCCT BERT HEST VFCT SPCR PPTT
> acpi0: wakeup devices
> acpiiort0 at acpi0
> smmu0 at acpiiort0 addr 0x14000000/0x100000: 128 CBs (128 S2-only)
> smmu1 at acpiiort0 addr 0x15000000/0x100000: 128 CBs (128 S2-only)
> acpimcfg0 at acpi0
> acpimcfg0: addr 0x10000000000, bus 0-31
> acpimcfg0: addr 0x7800000000, bus 0-31
> acpimcfg0: addr 0x1000000000, bus 0-31
> acpimcfg0: addr 0x5800000000, bus 0-31
> acpimcfg0: addr 0x6000000000, bus 0-31
> acpimcfg0: addr 0x7000000000, bus 0-7
> acpimcfg0: addr 0x600000000, bus 0-7
> acpimcfg0: addr 0x400000000, bus 0-7
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> "APMC0D40" at acpi0 not configured
> "APMC0D40" at acpi0 not configured
> dwiic0 at acpi0 I2C4 addr 0x126b0000/0x1000 irq 105
> iic0 at dwiic0
> ipmi0 at iic0 addr 0x10: version 2.0 interface SSIF
> pluart0 at acpi0 URT0 addr 0x12600000/0x1000 irq 98
> pluart0: console
> pluart1 at acpi0 URT1 addr 0x12610000/0x1000 irq 99
> smmu1: establishing sid 0x0
> ahci0 at acpi0 SAT0 addr 0x1c000000/0x1000 irq 111: AHCI 1.3.1
> ahci0: port 0: 6.0Gb/s
> scsibus0 at ahci0: 32 targets
> sd0 at scsibus0 targ 0 lun 0: <ATA, Samsung SSD 860, RVT0> naa.5002538e404a2eee
> sd0: 238475MB, 512 bytes/sector, 488397168 sectors, thin
> smmu1: establishing sid 0x800
> ahci1 at acpi0 SAT1 addr 0x1c100000/0x1000 irq 112: AHCI 1.3.1
> scsibus1 at ahci1: 32 targets
> smmu1: establishing sid 0x1000
> xhci0 at acpi0 USB0 addr 0x13800000/0x100000 irq 115, xHCI 1.10
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1
> smmu1: establishing sid 0x1800
> xhci1 at acpi0 USB1 addr 0x13900000/0x100000 irq 116, xHCI 1.10
> usb1 at xhci1: USB revision 3.0
> uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1
> acpibtn0 at acpi0: PWRB
> acpige0 at acpi0 irq 84
> acpige1 at acpi0 irq 72
> "PNP0C33" at acpi0 not configured
> "LNRO0007" at acpi0 not configured
> "APMC0D83" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D84" at acpi0 not configured
> "APMC0D87" at acpi0 not configured
> "APMC0D87" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D88" at acpi0 not configured
> "APMC0D85" at acpi0 not configured
> "APMC0D86" at acpi0 not configured
> acpipci0 at acpi0 PCI0
> pci0 at acpipci0
> smmu0: establishing sid 0x0
> ppb0 at pci0 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 131
> pci1 at ppb0 bus 1
> smmu0: establishing sid 0x100
> em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi, address 00:1b:21:e0:6f:7d
> acpipci1 at acpi0 PCI2
> pci2 at acpipci1
> smmu1: establishing sid 0x4000
> 0:0:0: bridge io address conflict 0x10000000/0x1000
> ppb1 at pci2 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 143
> pci3 at ppb1 bus 1
> smmu1: establishing sid 0x4100
> smmu1: establishing sid 0x4101
> amdgpu0 at pci3 dev 0 function 0 "ATI Polaris 12" rev 0x00
> drm0 at amdgpu0
> amdgpu0: msi
> azalia0 at pci3 dev 0 function 1 "ATI Radeon Pro Audio" rev 0x00: msi
> azalia0: no supported codecs
> acpipci2 at acpi0 PCI3
> pci4 at acpipci2
> smmu1: establishing sid 0x6000
> ppb2 at pci4 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 149
> pci5 at ppb2 bus 1
> acpipci3 at acpi0 PCI4
> pci6 at acpipci3
> smmu0: establishing sid 0x4000
> ppb3 at pci6 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 155
> pci7 at ppb3 bus 1
> acpipci4 at acpi0 PCI5
> pci8 at acpipci4
> smmu0: establishing sid 0x6000
> ppb4 at pci8 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 161
> pci9 at ppb4 bus 1
> acpipci5 at acpi0 PCI6
> pci10 at acpipci5
> smmu1: establishing sid 0x2000
> ppb5 at pci10 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 167
> pci11 at ppb5 bus 1
> acpipci6 at acpi0 PCI7
> pci12 at acpipci6
> smmu1: establishing sid 0x2800
> 0:0:0: bridge io address conflict 0x10000000/0x1000
> ppb6 at pci12 dev 0 function 0 "Ampere eMAG PCIe" rev 0x04: irq 173
> pci13 at ppb6 bus 1
> smmu1: establishing sid 0x2900
> 1:0:0: bridge io address conflict 0x10000000/0x1000
> ppb7 at pci13 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
> pci14 at ppb7 bus 2
> smmu1: establishing sid 0x2a00
> "ASPEED Technology AST2000" rev 0x41 at pci14 dev 0 function 0 not configured
> uhub2 at uhub0 port 1 configuration 1 interface 0 "American Megatrends Inc. Virtual Hub" rev 2.00/1.00 addr 2
> umass0 at uhub2 port 1 configuration 1 interface 0 "American Megatrends Inc. Virtual Cdrom Device" rev 2.00/1.00 addr 3
> umass0: using SCSI over Bulk-Only
> scsibus2 at umass0: 2 targets, initiator 0
> cd0 at scsibus2 targ 1 lun 0: <AMI, Virtual CDROM0, 1.00> removable serial.046bff20AAABBBBCCCC1
> umass1 at uhub2 port 2 configuration 1 interface 0 "American Megatrends Inc. Virtual HardDisk Device" rev 2.00/1.00 addr 4
> umass1: using SCSI over Bulk-Only
> scsibus3 at umass1: 2 targets, initiator 0
> sd1 at scsibus3 targ 1 lun 0: <AMI, Virtual HDisk0, 1.00> removable serial.046bff31AAABBBBCCCC3
> uhidev0 at uhub2 port 3 configuration 1 interface 0 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 5
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0 mux 1
> uhidev1 at uhub2 port 3 configuration 1 interface 1 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 5
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhub3 at uhub1 port 1 configuration 1 interface 0 "Cypress Semiconductor USB2 Hub" rev 2.00/90.15 addr 2
> vscsi0 at root
> scsibus4 at vscsi0: 256 targets
> softraid0 at root
> scsibus5 at softraid0: 256 targets
> root on sd0a (8662c5862198d925.a) swap on sd0b dump on sd0b
> amdgpu0: POLARIS12 8 CU rev 0x00
> amdgpu0: 1024x768, 32bpp
> wsdisplay0 at amdgpu0 mux 1
> wskbd0: connecting to wsdisplay0
> wsdisplay0: screen 0-5 added (std, vt100 emulation)
> simplefb0 at mainbus0: 1024x768, 32bpp
> wsdisplay1 at simplefb0 mux 1
> wsdisplay1: screen 0-5 added (std, vt100 emulation)