From: Alexander Bluhm <bluhm@openbsd.org>
Subject: Re: Reworking VMM's nested paging & guest memory (de-vmspace-ification)
To: Dave Voutila <dv@sisu.io>
Cc: tech@openbsd.org, mlarkin@openbsd.org, Hans-Joerg_Hoexer@genua.de
Date: Wed, 16 Apr 2025 19:41:11 +0200

On Tue, Apr 15, 2025 at 03:56:20PM -0400, Dave Voutila wrote:
> Alexander Bluhm <alexander.bluhm@gmx.net> writes:
> 
> > On Mon, Apr 14, 2025 at 08:18:46PM -0400, Dave Voutila wrote:
> >> OKs welcome, but I'd prefer some test reports before looking to commit,
> >> but since we're recently unlocked from release prep, I'd prefer sooner
> >> rather than later.
> >
> > It seems to work on my AMD CPU.  But with AMD SEV enabled, guest
> > does not start.
> >
> 
> Hmm. My AMD machine is too old and doesn't have a psp(4). :(
> 
> > Apr 15 20:14:37 ot52 vmd[99497]: psp_activate: ioctl: Input/output error
> > Apr 15 20:14:37 ot52 vmd[99497]: sev_activate: failed to activate guest: 0x1:0x3
> 
> This is a pretty early failure. From my reading of sev.c it doesn't seem
> to even have anything to do with the guest memory unless I'm missing
> something.

Seems to be false alarm.  My psp(4) is actually failing, and it
started doing that at Apr 15.  But now without your diff, I get the
same error message.

I will try to find out what is wrong with my setup.  This should
not be a show stopper for your diff.

> This seems odd. Hans-Joerg, any idea? I'm pretty sure I didn't touch
> anything to do with asids or SEV. The guest memory is mapped into the
> process differently, but at this point it should be mapped and
> available to vmd...otherwise other things would fail.
> 
> > Apr 15 20:14:37 ot52 vmd[99497]: vmd: SEV activatation failed for VCPU 0 failed - exiting.
> > Apr 15 20:14:37 ot52 vmd[99497]: psp_guest_shutdown: ioctl: Input/output error
> > Apr 15 20:14:37 ot52 vmd[99497]: failed to deactivate guest
> > Apr 15 20:14:37 ot52 vmd[99497]: start_vm: could not shutdown SEV
> > Apr 15 20:14:37 ot52 vmd[57809]: SIOCSIFDESCR: Device not configured
> > Apr 15 20:14:37 ot52 vmd[57809]: SIOCBRDGADD: Invalid argument
> > Apr 15 20:14:37 ot52 vmd[57809]: SIOCSIFRDOMAIN: Device not configured
> > Apr 15 20:14:37 ot52 vmd[57809]: SIOCGIFFLAGS: Device not configured
> >
> > OpenBSD 7.7-current (GENERIC.MP) #0: Tue Apr 15 16:36:45 CEST 2025
> >     root@ot52.obsd-lab.genua.de:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 274654408704 (261930MB)
> > avail mem = 266303348736 (253966MB)
> > random: good seed from bootblocks
> > mpath0 at root
> > scsibus0 at mpath0: 256 targets
> > mainbus0 at root
> > bios0 at mainbus0: SMBIOS rev. 3.3 @ 0xa9d1c000 (75 entries)
> > bios0: vendor American Megatrends Inc. version "2.5" date 09/26/2022
> > bios0: genua gsXXL
> > efi0 at bios0: UEFI 2.8
> > efi0: American Megatrends rev 0x50016
> > acpi0 at bios0: ACPI 6.0
> > acpi0: sleep states S0 S5
> > acpi0: tables DSDT FACP SSDT SPMI SSDT FIDT MCFG SSDT SSDT BERT HPET IVRS PCCT SSDT CRAT CDIT SSDT WSMT APIC ERST HEST
> > acpi0: wakeup devices B000(S3) C000(S3) B010(S3) C010(S3) B030(S3) C030(S3) B020(S3) C020(S3) B100(S3) C100(S3) B110(S3) C110(S3) B130(S3) C130(S3) B120(S3) C120(S3)
> > acpitimer0 at acpi0: 3579545 Hz, 32 bits
> > acpimcfg0 at acpi0
> > acpimcfg0: addr 0xe0000000, bus 0-255
> > acpihpet0 at acpi0: 14318180 Hz
> > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu0: cpuid 1
> > edx=178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
> > ecx=76da320b<SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND>
> > cpu0: cpuid 6 eax=4<ARAT> ecx=1<EFFFREQ>
> > cpu0: cpuid 7.0 ebx=219c97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> ecx=40068c<UMIP,PKU> edx=10
> > cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
> > cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG>
> > ecx=75c237ff<LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX>
> > cpu0: cpuid 80000007 edx=6799<HWPSTATE,ITSC>
> > cpu0: cpuid 80000008 ebx=91bef75f<INVLPGB,IBPB,IBRS,STIBP,STIBP_ALL,IBRS_PREF,IBRS_SM,SSBD>
> > cpu0: cpuid 8000001F eax=101fd3f<SME,SEV,PFLUSH_MSR,SEVES,SEVSNP,VMPL,SECTSC,HWECACHECOH,REQ64BITHOST,RESTINJ,ALTINJ,DBGSTSW,IBSDISALLOW,VTE,VMSARPROT> ecx=1fd edx=5
> > cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 32MB 64b/line 16-way L3 cache
> > cpu0: smt 0, core 0, package 0
> > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> > cpu0: apic clock running at 100MHz
> > cpu0: mwait min=64, max=64, C-substates=1.1, IBE
> > cpu1 at mainbus0: apid 2 (application processor)
> > cpu1: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu1: smt 0, core 1, package 0
> > cpu2 at mainbus0: apid 4 (application processor)
> > cpu2: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu2: smt 0, core 2, package 0
> > cpu3 at mainbus0: apid 6 (application processor)
> > cpu3: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu3: smt 0, core 3, package 0
> > cpu4 at mainbus0: apid 8 (application processor)
> > cpu4: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu4: smt 0, core 4, package 0
> > cpu5 at mainbus0: apid 10 (application processor)
> > cpu5: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu5: smt 0, core 5, package 0
> > cpu6 at mainbus0: apid 12 (application processor)
> > cpu6: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu6: smt 0, core 6, package 0
> > cpu7 at mainbus0: apid 14 (application processor)
> > cpu7: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu7: smt 0, core 7, package 0
> > cpu8 at mainbus0: apid 16 (application processor)
> > cpu8: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu8: smt 0, core 8, package 0
> > cpu9 at mainbus0: apid 18 (application processor)
> > cpu9: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu9: smt 0, core 9, package 0
> > cpu10 at mainbus0: apid 20 (application processor)
> > cpu10: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu10: smt 0, core 10, package 0
> > cpu11 at mainbus0: apid 22 (application processor)
> > cpu11: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu11: smt 0, core 11, package 0
> > cpu12 at mainbus0: apid 24 (application processor)
> > cpu12: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu12: smt 0, core 12, package 0
> > cpu13 at mainbus0: apid 26 (application processor)
> > cpu13: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu13: smt 0, core 13, package 0
> > cpu14 at mainbus0: apid 28 (application processor)
> > cpu14: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu14: smt 0, core 14, package 0
> > cpu15 at mainbus0: apid 30 (application processor)
> > cpu15: AMD EPYC 73F3 16-Core Processor, 3500.00 MHz, 19-01-01, patch 0a0011d5
> > cpu15: smt 0, core 15, package 0
> > cpu16 at mainbus0: apid 1 (application processor)
> > cpu16: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu16: smt 1, core 0, package 0
> > cpu17 at mainbus0: apid 3 (application processor)
> > cpu17: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu17: smt 1, core 1, package 0
> > cpu18 at mainbus0: apid 5 (application processor)
> > cpu18: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu18: smt 1, core 2, package 0
> > cpu19 at mainbus0: apid 7 (application processor)
> > cpu19: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu19: smt 1, core 3, package 0
> > cpu20 at mainbus0: apid 9 (application processor)
> > cpu20: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu20: smt 1, core 4, package 0
> > cpu21 at mainbus0: apid 11 (application processor)
> > cpu21: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu21: smt 1, core 5, package 0
> > cpu22 at mainbus0: apid 13 (application processor)
> > cpu22: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu22: smt 1, core 6, package 0
> > cpu23 at mainbus0: apid 15 (application processor)
> > cpu23: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu23: smt 1, core 7, package 0
> > cpu24 at mainbus0: apid 17 (application processor)
> > cpu24: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu24: smt 1, core 8, package 0
> > cpu25 at mainbus0: apid 19 (application processor)
> > cpu25: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu25: smt 1, core 9, package 0
> > cpu26 at mainbus0: apid 21 (application processor)
> > cpu26: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu26: smt 1, core 10, package 0
> > cpu27 at mainbus0: apid 23 (application processor)
> > cpu27: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu27: smt 1, core 11, package 0
> > cpu28 at mainbus0: apid 25 (application processor)
> > cpu28: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu28: smt 1, core 12, package 0
> > cpu29 at mainbus0: apid 27 (application processor)
> > cpu29: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu29: smt 1, core 13, package 0
> > cpu30 at mainbus0: apid 29 (application processor)
> > cpu30: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu30: smt 1, core 14, package 0
> > cpu31 at mainbus0: apid 31 (application processor)
> > cpu31: AMD EPYC 73F3 16-Core Processor, 3500.01 MHz, 19-01-01, patch 0a0011d5
> > cpu31: smt 1, core 15, package 0
> > ioapic0 at mainbus0: apid 240 pa 0xfec00000, version 21, 24 pins, can't remap
> > ioapic1 at mainbus0: apid 241 pa 0xb5400000, version 21, 32 pins, can't remap
> > ioapic2 at mainbus0: apid 242 pa 0xf4180000, version 21, 32 pins, can't remap
> > ioapic3 at mainbus0: apid 243 pa 0xb4180000, version 21, 32 pins, can't remap
> > ioapic4 at mainbus0: apid 244 pa 0xf5180000, version 21, 32 pins, can't remap
> > acpiprt0 at acpi0: bus 192 (S0D0)
> > acpiprt1 at acpi0: bus -1 (D0A0)
> > acpiprt2 at acpi0: bus -1 (D0A1)
> > acpiprt3 at acpi0: bus 193 (D0A2)
> > acpiprt4 at acpi0: bus 194 (D0A3)
> > acpiprt5 at acpi0: bus 195 (D0A4)
> > acpiprt6 at acpi0: bus -1 (BR14)
> > acpiprt7 at acpi0: bus 197 (D0A5)
> > acpiprt8 at acpi0: bus 198 (D0A6)
> > acpiprt9 at acpi0: bus 199 (B000)
> > acpiprt10 at acpi0: bus 200 (C000)
> > acpiprt11 at acpi0: bus -1 (C001)
> > acpiprt12 at acpi0: bus -1 (C002)
> > acpiprt13 at acpi0: bus 128 (S0D1)
> > acpiprt14 at acpi0: bus -1 (D1A0)
> > acpiprt15 at acpi0: bus -1 (D1A1)
> > acpiprt16 at acpi0: bus -1 (D1A2)
> > acpiprt17 at acpi0: bus -1 (D1A3)
> > acpiprt18 at acpi0: bus 129 (D1B0)
> > acpiprt19 at acpi0: bus 130 (D1B1)
> > acpiprt20 at acpi0: bus 131 (B010)
> > acpiprt21 at acpi0: bus 132 (C010)
> > acpiprt22 at acpi0: bus -1 (C011)
> > acpiprt23 at acpi0: bus -1 (C012)
> > acpiprt24 at acpi0: bus 0 (PCI0)
> > acpiprt25 at acpi0: bus -1 (D3A0)
> > acpiprt26 at acpi0: bus -1 (D3A1)
> > acpiprt27 at acpi0: bus -1 (D3B0)
> > acpiprt28 at acpi0: bus -1 (D3B1)
> > acpiprt29 at acpi0: bus -1 (D3B2)
> > acpiprt30 at acpi0: bus -1 (D3B3)
> > acpiprt31 at acpi0: bus 1 (B030)
> > acpiprt32 at acpi0: bus 2 (C030)
> > acpiprt33 at acpi0: bus -1 (C031)
> > acpiprt34 at acpi0: bus -1 (C032)
> > acpiprt35 at acpi0: bus 64 (S0D2)
> > acpiprt36 at acpi0: bus -1 (D2A0)
> > acpiprt37 at acpi0: bus -1 (D2A1)
> > acpiprt38 at acpi0: bus -1 (D2B0)
> > acpiprt39 at acpi0: bus -1 (D2B1)
> > acpiprt40 at acpi0: bus -1 (D2B2)
> > acpiprt41 at acpi0: bus -1 (D2B3)
> > acpiprt42 at acpi0: bus 65 (B020)
> > acpiprt43 at acpi0: bus 66 (C020)
> > acpiprt44 at acpi0: bus 67 (C021)
> > acpiprt45 at acpi0: bus 68 (C022)
> > acpipci0 at acpi0 S0D0: 0x00000000 0x00000011 0x00000001
> > acpipci1 at acpi0 S0D1: 0x00000000 0x00000011 0x00000001
> > acpipci2 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> > acpicmos0 at acpi0
> > com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> > com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
> > com1: console
> > "IPI0001" at acpi0 not configured
> > amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x400 irq 7, 184 pins
> > acpipci3 at acpi0 S0D2: 0x00000000 0x00000011 0x00000001
> > acpibtn0 at acpi0: PWRB
> > acpicpu0 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu1 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu2 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu3 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu4 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu5 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu6 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu7 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu8 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu9 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu10 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu11 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu12 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu13 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu14 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu15 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu16 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu17 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu18 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu19 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu20 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu21 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu22 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu23 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu24 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu25 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu26 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu27 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu28 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu29 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu30 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > acpicpu31 at acpi0: C2(0@30 io@0x814), C1(0@1 mwait), PSS
> > "PNP0C33" at acpi0 not configured
> > "PNP0C14" at acpi0 not configured
> > acpipwrres0 at acpi0: P0SA
> > acpipwrres1 at acpi0: P3SA
> > acpipwrres2 at acpi0: P0SA
> > acpipwrres3 at acpi0: P3SA
> > acpipwrres4 at acpi0: P0SA
> > acpipwrres5 at acpi0: P3SA
> > acpipwrres6 at acpi0: P0SA
> > acpipwrres7 at acpi0: P3SA
> > acpipwrres8 at acpi0: P0SA
> > acpipwrres9 at acpi0: P3SA
> > acpipwrres10 at acpi0: P0SA
> > acpipwrres11 at acpi0: P3SA
> > acpipwrres12 at acpi0: P0SA
> > acpipwrres13 at acpi0: P3SA
> > acpipwrres14 at acpi0: P0SA
> > acpipwrres15 at acpi0: P3SA
> > acpipwrres16 at acpi0: P0SA
> > acpipwrres17 at acpi0: P3SA
> > acpipwrres18 at acpi0: P0SA
> > acpipwrres19 at acpi0: P3SA
> > acpipwrres20 at acpi0: P0SA
> > acpipwrres21 at acpi0: P3SA
> > acpipwrres22 at acpi0: P0SA
> > acpipwrres23 at acpi0: P3SA
> > acpipwrres24 at acpi0: P0SA
> > acpipwrres25 at acpi0: P3SA
> > acpipwrres26 at acpi0: P0SA
> > acpipwrres27 at acpi0: P3SA
> > acpipwrres28 at acpi0: P0SA
> > acpipwrres29 at acpi0: P3SA
> > acpipwrres30 at acpi0: P0SA
> > acpipwrres31 at acpi0: P3SA
> > ipmi at mainbus0 not configured
> > cpu0: 3500 MHz: speeds: 3500 2500 1500 MHz
> > pci0 at mainbus0 bus 192
> > ksmn0 at pci0 dev 0 function 0 "AMD 17h Root Complex" rev 0x00
> > vendor "AMD", unknown product 0x164f (class system subclass IOMMU, rev 0x01) at pci0 dev 0 function 2 not configured
> > pchb0 at pci0 dev 1 function 0 "AMD 17h Host" rev 0x00
> > pchb1 at pci0 dev 2 function 0 "AMD 17h Host" rev 0x00
> > pchb2 at pci0 dev 3 function 0 "AMD 17h Host" rev 0x00
> > ppb0 at pci0 dev 3 function 3 "AMD 17h PCIE" rev 0x00: msi
> > pci1 at ppb0 bus 193
> > xhci0 at pci1 dev 0 function 0 "ASMedia ASM1042A xHCI" rev 0x00: msix, xHCI 1.0
> > usb0 at xhci0: USB revision 3.0
> > uhub0 at usb0 configuration 1 interface 0 "ASMedia xHCI root hub" rev 3.00/1.00 addr 1
> > ppb1 at pci0 dev 3 function 4 "AMD 17h PCIE" rev 0x00: msi
> > pci2 at ppb1 bus 194
> > xhci1 at pci2 dev 0 function 0 "ASMedia ASM1042A xHCI" rev 0x00: msix, xHCI 1.0
> > usb1 at xhci1: USB revision 3.0
> > uhub1 at usb1 configuration 1 interface 0 "ASMedia xHCI root hub" rev 3.00/1.00 addr 1
> > ppb2 at pci0 dev 3 function 5 "AMD 17h PCIE" rev 0x00: msi
> > pci3 at ppb2 bus 195
> > ppb3 at pci3 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
> > pci4 at ppb3 bus 196
> > "ASPEED Technology AST2000" rev 0x41 at pci4 dev 0 function 0 not configured
> > ppb4 at pci0 dev 3 function 6 "AMD 17h PCIE" rev 0x00: msi
> > pci5 at ppb4 bus 197
> > ahci0 at pci5 dev 0 function 0 "ASMedia ASM1061 AHCI" rev 0x02: msi, AHCI 1.2
> > ahci0: port 0: 6.0Gb/s
> > scsibus1 at ahci0: 32 targets
> > sd0 at scsibus1 targ 0 lun 0: <ATA, Samsung SSD 850, EMT0> naa.5002538d40ff4db3
> > sd0: 238475MB, 512 bytes/sector, 488397168 sectors, thin
> > ppb5 at pci0 dev 3 function 7 "AMD 17h PCIE" rev 0x00: msi
> > pci6 at ppb5 bus 198
> > bge0 at pci6 dev 0 function 0 "Broadcom BCM5720" rev 0x00, BCM5720 A0 (0x5720000), APE firmware NCSI 1.5.1.0: msi, address 00:25:90:bc:ff:2a
> > brgphy0 at bge0 phy 1: BCM5720C, rev. 0
> > bge1 at pci6 dev 0 function 1 "Broadcom BCM5720" rev 0x00, BCM5720 A0 (0x5720000), APE firmware NCSI 1.5.1.0: msi, address 00:25:90:bc:ff:2b
> > brgphy1 at bge1 phy 2: BCM5720C, rev. 0
> > pchb3 at pci0 dev 4 function 0 "AMD 17h Host" rev 0x00
> > pchb4 at pci0 dev 5 function 0 "AMD 17h Host" rev 0x00
> > pchb5 at pci0 dev 7 function 0 "AMD 17h Host" rev 0x00
> > ppb6 at pci0 dev 7 function 1 "AMD 17h PCIE" rev 0x00
> > pci7 at ppb6 bus 199
> > vendor "AMD", unknown product 0x148a (class instrumentation unknown subclass 0x00, rev 0x00) at pci7 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci7 dev 0 function 2 not configured
> > pchb6 at pci0 dev 8 function 0 "AMD 17h Host" rev 0x00
> > ppb7 at pci0 dev 8 function 1 "AMD 17h PCIE" rev 0x00
> > pci8 at ppb7 bus 200
> > vendor "AMD", unknown product 0x1485 (class instrumentation unknown subclass 0x00, rev 0x00) at pci8 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci8 dev 0 function 2 not configured
> > pci9 at mainbus0 bus 128
> > ksmn1 at pci9 dev 0 function 0 "AMD 17h Root Complex" rev 0x00
> > vendor "AMD", unknown product 0x164f (class system subclass IOMMU, rev 0x01) at pci9 dev 0 function 2 not configured
> > pchb7 at pci9 dev 1 function 0 "AMD 17h Host" rev 0x00
> > pchb8 at pci9 dev 2 function 0 "AMD 17h Host" rev 0x00
> > pchb9 at pci9 dev 3 function 0 "AMD 17h Host" rev 0x00
> > ppb8 at pci9 dev 3 function 1 "AMD 17h PCIE" rev 0x00: msi
> > pci10 at ppb8 bus 129
> > ppb9 at pci9 dev 3 function 2 "AMD 17h PCIE" rev 0x00: msi
> > pci11 at ppb9 bus 130
> > pchb10 at pci9 dev 4 function 0 "AMD 17h Host" rev 0x00
> > pchb11 at pci9 dev 5 function 0 "AMD 17h Host" rev 0x00
> > pchb12 at pci9 dev 7 function 0 "AMD 17h Host" rev 0x00
> > ppb10 at pci9 dev 7 function 1 "AMD 17h PCIE" rev 0x00
> > pci12 at ppb10 bus 131
> > vendor "AMD", unknown product 0x148a (class instrumentation unknown subclass 0x00, rev 0x00) at pci12 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci12 dev 0 function 2 not configured
> > pchb13 at pci9 dev 8 function 0 "AMD 17h Host" rev 0x00
> > ppb11 at pci9 dev 8 function 1 "AMD 17h PCIE" rev 0x00
> > pci13 at ppb11 bus 132
> > vendor "AMD", unknown product 0x1485 (class instrumentation unknown subclass 0x00, rev 0x00) at pci13 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci13 dev 0 function 2 not configured
> > pci14 at mainbus0 bus 0
> > ksmn2 at pci14 dev 0 function 0 "AMD 17h Root Complex" rev 0x00
> > vendor "AMD", unknown product 0x164f (class system subclass IOMMU, rev 0x01) at pci14 dev 0 function 2 not configured
> > pchb14 at pci14 dev 1 function 0 "AMD 17h Host" rev 0x00
> > pchb15 at pci14 dev 2 function 0 "AMD 17h Host" rev 0x00
> > pchb16 at pci14 dev 3 function 0 "AMD 17h Host" rev 0x00
> > pchb17 at pci14 dev 4 function 0 "AMD 17h Host" rev 0x00
> > pchb18 at pci14 dev 5 function 0 "AMD 17h Host" rev 0x00
> > pchb19 at pci14 dev 7 function 0 "AMD 17h Host" rev 0x00
> > ppb12 at pci14 dev 7 function 1 "AMD 17h PCIE" rev 0x00
> > pci15 at ppb12 bus 1
> > vendor "AMD", unknown product 0x148a (class instrumentation unknown subclass 0x00, rev 0x00) at pci15 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci15 dev 0 function 2 not configured
> > pchb20 at pci14 dev 8 function 0 "AMD 17h Host" rev 0x00
> > ppb13 at pci14 dev 8 function 1 "AMD 17h PCIE" rev 0x00
> > pci16 at ppb13 bus 2
> > vendor "AMD", unknown product 0x1485 (class instrumentation unknown subclass 0x00, rev 0x00) at pci16 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci16 dev 0 function 2 not configured
> > xhci2 at pci16 dev 0 function 3 vendor "AMD", unknown product 0x148c rev 0x00: msix, xHCI 1.10
> > usb2 at xhci2: USB revision 3.0
> > uhub2 at usb2 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
> > piixpm0 at pci14 dev 20 function 0 "AMD FCH SMBus" rev 0x61: polling
> > iic0 at piixpm0
> > iic1 at piixpm0
> > pcib0 at pci14 dev 20 function 3 "AMD FCH LPC" rev 0x51
> > pchb21 at pci14 dev 24 function 0 vendor "AMD", unknown product 0x1650 rev 0x00
> > pchb22 at pci14 dev 24 function 1 vendor "AMD", unknown product 0x1651 rev 0x00
> > pchb23 at pci14 dev 24 function 2 vendor "AMD", unknown product 0x1652 rev 0x00
> > pchb24 at pci14 dev 24 function 3 vendor "AMD", unknown product 0x1653 rev 0x00
> > pchb25 at pci14 dev 24 function 4 vendor "AMD", unknown product 0x1654 rev 0x00
> > pchb26 at pci14 dev 24 function 5 vendor "AMD", unknown product 0x1655 rev 0x00
> > pchb27 at pci14 dev 24 function 6 vendor "AMD", unknown product 0x1656 rev 0x00
> > pchb28 at pci14 dev 24 function 7 vendor "AMD", unknown product 0x1657 rev 0x00
> > isa0 at pcib0
> > isadma0 at isa0
> > pcppi0 at isa0 port 0x61
> > spkr0 at pcppi0
> > pci17 at mainbus0 bus 64
> > ksmn3 at pci17 dev 0 function 0 "AMD 17h Root Complex" rev 0x00
> > vendor "AMD", unknown product 0x164f (class system subclass IOMMU, rev 0x01) at pci17 dev 0 function 2 not configured
> > pchb29 at pci17 dev 1 function 0 "AMD 17h Host" rev 0x00
> > pchb30 at pci17 dev 2 function 0 "AMD 17h Host" rev 0x00
> > pchb31 at pci17 dev 3 function 0 "AMD 17h Host" rev 0x00
> > pchb32 at pci17 dev 4 function 0 "AMD 17h Host" rev 0x00
> > pchb33 at pci17 dev 5 function 0 "AMD 17h Host" rev 0x00
> > pchb34 at pci17 dev 7 function 0 "AMD 17h Host" rev 0x00
> > ppb14 at pci17 dev 7 function 1 "AMD 17h PCIE" rev 0x00
> > pci18 at ppb14 bus 65
> > vendor "AMD", unknown product 0x148a (class instrumentation unknown subclass 0x00, rev 0x00) at pci18 dev 0 function 0 not configured
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci18 dev 0 function 2 not configured
> > pchb35 at pci17 dev 8 function 0 "AMD 17h Host" rev 0x00
> > ppb15 at pci17 dev 8 function 1 "AMD 17h PCIE" rev 0x00
> > pci19 at ppb15 bus 66
> > vendor "AMD", unknown product 0x1485 (class instrumentation unknown subclass 0x00, rev 0x00) at pci19 dev 0 function 0 not configured
> > ccp0 at pci19 dev 0 function 1 "AMD 17h Crypto" rev 0x00: msix
> > psp0 at ccp0: vers 2, api 1.52, build 4, SEV, SEV-ES
> > vendor "AMD", unknown product 0x1498 (class crypto subclass miscellaneous, rev 0x00) at pci19 dev 0 function 2 not configured
> > xhci3 at pci19 dev 0 function 3 vendor "AMD", unknown product 0x148c rev 0x00: msix, xHCI 1.10
> > usb3 at xhci3: USB revision 3.0
> > uhub3 at usb3 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
> > ppb16 at pci17 dev 8 function 2 "AMD 17h PCIE" rev 0x00
> > pci20 at ppb16 bus 67
> > ahci1 at pci20 dev 0 function 0 "AMD FCH AHCI" rev 0x51: msi, AHCI 1.3.1
> > scsibus2 at ahci1: 32 targets
> > ppb17 at pci17 dev 8 function 3 "AMD 17h PCIE" rev 0x00
> > pci21 at ppb17 bus 68
> > ahci2 at pci21 dev 0 function 0 "AMD FCH AHCI" rev 0x51: msi, AHCI 1.3.1
> > scsibus3 at ahci2: 32 targets
> > vmm0 at mainbus0: SVM/RVI
> > efifb0 at mainbus0: 1024x768, 32bpp
> > wsdisplay0 at efifb0 mux 1
> > wsdisplay0: screen 0-5 added (std, vt100 emulation)
> > uhub4 at uhub2 port 2 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
> > uhidev0 at uhub4 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> > uhidev0: iclass 3/1
> > ukbd0 at uhidev0: 8 variable keys, 6 key codes
> > wskbd0 at ukbd0 mux 1
> > wskbd0: connecting to wsdisplay0
> > uhidev1 at uhub4 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> > uhidev1: iclass 3/1
> > ums0 at uhidev1: 3 buttons, Z dir
> > wsmouse0 at ums0 mux 0
> > vscsi0 at root
> > scsibus4 at vscsi0: 256 targets
> > softraid0 at root
> > scsibus5 at softraid0: 256 targets
> > root on sd0a (6dcfb315de8e2773.a) swap on sd0b dump on sd0b