From: Crystal Kolipe <kolipe.c@exoticsilicon.com>
Subject: Re: vmd(8): Use 32-bit direct kernel launch for both amd64 and i386
To: Mike Larkin <mlarkin@nested.page>
Cc: tech@openbsd.org
Date: Wed, 20 Aug 2025 18:38:29 +0100

On Wed, Aug 20, 2025 at 09:50:23AM -0700, Mike Larkin wrote:
> On Wed, Aug 20, 2025 at 02:23:45PM +0100, Crystal Kolipe wrote:
> > On Wed, Aug 20, 2025 at 02:24:34PM +0200, Hans-Jrg Hxer wrote:
> > > The diff can be tested with amd64 and i386 ramdisk kernels like this:
> > >
> > > # vmctl start -c -b i386/bsd.rd myvm
> > > # vmctl start -c -b amd64/bsd.rd myvm
> > >
> > > Using a BIOS boot image (eg. /etc/firmware/vmm-bios) is not affected by
> > > this change.
> >
> > On a few of our systems this diff is throwing vm_resetcpu: failed, and the
> > corresponding vms don't start.
> >
> > Does this change rely on any other recent commits?
> >
> > If not, I suspect that some of our local changes are breaking it.
> >
> > We have various production systems deployed with vms that boot directly in to
> > the kernel without using a bios image, including several custom kernel
> > configs, so I would like to test this on them, but unfortunately most of the
> > interesting systems are running a heavily modified 7.7-release rather than
> > -current.
> >
> 
> Can you provide a dmesg of one of these systems? (the host dmesg)
> 
> We have an idea why this might be failing.
> 
> If you don't want to provide the whole thing, just the cpuX lines would
> be fine.

Sure, here is one example:

cpu0 at mainbus0: apid 32 (boot processor)
cpu0: Intel(R) Core(TM) Ultra 7 155H, 4490.94 MHz, 06-aa-04, patch 0000001c
cpu0: cpuid 1 edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> ecx=77fafbff<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND>
cpu0: cpuid 6 eax=dfcff7<SENSOR,ARAT> ecx=409<EFFFREQ>
cpu0: cpuid 7.0 ebx=239c27eb<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,SHA> ecx=994007ac<UMIP,PKU,WAITPKG,PKS> edx=fc18c410<MD_CLEAR,IBT,IBRS,IBPB,STIBP,L1DF,SSBD>
cpu0: cpuid a vers=5, gp=8, gpwidth=48, ff=3, ffwidth=48
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=121<LAHF,ABM,3DNOWP>
cpu0: cpuid 80000007 edx=100<ITSC>
cpu0: msr 10a=d89fd6b<IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,TAA_NO,MISC_PKG_CT,ENERGY_FILT,DOITM,SBDR_SSDP_N,FBSDP_NO,PSDP_NO,RRSBA,OVERCLOCK,PBRSB_NO,GDS_NO,RFDS_NO>
cpu0: 48KB 64b/line 12-way D-cache, 64KB 64b/line 16-way I-cache, 2MB 64b/line 16-way L2 cache, 24MB 64b/line 12-way L3 cache
cpu0: smt 0, core 16, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 38MHz
cpu0: mwait min=64, max=64, C-substates=0.2.0.2.1.1.1.1, IBE
cpu1 at mainbus0: apid 16 (application processor)
cpu1: Intel(R) Core(TM) Ultra 7 155H, 4790.36 MHz, 06-aa-04, patch 0000001c
cpu1: smt 0, core 8, package 0
cpu2 at mainbus0: apid 24 (application processor)
cpu2: Intel(R) Core(TM) Ultra 7 155H, 4790.25 MHz, 06-aa-04, patch 0000001c
cpu2: smt 0, core 12, package 0
cpu3 at mainbus0: apid 40 (application processor)
cpu3: Intel(R) Core(TM) Ultra 7 155H, 4113.60 MHz, 06-aa-04, patch 0000001c
cpu3: smt 0, core 20, package 0
cpu4 at mainbus0: apid 48 (application processor)
cpu4: Intel(R) Core(TM) Ultra 7 155H, 4191.55 MHz, 06-aa-04, patch 0000001c
cpu4: smt 0, core 24, package 0
cpu5 at mainbus0: apid 56 (application processor)
cpu5: Intel(R) Core(TM) Ultra 7 155H, 4091.64 MHz, 06-aa-04, patch 0000001c
cpu5: smt 0, core 28, package 0
cpu6 at mainbus0: apid 0 (application processor)
cpu6: not used
cpu7 at mainbus0: apid 2 (application processor)
cpu7: not used
cpu8 at mainbus0: apid 4 (application processor)
cpu8: not used
cpu9 at mainbus0: apid 6 (application processor)
cpu9: not used
cpu10 at mainbus0: apid 8 (application processor)
cpu10: not used
cpu11 at mainbus0: apid 10 (application processor)
cpu11: not used
cpu12 at mainbus0: apid 12 (application processor)
cpu12: not used
cpu13 at mainbus0: apid 14 (application processor)
cpu13: not used
cpu14 at mainbus0: apid 64 (application processor)
cpu14: not used
cpu15 at mainbus0: apid 66 (application processor)
cpu15: not used

Cores 6 - 15 are disabled by this local patch:

--- sys/arch/amd64/amd64/cpu.c.dist	Sun Feb 25 22:33:09 2024
+++ sys/arch/amd64/amd64/cpu.c	Fri Aug 16 12:47:53 2024
@@ -696,20 +696,29 @@
 		 */
 		printf("apid %d (application processor)\n", caa->cpu_apicid);
 
-#if defined(MULTIPROCESSOR)
-		cpu_intr_init(ci);
-		cpu_start_secondary(ci);
-		clockqueue_init(&ci->ci_queue);
-		sched_init_cpu(ci);
-		ncpus++;
-		if (ci->ci_flags & CPUF_PRESENT) {
-			ci->ci_next = cpu_info_list->ci_next;
-			cpu_info_list->ci_next = ci;
+/*
+ * Only activate the first 6 cores, which are the P-cores on this machine.
+ */
+
+		if (cpunum < 6) {
+		#if defined(MULTIPROCESSOR)
+				cpu_intr_init(ci);
+				cpu_start_secondary(ci);
+				clockqueue_init(&ci->ci_queue);
+				sched_init_cpu(ci);
+				ncpus++;
+				if (ci->ci_flags & CPUF_PRESENT) {
+					ci->ci_next = cpu_info_list->ci_next;
+					cpu_info_list->ci_next = ci;
+				}
+		#else
+				printf("%s: not started\n", sc->sc_dev.dv_xname);
+		#endif
+				break;
+		} else {
+		printf ("%s: not used\n", sc->sc_dev.dv_xname);
+		break ;
 		}
-#else
-		printf("%s: not started\n", sc->sc_dev.dv_xname);
-#endif
-		break;
 
 	default:
 		panic("unknown processor type??");

As noted previously, this is 7.7-release with local patches unrelated to vmd,
and loadfile_elf.c updated to HEAD, with hshoexer@'s patch applied.

Note: with just the previous patch by bluhm@ applied, (I.E. loadfile_elf.c at
HEAD), vmd works as expected, (no regressions).