From: Mike Larkin <mlarkin@nested.page>
Subject: Re: vmd(8): Use 32-bit direct kernel launch for both amd64 and i386
To: tech@openbsd.org
Date: Wed, 20 Aug 2025 12:02:50 -0700

On Wed, Aug 20, 2025 at 06:38:29PM +0100, Crystal Kolipe wrote:
> On Wed, Aug 20, 2025 at 09:50:23AM -0700, Mike Larkin wrote:
> > On Wed, Aug 20, 2025 at 02:23:45PM +0100, Crystal Kolipe wrote:
> > > On Wed, Aug 20, 2025 at 02:24:34PM +0200, Hans-Jrg Hxer wrote:
> > > > The diff can be tested with amd64 and i386 ramdisk kernels like this:
> > > >
> > > > # vmctl start -c -b i386/bsd.rd myvm
> > > > # vmctl start -c -b amd64/bsd.rd myvm
> > > >
> > > > Using a BIOS boot image (eg. /etc/firmware/vmm-bios) is not affected by
> > > > this change.
> > >
> > > On a few of our systems this diff is throwing vm_resetcpu: failed, and the
> > > corresponding vms don't start.
> > >
> > > Does this change rely on any other recent commits?
> > >
> > > If not, I suspect that some of our local changes are breaking it.
> > >
> > > We have various production systems deployed with vms that boot directly in to
> > > the kernel without using a bios image, including several custom kernel
> > > configs, so I would like to test this on them, but unfortunately most of the
> > > interesting systems are running a heavily modified 7.7-release rather than
> > > -current.
> > >
> >
> > Can you provide a dmesg of one of these systems? (the host dmesg)
> >
> > We have an idea why this might be failing.
> >
> > If you don't want to provide the whole thing, just the cpuX lines would
> > be fine.
>
> Sure, here is one example:
>
> cpu0 at mainbus0: apid 32 (boot processor)
> cpu0: Intel(R) Core(TM) Ultra 7 155H, 4490.94 MHz, 06-aa-04, patch 0000001c

Thanks. Wasn't the cause we thought. We thought it might be running on a CPU
that doesn't have unrestricted guest mode, but this one's new (and the non-UG
ones are from like 2010 or so).

-ml

> cpu0: cpuid 1 edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> ecx=77fafbff<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND>
> cpu0: cpuid 6 eax=dfcff7<SENSOR,ARAT> ecx=409<EFFFREQ>
> cpu0: cpuid 7.0 ebx=239c27eb<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,SHA> ecx=994007ac<UMIP,PKU,WAITPKG,PKS> edx=fc18c410<MD_CLEAR,IBT,IBRS,IBPB,STIBP,L1DF,SSBD>
> cpu0: cpuid a vers=5, gp=8, gpwidth=48, ff=3, ffwidth=48
> cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
> cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=121<LAHF,ABM,3DNOWP>
> cpu0: cpuid 80000007 edx=100<ITSC>
> cpu0: msr 10a=d89fd6b<IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,TAA_NO,MISC_PKG_CT,ENERGY_FILT,DOITM,SBDR_SSDP_N,FBSDP_NO,PSDP_NO,RRSBA,OVERCLOCK,PBRSB_NO,GDS_NO,RFDS_NO>
> cpu0: 48KB 64b/line 12-way D-cache, 64KB 64b/line 16-way I-cache, 2MB 64b/line 16-way L2 cache, 24MB 64b/line 12-way L3 cache
> cpu0: smt 0, core 16, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 38MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.0.2.1.1.1.1, IBE
> cpu1 at mainbus0: apid 16 (application processor)
> cpu1: Intel(R) Core(TM) Ultra 7 155H, 4790.36 MHz, 06-aa-04, patch 0000001c
> cpu1: smt 0, core 8, package 0
> cpu2 at mainbus0: apid 24 (application processor)
> cpu2: Intel(R) Core(TM) Ultra 7 155H, 4790.25 MHz, 06-aa-04, patch 0000001c
> cpu2: smt 0, core 12, package 0
> cpu3 at mainbus0: apid 40 (application processor)
> cpu3: Intel(R) Core(TM) Ultra 7 155H, 4113.60 MHz, 06-aa-04, patch 0000001c
> cpu3: smt 0, core 20, package 0
> cpu4 at mainbus0: apid 48 (application processor)
> cpu4: Intel(R) Core(TM) Ultra 7 155H, 4191.55 MHz, 06-aa-04, patch 0000001c
> cpu4: smt 0, core 24, package 0
> cpu5 at mainbus0: apid 56 (application processor)
> cpu5: Intel(R) Core(TM) Ultra 7 155H, 4091.64 MHz, 06-aa-04, patch 0000001c
> cpu5: smt 0, core 28, package 0
> cpu6 at mainbus0: apid 0 (application processor)
> cpu6: not used
> cpu7 at mainbus0: apid 2 (application processor)
> cpu7: not used
> cpu8 at mainbus0: apid 4 (application processor)
> cpu8: not used
> cpu9 at mainbus0: apid 6 (application processor)
> cpu9: not used
> cpu10 at mainbus0: apid 8 (application processor)
> cpu10: not used
> cpu11 at mainbus0: apid 10 (application processor)
> cpu11: not used
> cpu12 at mainbus0: apid 12 (application processor)
> cpu12: not used
> cpu13 at mainbus0: apid 14 (application processor)
> cpu13: not used
> cpu14 at mainbus0: apid 64 (application processor)
> cpu14: not used
> cpu15 at mainbus0: apid 66 (application processor)
> cpu15: not used
>
> Cores 6 - 15 are disabled by this local patch:
>
> --- sys/arch/amd64/amd64/cpu.c.dist	Sun Feb 25 22:33:09 2024
> +++ sys/arch/amd64/amd64/cpu.c	Fri Aug 16 12:47:53 2024
> @@ -696,20 +696,29 @@
>  		 */
>  		printf("apid %d (application processor)\n", caa->cpu_apicid);
>
> -#if defined(MULTIPROCESSOR)
> -		cpu_intr_init(ci);
> -		cpu_start_secondary(ci);
> -		clockqueue_init(&ci->ci_queue);
> -		sched_init_cpu(ci);
> -		ncpus++;
> -		if (ci->ci_flags & CPUF_PRESENT) {
> -			ci->ci_next = cpu_info_list->ci_next;
> -			cpu_info_list->ci_next = ci;
> +/*
> + * Only activate the first 6 cores, which are the P-cores on this machine.
> + */
> +
> +		if (cpunum < 6) {
> +		#if defined(MULTIPROCESSOR)
> +				cpu_intr_init(ci);
> +				cpu_start_secondary(ci);
> +				clockqueue_init(&ci->ci_queue);
> +				sched_init_cpu(ci);
> +				ncpus++;
> +				if (ci->ci_flags & CPUF_PRESENT) {
> +					ci->ci_next = cpu_info_list->ci_next;
> +					cpu_info_list->ci_next = ci;
> +				}
> +		#else
> +				printf("%s: not started\n", sc->sc_dev.dv_xname);
> +		#endif
> +				break;
> +		} else {
> +		printf ("%s: not used\n", sc->sc_dev.dv_xname);
> +		break ;
>  		}
> -#else
> -		printf("%s: not started\n", sc->sc_dev.dv_xname);
> -#endif
> -		break;
>
>  	default:
>  		panic("unknown processor type??");
>
> As noted previously, this is 7.7-release with local patches unrelated to vmd,
> and loadfile_elf.c updated to HEAD, with hshoexer@'s patch applied.
>
> Note: with just the previous patch by bluhm@ applied, (I.E. loadfile_elf.c at
> HEAD), vmd works as expected, (no regressions).