From: Mike Larkin <mlarkin@nested.page>
Subject: Re: SEV-ES multiprocessor support / handle WBINDV
To: Stefan Fritsch <sf@sfritsch.de>
Cc: tech@openbsd.org
Date: Tue, 25 Nov 2025 11:14:14 -0800

On Tue, Nov 25, 2025 at 08:02:28PM +0100, Stefan Fritsch wrote:
> Hi Mike,
>
> On Tue, 25 Nov 2025, Mike Larkin wrote:
>
> > On Tue, Nov 25, 2025 at 10:41:45AM +0100, Stefan Fritsch wrote:
> > > Hi,
> > >
> > > On Thu, 18 Sep 2025, Stefan Fritsch wrote:
> > > > one remaining problem with SEV-ES is that we don't support multiprocessor
> > >
> > > for multiprocessor support with SEV-ES, we need to handle WBINDV in the
> > > VC trap handler.
> > >
> > > This is the the first part of the larger diff that I have sent in the
> > > mail quoted above.
> > >
> > > ok?
> > >
> >
> > I don't understand this. We're saying if the guest does a WBINVD, we inject that
> > back into the guest via a #VC, but then in the trap handler we ... do nothing?
> > We just advance %rip?
> >
> > What happened to the guest's desired wbinvd/wbnoinvd?
>
> Sorry, I should have sent a diff with longer context. After the switch
> statement we copy the exit code to the GHCB and then exit to the
> hypervisor. As WBINVD does not take any arguments, no more information or
> handling is neccessary. Diff with more context below.
>
> Cheers,
> Stefan
>

I'm still not seeing it. Sorry if I'm being dense here.

The context below helps. So we take a #VC trap for our (guest's) own
WBINVD/WBNOINVD, then call vmgexit and end up in svm_handle_vmgexit in
vmm_machdep.c . It's unclear from this diff whether or not we are using the
GHCB MSR protocol, but regardless if we are or aren't, there is no provision
for handling these instructions in vmm in either case. So we take a needless
exit and then return back to the guest after making a round-trip journey to
vmm to ... do nothing?

Since this is just a wbinvd, why don't we just handle it in the #VC trap
handler? There should not be any side effects of handling it ourselves here.

Eg, in the case SVM_VMEXIT_WBINVD below, just call wbinvd, advance tf_rip, and
then iret? (maybe special case needed for wbnoinvd?)

What am I missing?

-ml

> --- a/sys/arch/amd64/amd64/trap.c
> +++ b/sys/arch/amd64/amd64/trap.c
> @@ -428,40 +428,50 @@ vctrap(struct trapframe *frame, int user, int *sig, int *code)
> ...
> +	case SVM_VMEXIT_WBINVD:
> +		/*
> +		 * There is no special GHCB request for WBNOINVD.
> +		 * Signal WBINVD to emulate WBNOINVD.
> +		 */
> +		if (*rip == 0xf3)
> +			frame->tf_rip += 3;
> +		else
> +			frame->tf_rip += 2;
> +		break;
>  	default:
>  		panic("invalid exit code 0x%llx", ghcb_regs.exitcode);
>  	}
>
>  	/* Always required */
>  	ghcb_sync_val(GHCB_SW_EXITCODE, GHCB_SZ64, &syncout);
>  	ghcb_sync_val(GHCB_SW_EXITINFO1, GHCB_SZ64, &syncout);
>  	ghcb_sync_val(GHCB_SW_EXITINFO2, GHCB_SZ64, &syncout);
>
>  	/* Sync out to GHCB */
>  	ghcb = (struct ghcb_sa *)ghcb_vaddr;
>  	ghcb_sync_out(frame, &ghcb_regs, ghcb, &syncout);
>
>  	wrmsr(MSR_SEV_GHCB, ghcb_paddr);
>
>  	/* Call hypervisor. */
>  	vmgexit();
>
>  	/* Verify response */
>  	if (ghcb_verify_bm(ghcb->valid_bitmap, syncin.valid_bitmap)) {
>
>