From: Mark Patruck Subject: Re: Use acpipci in VMs (was: PCI BAR mapping in qemu VMs) To: Stefan Fritsch Cc: tech@openbsd.org, Mark Kettenis Date: Fri, 21 Mar 2025 15:38:59 +0100 On 21.03.2025 14:24, Stefan Fritsch wrote: >Hi, > >On Sun, 16 Mar 2025, Stefan Fritsch wrote: >> there were some reports that vio on KVM/qemu sometimes panics with >> >> vq_size not power of two: 65535 >> >> but I could never reproduce it. bluhm@ now got me a test setup where the >> bsd kernel is PXE booted on qemu in 440fx mode, and there it is >> reproducible. >> >> After some debugging it seems that seabios or ipxe maps the PCI BARs at >> 0x380000000000-0x380080000000 which is outside the allowed range in >> pci_init_extents(). On the other hand, in 440fx mode, qemu seems to >> produce ACPI 1.x tables and there is a check in acpipci_attach() that for >> ACPI < 5.x, the PCI infos from _CRS are not used. OpenBSD will then >> disable the BARs and when mapping them again in vio_attach(), it will >> sometimes choose adresses that do not work, reads return 0xff and writes >> are ignored. I guess this is becuase the address (in my case 0xbff14000) >> lies outside the PCI window of the emulated chipset. >> >> I have put dmesg, acpi tables and other info at >> https://www.sfritsch.de/~stf/vq-panic/ >> >> Qemu in q35 mode produces ACPI 3.x tables, so it may also be affected. >> >> There may be three ways to fix this: >> >> 1) increase the allowed range for pcimem in pci_init_extents(). This is >> what the diff below does. >> >> 2) somehow make acpipci_attach() use the ACPI infos on qemu. I have >> verified that removing the version check fixes the issue. Since removing >> the version check seems to break many other systems, this would have to be >> a qemu specific quirk. >> >> 3) try to make OpenBSD reliably map the BARs somewhere where it works. Is >> there a way for OpenBSD to get the info where the PCI window is without >> trusting ACPI? >> >> I remember at least one report of this issue on i386. Any idea how to fix >> it there? > >Mark Patruck noticed that these issues seem to be caused by some >relatively recent changes in seabios. > >https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/message/R7FOQMMYWVX577QNIA2AKUAGOZKNJIAP/ >https://gitlab.com/qemu-project/seabios/-/commit/df9dd418b3b0e586cb208125094620fc7f90f23d > >A workaround seems to be to configure the VM with <= 3GB memory. > >The problem may become more wide-spread with 7.7, since we now default to >virtio 1.x, which uses MMIO on qemu, compared to virtio 0.9 which uses PIO >BARs. Therefore it would be nice to get a fix in before the release, if it >is not too late already. > >The diff below uses acpipci / _CRS also with old ACPI versions if running >on a hypervisor. I think the chance that it will break unrelated systems >is low. It does not change behavior on vmd, where no acpi attaches at all. With the first solution (1), as well as the diff below (2) i don't get any conflicts like 0:28:0: bridge mem address conflict 0x383800000000/0x800000000 0:28:1: bridge mem address conflict 0x383000000000/0x800000000 0:28:2: bridge mem address conflict 0x382800000000/0x800000000 0:28:3: bridge mem address conflict 0x382000000000/0x800000000 0:30:0: bridge mem address conflict 0x380000000000/0x2000000000 5:1:0: bridge mem address conflict 0x381800000000/0x800000000 5:4:0: bridge mem address conflict 0x380000000000/0x800000000 6:3:0: mem address conflict 0x381800000000/0x4000 6:18:0: mem address conflict 0x381800004000/0x4000 9:1:0: mem address conflict 0x380000000000/0x4000 anymore and also passed through devices work flawlessly ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW 9.152.77998 API 1.15, msix, 4 queues, address 40:a6:b7:c0:b5:b2 ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW 9.152.77998 API 1.15, msix, 4 queues, address 40:a6:b7:c0:b5:b3 when running QEMU >=9.0. Both solutions work. >ok? > > >diff --git a/sys/arch/amd64/pci/acpipci.c b/sys/arch/amd64/pci/acpipci.c >index 51cd1360383..2e3236772bb 100644 >--- a/sys/arch/amd64/pci/acpipci.c >+++ b/sys/arch/amd64/pci/acpipci.c >@@ -152,7 +152,7 @@ acpipci_attach(struct device *parent, struct device *self, void *aux) > > aml_parse_resource(&res, acpipci_parse_resources, sc); > >- if (sc->sc_acpi->sc_major < 5) { >+ if (sc->sc_acpi->sc_major < 5 && (cpu_ecxfeature & CPUIDECX_HV) == 0) { > extent_destroy(sc->sc_ioex); > extent_destroy(sc->sc_memex); > > -- Mark Patruck ( mark at wrapped.cx ) GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51 https://www.wrapped.cx