Repository navigation
docs: use kvm-arm.vgic_v3_group1_trap=1 for Lima microVM setup on arm64 - #3149
Merged
Scott Funkenhauser (sfunkenhauser) merged 1 commit intoOct 23, 2026
Conversation
On arm64 kernels 6.19 and newer, KVM clears the active bit on GICv3 LPI list registers during VM exits (Linux commit 73c9726975af). Under nested virtualization, hardware EOI in the L2 microVM guest does not clear ICH_AP1R0_EL2 and stalls subsequent virtio interrupts. Setting kvm-arm.vgic_v3_group1_trap=1 on the Lima guest kernel command line traps Group 1 EOI writes to EL2 so KVM clears the active priority in software. This allows docs/dev/microvm-local.md to use the default Ubuntu 26.04 Lima image instead of pinning to Ubuntu 24.04 and installing a newer git from a PPA.
Benjamin Elder (BenTheElder)
added
kind/docs
2 tasks done
Scott Funkenhauser (sfunkenhauser)
approved these changes
Oct 23, 2026
Collaborator
Author
|
NOTE: Running linux amd64 containers with emulation under rosetta, at least on my M4, will fail on this newer kernel, rosetta is missing some instructions. you can instead use qemu but ... it will be slow. So there are reasons to hold back, in the short term, and not necessarily rush to adopt this from a working setup. However, we should just ... NOT do that, it's a huge pitfall, and relatively easily avoided, that one is #3152 but I need to talk to tim/dima/eitan about if this is in scope or post-freeze. |
Davanum Srinivas (dims)
pushed a commit
to dims/substrate
that referenced
this pull request
Oct 23, 2026
…-substrate#3152) <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Upgrading to current Ubuntu in limactl (agent-substrate#3149) on macOS led to emulation errors with rosetta. Using qemu is super slow. We should just cross-compile, pretty much always, that's a key benefit of buildx and multi-stage builds. It's WAY faster. Compiling under emulation is major kludge. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, I let it draft this. Then hand edited, bash and dockerfiles I'm quite familiar with :-) <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On arm64 kernels 6.19 and newer, KVM clears the active bit on GICv3 LPI list registers during VM exits (Linux commit 73c9726975af, reverting that also works but isn't clearly the right fix).
Under nested virtualization, hardware EOI in the L2 microVM guest does not clear ICH_AP1R0_EL2 and stalls subsequent virtio interrupts, hanging microVM boot.
This was found by bisecting the kernel previously, which we "solved" by pinning to <= 6.18.
This is commonly encountered when setting up limactl defaults, enabling nested virtualization, and then testing substrate on an apple silicon mac.
Setting
kvm-arm.vgic_v3_group1_trap=1on the Lima guest kernel command line traps Group 1 EOI writes to EL2 so KVM clears the active priority in software.Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.)
This allows our developer docs to use the default Ubuntu 26.04 Lima image instead of pinning to Ubuntu 24.04 and installing a newer git from a PPA.
FYI Scott Funkenhauser (@sfunkenhauser) I finally got back to this and found a suitable workaround.
Still have not file the kernel bug thread, I haven't had time to figure out how to do that the right way without sending them unwanted ~slop ... we should do that at some point.
AI use: Yes, absolutely, I do not have time to bisect the kernel :-), I however vetted this change.