Skip to content

[windows] Add an Impeller Vulkan backend presented through DirectComposition#189586

Draft
sero583 wants to merge 5 commits into
flutter:masterfrom
sero583:impeller-vulkan-windows-dcomp
Draft

[windows] Add an Impeller Vulkan backend presented through DirectComposition#189586
sero583 wants to merge 5 commits into
flutter:masterfrom
sero583:impeller-vulkan-windows-dcomp

Conversation

@sero583

@sero583 sero583 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Adds an opt-in Vulkan rendering path to the Windows embedder: Impeller renders with its existing Vulkan backend, and the content is presented through DXGI and DirectComposition, so the OS compositor owns presentation. There is no VkSurfaceKHR and no Vulkan swapchain anywhere in this path, addressing the review feedback on #183382 that a KHR swapchain would block compositor integration, resize synchronization, and future platform views.

Activation uses the engine switch mechanism, in line with the review guidance to gate the backend behind flags rather than public API. The engine accepts --impeller-backend=vulkan next to the existing --enable-impeller handling; until the flutter_tools plumbing lands in a follow-up, the switch reaches the engine through the standard FLUTTER_ENGINE_SWITCHES environment protocol (debug and profile only):

set FLUTTER_ENGINE_SWITCHES=2
set FLUTTER_ENGINE_SWITCH_1=enable-impeller=true
set FLUTTER_ENGINE_SWITCH_2=impeller-backend=vulkan

Without the explicit backend switch, or when no capable device exists, nothing changes: ANGLE remains the default and the fallback. When Vulkan is active, ANGLE is not initialized at all.

How it works

CompositorVulkan (FlutterCompositor)
  backing store = shared D3D11 texture (keyed mutex, NT handle)
                  imported into Vulkan, wrapped as the resolve target of an
                  MSAA color attachment with a depth-stencil (matching
                  SurfaceVK::WrapSwapchainImage) that Impeller renders into
  present       = keyed-mutex bracketed CopyResource into a flip-model
                  composition swapchain, Present(0), DComp commit
  • VulkanManager initializes the instance and device with dynamic loading (vulkan-1.dll; missing runtimes fall back before engine start) and no windowing extensions. Device selection requires a graphics queue, a valid adapter LUID, and Direct3D 11 texture import support, verified per device through vkGetPhysicalDeviceImageFormatProperties2 before selection, so a successful Create() guarantees the interop works. Every rejection path logs its reason for field diagnosis.
  • DCompPresenter owns a Direct3D 11 device on the LUID-matched adapter, allocates the shared textures, and presents them through a composition swapchain rooted in a DirectComposition visual on the window. Because content reaches the screen only through the DComp commit, the window chrome and the Flutter content resize atomically.
  • VulkanImportedImage binds a VkImage to the shared texture's memory (dedicated allocation, VkImportMemoryWin32HandleInfoKHR).
  • The interop direction is dictated by the drivers: Direct3D 11 texture memory is reported to Vulkan as import-only (measured on AMD Adrenalin: DEDICATED_ONLY | IMPORTABLE), so the allocation originates on the Direct3D side. This matches the direction used by Chromium and Dawn.
  • CompositorVulkan renders each backing store with Impeller. The imported image is reported as a swapchain image so the render pass leaves it in VK_IMAGE_LAYOUT_GENERAL rather than a sampled layout the imported image does not support. GPUSurfaceVulkanImpeller gains a render_to_surface flag: with an external view embedder present the root surface is a no-op and all content arrives through backing stores, matching the OpenGL Impeller path.
  • The SDF rendering option is an ANGLE/OpenGL Impeller default and is not applied when the Vulkan backend is active.

Synchronization

The engine performs a host sync for all layers before invoking the compositor present callback (documented in FlutterVulkanBackingStore), so Vulkan writes are complete before the copy. The Direct3D side still brackets its access with the shared texture's keyed mutex, which performs the cross-device flushes.

Depends on

The Impeller hardening PR (#189580), which includes the CapabilitiesVK change that allows an embedder to create a surfaceless Vulkan context (no VK_KHR_surface / swapchain), required by this presentation path.

Not in this PR

  • Platform views: the compositor rejects multi-layer scenes with a clear error. The visual-tree architecture is what makes them possible later.
  • flutter_tools changes: activation is manual until the backend matures.
  • Any embedder API change: the existing FlutterVulkanRendererConfig, FlutterCompositor, and FlutterVulkanBackingStore surfaces are used as-is. The engine-side kFlutterBackingStoreTypeVulkan Impeller path is implemented here (it previously logged "Unimplemented").

Testing

Regression tests run in flutter_windows_unittests and impeller_unittests; hardware-dependent tests skip cleanly on machines without Vulkan:

  • VulkanManagerTest (7): dynamic loading, create-or-null, LUID validity, required extensions present, no windowing extensions ever, proc resolution, teardown.
  • VulkanImportedImageTest: argument validation.
  • DCompPresenterTest (5): unknown LUID rejection, present-before-bind rejection, the full pipeline (allocate, import, keyed-mutex present through a live DComp tree, evict), swapchain resize in place, and an 8-cycle allocate/import/evict lifecycle.
  • FlutterWindowsEngineTest: the backend requires the explicit switch, is ignored without Impeller, either activates (ANGLE skipped) or falls back without failing engine construction, and does not enable SDF rendering when Vulkan is active.
  • GPUSurfaceVulkanImpeller.RenderToSurfaceDisabledYieldsNoOpFrame: with an external view embedder the root surface yields a no-op frame and never pulls an image from the delegate.

Performance

Release-mode comparison of this Vulkan/DComp path against the default ANGLE/DXGI path (Impeller both sides), AMD Radeon RX 6750 XT, Windows 10, same benchmark app and binary:

Scene ANGLE/DXGI Vulkan/DComp
Transform & Clip 13.8 96.1
Image Composition 31.2 143.5
Custom Painter Heavy 90.3 108.8
Widget Cascade 136.7 138.3
Opacity Tree 143.8 49.0
Shader Mask Matrix 96.8 22.5
Geomean (avg fps) 39.7 43.9

Native Vulkan wins substantially on composition/clip/transform-heavy scenes (no ANGLE GL-to-D3D11 translation), and regresses on offscreen-layer-heavy scenes (save layers, shader masks), where the 1%-low frame rate drops sharply. Net a modest geomean gain with a clear optimization target (the offscreen/save-layer path) called out honestly rather than cherry-picked.

Manual verification on AMD Radeon RX 6750 XT (Windows 10): the backend activates, renders the full 9-scene benchmark suite in both debug and release AOT builds with no Vulkan validation errors under VK_LAYER_KHRONOS_validation, and presents through the DirectComposition tree with correct visuals throughout the runs. Window resizing is exercised by DCompPresenterTest.ResizeRecreatesSwapchainBuffers; because presentation goes through a single DComp commit, chrome and content resize atomically by construction.

Part of #181711. Design doc: https://flutter.dev/go/impeller-backend-desktop

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

sero583 added 3 commits July 16, 2026 14:37
…urce exhaustion

Stability fixes for the Vulkan backend discovered through sustained
stress testing on AMD RDNA 2 hardware and Mesa dzn (Vulkan on D3D12 in
WSL2), first published as part of flutter#183382 and rebased onto the current
FenceWaiterVK submission architecture.

Device loss handling:
- ContextVK gains an atomic device_lost_ flag. CreateCommandBuffer()
  and CommandQueueVK::Submit() short-circuit once the device is lost,
  since any Vulkan call on a driver that entered a corrupted OOM state
  can access-fault inside the ICD.
- On vkQueueSubmit failure, tracked objects are abandoned without
  making Vulkan calls (AbandonForDriverCrash on command pools and
  descriptor pools), the device is marked lost, and per-heap memory
  budgets are logged for diagnosis.
- FenceWaiterVK releases rather than destroys the fence when the
  submit callback fails; some drivers partially track the fence even
  though the submission failed, and vkDestroyFence then crashes
  (VUID-vkDestroyFence-fence-01120 observed on AMD).

Submission backpressure:
- CommandQueueVK caps concurrent in-flight submissions (3) with a
  condition variable, releasing the slot from the fence completion
  callback. Prevents unbounded host memory growth when the CPU
  outpaces the GPU.
- ContextVK::FlushCommandBuffers() submits in chunks of 64 to keep
  per-submit driver allocations bounded on frames that accumulate very
  large command buffer counts.
- GPUSurfaceVulkanImpeller (embedder-managed image path) adds a
  two-entry frame fence ring mirroring the KHR swapchain's
  backpressure, with a bounded wait that skips the frame under extreme
  GPU pressure. Image views for embedder images are now cached instead
  of being allocated (and leaked) per frame, and in-flight frames are
  drained before views are destroyed on resize or teardown.

Command pool lifecycle:
- BackgroundCommandPoolVK releases its handles without Vulkan calls
  when the recycler is gone; the device may already be destroyed at
  that point. This is the same class of teardown race as the
  CommandPoolVK destructor fix that landed in flutter#186749.
- The recycled pool list is capped at 8 entries, evicting oldest
  first.
- unused_command_buffers_ is annotated IPLR_GUARDED_BY(pool_mutex_).

Descriptor pools:
- Reset via vkResetDescriptorPool on reclaim. Reclaimed pools
  previously stayed at capacity and produced
  VK_ERROR_OUT_OF_POOL_MEMORY on every reuse.

Barrier and render pass correctness:
- Six barrier fixes in the blit pass: pre-copy barriers use transfer
  stage and access flags, post-copy barriers flush transfer writes,
  and mipmap generation barriers match the actual current layout.
  Fixes corrupted glyph atlases and SYNC-HAZARD-WRITE-AFTER-WRITE
  reported by the synchronization validation layer on AMD RDNA 2.
- RenderPassBuilderVK moves subpass dependencies to a counted builder,
  drops eByRegion from external (VK_SUBPASS_EXTERNAL) dependencies
  where it has no meaning per Vulkan spec section 7.1, and gains
  SetFramebufferFetchEnabled() so input attachment references and
  self-dependencies are omitted on drivers without framebuffer fetch
  support. The flag is wired through pipeline and render pass creation
  and the compat render pass used for PSO construction.

Driver workarounds:
- DriverInfoVK exposes GetDriverID() (Vulkan 1.2) and detects Mesa
  dzn.
- New skip_sub_region_buffer_to_image_copy workaround: Mesa dzn
  reports minImageTransferGranularity of (0,0,0) and corrupts textures
  on sub-region buffer-to-image copies; the blit pass stages such
  copies through a full-size temporary texture.

Memory management:
- The VMA buffer pool is capped at 256 blocks (about 1 GB); the pool
  otherwise grows without bound when high-water-mark demand spikes and
  fragmentation prevents block reuse.
- Image allocation failure paths no longer leak the VMA allocation
  when image view creation fails.
- On Windows, the working set is periodically trimmed when Resizable
  BAR mappings inflate RSS far beyond actual VMA usage (gated by
  cooldown, RSS threshold, and RSS-to-VMA ratio).

KHR swapchain robustness:
- The synchronizer fence is re-signaled with an empty submit on every
  early return after a failed or unusable acquireNextImageKHR
  (including the surface-lost path added in flutter#183338). The fence was
  reset by WaitForFence, and without a new signal the next wrap-around
  to the same synchronizer deadlocks in waitForFences with an infinite
  timeout.
- AcquireNextDrawable short-circuits when the device is marked lost.
- Zero-extent surfaces (minimized windows) skip the frame instead of
  attempting swapchain recreation, preserving recovery on restore.

Embedder mode:
- CapabilitiesVK enumerates instance layers in embedder mode so that
  validation layers can be detected and enabled when requested.
…ottle tests

Adjust the descriptor pool changes to keep the current set-recycling
semantics (used sets return to the unused cache on reclaim) instead of
resetting pools on reclaim, which would have invalidated the cached set
handles that DescriptorsAreRecycled encodes as intended behavior. The
safety fixes remain: AbandonForDriverCrash, handle release on teardown
when the context or recycler is gone, propagating CreateNewPool
failures, and registering a new descriptor set in the cache only after
the allocation succeeded (a garbage handle was previously cached when
vkAllocateDescriptorSets failed).

Guard the DeathRattle test helper against invoking a moved-from
callback. WaitForReclaim moves the local instance into a
UniqueResourceVKT, and the moved-from local still runs the destructor;
calling the empty std::function aborts with bad_function_call. This
reproduces on unmodified master when running impeller_unittests on a
Windows host and made every CommandPoolRecyclerVKTest case crash there.

New tests:
- CommandQueueVKTest.SubmitAfterDeviceLostIsCancelled: no submission
  reaches the queue once the device is marked lost.
- CommandQueueVKTest.ThrottleAllowsSustainedSubmissions: in-flight
  slots are released as submissions complete.
- ContextVKTest.CreateCommandBufferShortCircuitsAfterDeviceLost.
An embedder that presents rendered images itself, for example through a
platform compositor, supplies a Vulkan device without the surface, WSI, or
swapchain extensions. CapabilitiesVK previously rejected such a device
because it required VK_KHR_surface and a windowing-system integration
extension. Relax those requirements in embedder mode when the surface
extension is absent, so presentation becomes the embedder's
responsibility. Embedders that do provide the surface extension are
unaffected.

Adds regression tests covering the surfaceless case and the case where
the embedder still provides the surface extension.
@github-actions github-actions Bot added platform-android Android applications specifically engine flutter/engine related. See also e: labels. platform-windows Building on or for Windows specifically a: desktop Running on desktop e: impeller Impeller rendering backend issues and features requests team-android Owned by Android platform team team-windows Owned by the Windows platform team labels Jul 16, 2026
Follow-ups to the first review pass on the Vulkan hardening change:

- AllocatedTextureSourceVK: destroy the already-created image views
  before the image when a render target view fails to create partway
  through the per-subresource view loop.
- BlitPassVK: pick the destination barrier's source scope from the
  texture's tracked layout in OnCopyTextureToTextureCommand, mirroring
  OnCopyBufferToTextureCommand. The unconditional eTransferWrite scope
  can trip best-practices validation when the destination was last
  sampled; the WAW protection for back-to-back transfer writes in the
  atlas growth path is preserved in the eTransferDstOptimal branch.
- AllocatorVK: make the Windows working-set trim cooldown a
  process-wide atomic instead of thread_local, since EmptyWorkingSet
  affects the whole process.
- GPUSurfaceVulkanImpeller: on a lost device, abandon the cached image
  views and frame fences instead of waiting on or destroying them, in
  both the destructor and the resize drain, matching the
  AbandonForDriverCrash policy. Adds a teardown-after-device-loss test.
@sero583
sero583 force-pushed the impeller-vulkan-windows-dcomp branch from cf1f238 to 7c847d7 Compare July 17, 2026 11:16
…osition

Adds an opt-in Vulkan rendering path to the Windows embedder: Impeller
renders with its existing Vulkan backend and the content is presented
through DXGI and DirectComposition, so the OS compositor owns
presentation. There is no VkSurfaceKHR and no Vulkan swapchain in this
path, addressing the review feedback on the original proposal that a
KHR swapchain would block compositor integration, resize
synchronization, and future platform views.

Activation uses engine switches, the same mechanism as the existing
Impeller opt-in, and requires both switches:

  --enable-impeller --impeller-backend=vulkan

Nothing changes otherwise; ANGLE remains the default and the fallback.
Wiring these switches into the flutter tool's command line is left to a
follow-up, so for now they are supplied through an embedder's switch
list or the engine switch environment protocol.

Components:

- VulkanManager: dynamic vulkan-1.dll loading (a missing runtime falls
  back before engine start), instance and device creation with no
  windowing extensions. Device selection requires a graphics queue, a
  valid adapter LUID, and Direct3D 11 texture import support, verified
  per device via vkGetPhysicalDeviceImageFormatProperties2, so a
  successful Create() guarantees the interop works. Every rejection
  logs its reason.
- DCompPresenter: a Direct3D 11 device on the LUID-matched adapter
  allocates the shared render target textures (keyed mutex, NT handle)
  and presents them by copying into a flip-model composition swapchain
  rooted in a DirectComposition visual. Content reaches the screen
  only through the DComp commit, so window chrome and Flutter content
  resize atomically.
- VulkanImportedImage: binds a VkImage to a shared texture's memory
  (dedicated allocation, VkImportMemoryWin32HandleInfoKHR). The import
  direction is dictated by the drivers: Direct3D 11 texture memory is
  reported to Vulkan as import-only (measured on AMD Adrenalin as
  DEDICATED_ONLY | IMPORTABLE), so the allocation originates on the
  Direct3D side, the same direction Chromium and Dawn use.
- CompositorVulkan: a FlutterCompositor whose backing stores pair a
  shared texture with its Vulkan import. The engine host-syncs all
  layers before the present callback, and the Direct3D side brackets
  its access with the keyed mutex. Platform views are rejected with a
  clear error; the visual tree architecture is what enables them
  later.
- Impeller Vulkan backing stores: the engine renders each compositor
  backing store into the imported image with Impeller. The image is
  wrapped as the resolve target of an MSAA color attachment with a
  depth-stencil, matching SurfaceVK::WrapSwapchainImage, and reported
  as a swapchain image so the render pass leaves it in
  VK_IMAGE_LAYOUT_GENERAL rather than a sampled layout the imported
  image does not support. GPUSurfaceVulkanImpeller gains a
  render_to_surface flag: when an external view embedder is present the
  root surface is a no-op and all content arrives through backing
  stores, matching the OpenGL Impeller path.
- FlutterWindowsEngine: parses --impeller-backend=vulkan next to the
  existing enable-impeller handling, skips EGL initialization entirely
  when Vulkan is active, and supplies a kVulkan renderer config whose
  image callbacks are never invoked because the compositor is
  supplied. The SDF rendering default is an ANGLE/OpenGL Impeller
  option and is not applied when the Vulkan backend is active.

Tests (hardware-dependent cases skip cleanly without Vulkan):

- VulkanManagerTest: loading, create-or-null, LUID validity, required
  extensions, no windowing extensions ever, proc resolution, teardown.
- VulkanImportedImageTest: argument validation.
- DCompPresenterTest: unknown LUID rejection, present-before-bind
  rejection, the full allocate/import/present/evict pipeline against a
  live DirectComposition tree, in-place swapchain resize, and repeated
  lifecycle cycles.
- FlutterWindowsEngineTest: the backend requires the explicit switch,
  is ignored without Impeller, either activates with ANGLE skipped or
  falls back without failing engine construction, and does not enable
  SDF rendering when Vulkan is active.
- GPUSurfaceVulkanImpeller: with render_to_surface disabled the root
  surface yields a no-op frame and never pulls an image from the
  delegate.
@sero583

sero583 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

oops

@sero583
sero583 force-pushed the impeller-vulkan-windows-dcomp branch from 7c847d7 to 1cd99c3 Compare July 17, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: desktop Running on desktop e: impeller Impeller rendering backend issues and features requests engine flutter/engine related. See also e: labels. platform-android Android applications specifically platform-windows Building on or for Windows specifically team-android Owned by Android platform team team-windows Owned by the Windows platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant