Skip to content

Commit 4db99a7

Browse files
authored
[Embedder] Support render texture for vulkan (flutter#188855)
<!-- Thanks for filing a pull request! Reviewers are typically assigned within a week of filing a request. To learn more about code review, see our documentation on Tree Hygiene: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md --> This PR adds external texture support for the Vulkan embedder API. Embedders using Flutter in Vulkan mode can now register a texture frame callback and have the engine directly sample embedder-owned `VkImage`s during composition. Both the Impeller and Skia Vulkan backends are supported, including RGBA/BGRA formats and multi-planar YUV formats (e.g. NV12) via YCbCr conversion. ## API changes (`embedder.h`) - Added `FlutterVulkanExternalTexture`, describing an embedder-owned `VkImage` handed to the engine: - `image`: handle to the `VkImage`, which must be in the `VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL` layout when provided to the engine. - `format`: the `VkFormat` of the image (for example `VK_FORMAT_R8G8B8A8_UNORM`). - Optional `width`/`height` (non-zero specifies the texture size). - Optional `destruction_callback` + `user_data`, invoked on an engine-managed thread when the texture can be collected. - Added `FlutterVulkanTextureFrameCallback`, invoked by the engine when a texture marked with a new frame available needs to be resolved. The embedder must perform a host sync before returning so the engine can sample the `VkImage` without additional synchronization. - Added `external_texture_frame_callback` to `FlutterVulkanRendererConfig`. ## Engine changes - Added `EmbedderExternalTextureVulkan` (`embedder_external_texture_vulkan.{h,cc}`), a `flutter::Texture` implementation that resolves embedder textures on both backends: - Impeller: `EmbedderExternalTextureSourceVulkan` implements `impeller::TextureSourceVK`, creating the image view and a `YUVConversionVK` when the format requires YCbCr conversion. - Skia: wraps the `VkImage` into a Skia image, dynamically selecting the `SkColorType` based on the `VkFormat`. - Extended `EmbedderExternalTextureResolver` to create Vulkan external textures. - Wired the new callback through `embedder.cc` and `embedder_surface_vulkan.cc`. ## Tests - Added unit tests in `embedder_vk_unittests.cc` covering RGBA, BGRA and NV12 textures on both Impeller and Skia, plus destruction-callback variants (`RenderTextureWith*Vulkan`, `Render*TextureWith*Vulkan`, `RenderTextureWith*VulkanDestructCallback`). - Added NV12 fixtures (`texture.nv12`, `external_texture_nv12.png`). - Extended `TestVulkanContext` and `EmbedderTestContextVulkan` to support external texture rendering and sampler YCbCr conversion. Addresses: flutter#117937 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent b56fbef commit 4db99a7

15 files changed

Lines changed: 1507 additions & 19 deletions

‎engine/src/flutter/shell/platform/embedder/BUILD.gn‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,8 @@ template("embedder_source_set") {
167167

168168
if (embedder_enable_vulkan) {
169169
sources += [
170+
"embedder_external_texture_vulkan.cc",
171+
"embedder_external_texture_vulkan.h",
170172
"embedder_surface_vulkan.cc",
171173
"embedder_surface_vulkan.h",
172174
]
@@ -176,6 +178,7 @@ template("embedder_source_set") {
176178
"embedder_surface_vulkan_impeller.cc",
177179
"embedder_surface_vulkan_impeller.h",
178180
]
181+
deps += [ "//flutter/impeller/display_list" ]
179182
}
180183

181184
deps += [
@@ -257,6 +260,8 @@ test_fixtures("fixtures") {
257260
"fixtures/gradient_metal.png",
258261
"fixtures/external_texture_impeller.png",
259262
"fixtures/external_texture_metal.png",
263+
"fixtures/external_texture_nv12.png",
264+
"fixtures/texture.nv12",
260265
"fixtures/gradient_xform.png",
261266
"fixtures/scene_without_custom_compositor.png",
262267
"fixtures/scene_without_custom_compositor_with_xform.png",

‎engine/src/flutter/shell/platform/embedder/embedder.cc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2385,6 +2385,29 @@ FlutterEngineResult FlutterEngineInitialize(size_t version,
23852385
external_texture_metal_callback);
23862386
}
23872387
}
2388+
#endif
2389+
#ifdef SHELL_ENABLE_VULKAN
2390+
if (config->type == kVulkan) {
2391+
const FlutterVulkanRendererConfig* vulkan_config = &config->vulkan;
2392+
if (auto callback = SAFE_ACCESS(vulkan_config,
2393+
external_texture_frame_callback, nullptr)) {
2394+
auto external_texture_vulkan_callback =
2395+
[frame_callback = callback, user_data](
2396+
int64_t texture_identifier, size_t width,
2397+
size_t height) -> std::unique_ptr<FlutterVulkanExternalTexture> {
2398+
std::unique_ptr<FlutterVulkanExternalTexture> texture =
2399+
std::make_unique<FlutterVulkanExternalTexture>();
2400+
texture->struct_size = sizeof(FlutterVulkanExternalTexture);
2401+
if (!frame_callback(user_data, texture_identifier, width, height,
2402+
texture.get())) {
2403+
return nullptr;
2404+
}
2405+
return texture;
2406+
};
2407+
external_texture_resolver = std::make_unique<ExternalTextureResolver>(
2408+
external_texture_vulkan_callback);
2409+
}
2410+
}
23882411
#endif
23892412
auto custom_task_runners = SAFE_ACCESS(args, custom_task_runners, nullptr);
23902413
auto thread_config_callback = [&custom_task_runners](

‎engine/src/flutter/shell/platform/embedder/embedder.h‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -968,6 +968,38 @@ typedef bool (*FlutterVulkanPresentCallback)(
968968
void* /* user data */,
969969
const FlutterVulkanImage* /* image */);
970970

971+
typedef struct {
972+
/// The size of this struct. Must be sizeof(FlutterVulkanExternalTexture).
973+
size_t struct_size;
974+
/// Handle to the VkImage that is owned by the embedder. The engine will
975+
/// sample from this image during composition. The VkImage must be in the
976+
/// VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL layout when provided to the
977+
/// engine.
978+
FlutterVulkanImageHandle image;
979+
/// The VkFormat of the image (for example: VK_FORMAT_R8G8B8A8_UNORM).
980+
uint32_t format;
981+
/// User data to be returned on the invocation of the destruction callback.
982+
void* user_data;
983+
/// Callback invoked (on an engine managed thread) that asks the embedder to
984+
/// collect the texture. This is optional and can be null.
985+
VoidCallback destruction_callback;
986+
/// Optional parameters for texture height/width, default is 0, non-zero means
987+
/// the texture has the specified width/height.
988+
/// Physical width of the texture.
989+
size_t width;
990+
/// Physical height of the texture.
991+
size_t height;
992+
} FlutterVulkanExternalTexture;
993+
994+
/// Callback to provide an external texture for a given texture_id.
995+
/// See: external_texture_frame_callback.
996+
typedef bool (*FlutterVulkanTextureFrameCallback)(
997+
void* /* user data */,
998+
int64_t /* texture identifier */,
999+
size_t /* width */,
1000+
size_t /* height */,
1001+
FlutterVulkanExternalTexture* /* texture out */);
1002+
9711003
typedef struct {
9721004
/// The size of this struct. Must be sizeof(FlutterVulkanRendererConfig).
9731005
size_t struct_size;
@@ -1031,7 +1063,13 @@ typedef struct {
10311063
/// without any additional synchronization.
10321064
/// Not used if a FlutterCompositor is supplied in FlutterProjectArgs.
10331065
FlutterVulkanPresentCallback present_image_callback;
1034-
1066+
/// When the embedder specifies that a texture has a frame available, the
1067+
/// engine will call this method (on an internal engine managed thread) so
1068+
/// that external texture details can be supplied to the engine for subsequent
1069+
/// composition. Prior to returning from this callback, the embedder must
1070+
/// perform a host sync, and so the engine can sample the VkImage without any
1071+
/// additional synchronization.
1072+
FlutterVulkanTextureFrameCallback external_texture_frame_callback;
10351073
} FlutterVulkanRendererConfig;
10361074

10371075
typedef struct {

‎engine/src/flutter/shell/platform/embedder/embedder_external_texture_resolver.cc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,12 @@ EmbedderExternalTextureResolver::EmbedderExternalTextureResolver(
2121
: metal_callback_(std::move(metal_callback)) {}
2222
#endif
2323

24+
#ifdef SHELL_ENABLE_VULKAN
25+
EmbedderExternalTextureResolver::EmbedderExternalTextureResolver(
26+
EmbedderExternalTextureVulkan::ExternalTextureCallback vulkan_callback)
27+
: vulkan_callback_(std::move(vulkan_callback)) {}
28+
#endif
29+
2430
std::unique_ptr<Texture>
2531
EmbedderExternalTextureResolver::ResolveExternalTexture(int64_t texture_id) {
2632
#ifdef SHELL_ENABLE_GL
@@ -37,6 +43,13 @@ EmbedderExternalTextureResolver::ResolveExternalTexture(int64_t texture_id) {
3743
}
3844
#endif
3945

46+
#ifdef SHELL_ENABLE_VULKAN
47+
if (vulkan_callback_) {
48+
return std::make_unique<EmbedderExternalTextureVulkan>(texture_id,
49+
vulkan_callback_);
50+
}
51+
#endif
52+
4053
return nullptr;
4154
}
4255

@@ -53,6 +66,12 @@ bool EmbedderExternalTextureResolver::SupportsExternalTextures() {
5366
}
5467
#endif
5568

69+
#ifdef SHELL_ENABLE_VULKAN
70+
if (vulkan_callback_) {
71+
return true;
72+
}
73+
#endif
74+
5675
return false;
5776
}
5877

‎engine/src/flutter/shell/platform/embedder/embedder_external_texture_resolver.h‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@
1717
#include "flutter/shell/platform/embedder/embedder_external_texture_metal.h"
1818
#endif
1919

20+
#ifdef SHELL_ENABLE_VULKAN
21+
#include "flutter/shell/platform/embedder/embedder_external_texture_vulkan.h"
22+
#endif
23+
2024
namespace flutter {
2125
class EmbedderExternalTextureResolver {
2226
public:
@@ -34,6 +38,11 @@ class EmbedderExternalTextureResolver {
3438
EmbedderExternalTextureMetal::ExternalTextureCallback metal_callback);
3539
#endif
3640

41+
#ifdef SHELL_ENABLE_VULKAN
42+
explicit EmbedderExternalTextureResolver(
43+
EmbedderExternalTextureVulkan::ExternalTextureCallback vulkan_callback);
44+
#endif
45+
3746
std::unique_ptr<Texture> ResolveExternalTexture(int64_t texture_id);
3847

3948
bool SupportsExternalTextures();
@@ -47,6 +56,9 @@ class EmbedderExternalTextureResolver {
4756
EmbedderExternalTextureMetal::ExternalTextureCallback metal_callback_;
4857
#endif
4958

59+
#ifdef SHELL_ENABLE_VULKAN
60+
EmbedderExternalTextureVulkan::ExternalTextureCallback vulkan_callback_;
61+
#endif
5062
FML_DISALLOW_COPY_AND_ASSIGN(EmbedderExternalTextureResolver);
5163
};
5264
} // namespace flutter

0 commit comments

Comments
 (0)