Skip to content

[Impeller] SurfaceTextures created by some critical plugins do not render in OpenGL. #141636

Closed

Description

@matanlurey

Blocking #139702, blocked by #141890.


As of HEAD, using Impeller and the OpenGLES backend:

<!-- AndroidManifest.xml -->
<meta-data android:name="io.flutter.embedding.android.EnableImpeller" android:value="true" />
<meta-data android:name="io.flutter.embedding.android.ImpellerBackend" android:value="opengles" />

... plugins considered important (i.e. video_player, camera) do not render:

packages/packages/video_player/video_player/example
$ cd $PACKAGES/packages/video_player/video_player/example

$ flutter run --local-engine=$ENGINE/out/android_debug_unopt_arm64 --local-engine-host=$ENGINE/out/host_debug_unopt_arm64

image

Do note that my contrived sample app does render, but it uses the surface different than the plugins:

surface_texture

image

Activity

  1. added
    platform-androidAndroid applications specifically
    engineflutter/engine related. See also e: labels.
    P1High-priority issues at the top of the work list
    e: impellerImpeller rendering backend issues and features requests
    a: pluginsSupport for writing, building, and running plugin packages
    fyi-androidFor the attention of Android platform team
    on Jan 16, 2024
  2. moved this to 🤔 Needs Triage in Impelleron Jan 16, 2024
  3. matanlurey commented on Jan 16, 2024

    @matanlurey
    ContributorAuthor

    I checked an assumption that potentially frames aren't being pumped:

    diff --git a/shell/platform/android/surface_texture_external_texture_gl.cc b/shell/platform/android/surface_texture_external_texture_gl.cc
    index b1cadee7f30..87ee983f8f4 100644
    --- a/shell/platform/android/surface_texture_external_texture_gl.cc
    +++ b/shell/platform/android/surface_texture_external_texture_gl.cc
    @@ -70,7 +70,9 @@ SurfaceTextureExternalTextureImpellerGL::
             const fml::jni::ScopedJavaGlobalRef<jobject>& surface_texture,
             const std::shared_ptr<PlatformViewAndroidJNI>& jni_facade)
         : SurfaceTextureExternalTexture(id, surface_texture, jni_facade),
    -      impeller_context_(context) {}
    +      impeller_context_(context) {
    +  FML_LOG(ERROR) << "SurfaceTextureExternalTextureImpellerGL";
    +}
     
     SurfaceTextureExternalTextureImpellerGL::
         ~SurfaceTextureExternalTextureImpellerGL() {}
    @@ -105,6 +107,7 @@ void SurfaceTextureExternalTextureImpellerGL::ProcessFrame(
       Update();
     
       dl_image_ = impeller::DlImageImpeller::Make(texture_);
    +  FML_LOG(ERROR) << "SurfaceTextureExternalTextureImpellerGL::ProcessFrame";
     }
     
     void SurfaceTextureExternalTextureImpellerGL::Detach() {

    ... but it is:

    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
    E/flutter (21273): [ERROR:flutter/shell/platform/android/surface_texture_external_texture_gl.cc(110)] SurfaceTextureExternalTextureImpellerGL::ProcessFrame
  4. matanlurey commented on Jan 16, 2024

    @matanlurey
    ContributorAuthor

    Ok I appear to have narrowed it down to something, TBD if this is the problem or not:

    There is no representation of kTextureExternalOES within impeller::TextureGLES. That is, despite this:

    // shell/platform/android/surface_texture_external_texture_gl.cc
    void SurfaceTextureExternalTextureImpellerGL::ProcessFrame(
        PaintContext& context,
        const SkRect& bounds) {
      if (state_ == AttachmentState::kUninitialized) {
        // Generate the texture handle.
        impeller::TextureDescriptor desc;
        desc.type = impeller::TextureType::kTextureExternalOES;
        desc.storage_mode = impeller::StorageMode::kDevicePrivate;
        desc.format = impeller::PixelFormat::kR8G8B8A8UNormInt;
        desc.size = {static_cast<int>(bounds.width()),
                     static_cast<int>(bounds.height())};
        desc.mip_count = 1;
        texture_ = std::make_shared<impeller::TextureGLES>(
            impeller_context_->GetReactor(), desc,
            impeller::TextureGLES::IsWrapped::kWrapped);

    ... it's later interpreted as A8UNormInt (maybe OK, maybe not) and Texture2D (not OK). Will investigate further.

    (See flutter-team-archive/engine#44559)

  5. matanlurey commented on Jan 16, 2024

    @matanlurey
    ContributorAuthor

    Ok that above bit might be a false flag. I can confirm this code branch is never executed though:

        case TextureType::kTextureExternalOES:
          texture_type = GL_TEXTURE_EXTERNAL_OES;
          texture_target = GL_TEXTURE_EXTERNAL_OES;
          break;

    Found with this patch:

    diff --git a/impeller/renderer/backend/gles/texture_gles.cc b/impeller/renderer/backend/gles/texture_gles.cc
    index 8d6bdc2609f..f6773125b57 100644
    --- a/impeller/renderer/backend/gles/texture_gles.cc
    +++ b/impeller/renderer/backend/gles/texture_gles.cc
    @@ -74,6 +74,13 @@ TextureGLES::TextureGLES(std::shared_ptr<ReactorGLES> reactor,
         return;
       }
     
    +  // Debug check the texture format.
    +  if (desc.type == TextureType::kTextureExternalOES) {
    +    FML_LOG(ERROR) << "TextureGLES w/ kTextureExternalOES";
    +    FML_LOG(ERROR) << "... becomes:"
    +                   << TextureDescriptorToString(GetTextureDescriptor());
    +  }
    +
       is_valid_ = true;
     }
     
    @@ -168,6 +175,9 @@ bool TextureGLES::OnSetContents(const uint8_t* contents,
     // |Texture|
     bool TextureGLES::OnSetContents(std::shared_ptr<const fml::Mapping> mapping,
                                     size_t slice) {
    +  if (GetTextureDescriptor().type == TextureType::kTextureExternalOES) {
    +    FML_LOG(ERROR) << "OnSetContents w/ kTextureExternalOES";
    +  }
       if (!mapping) {
         return false;
       }
    @@ -221,6 +231,7 @@ bool TextureGLES::OnSetContents(std::shared_ptr<const fml::Mapping> mapping,
           texture_target = GL_TEXTURE_CUBE_MAP_POSITIVE_X + slice;
           break;
         case TextureType::kTextureExternalOES:
    +      FML_LOG(ERROR) << "kTextureExternalOES";
           texture_type = GL_TEXTURE_EXTERNAL_OES;
           texture_target = GL_TEXTURE_EXTERNAL_OES;
           break;
  6. johnmccutchan commented on Jan 17, 2024

    @johnmccutchan
    Contributor

    @bdero (who did a bunch of the OES Impeller plumbing work) could something have regressed in the OES code paths? I know we had to rewrite shaders and there were other complications in Impeller. AFAIK, this code is not tested today.

  7. 60 remaining items

  8. matanlurey commented on Mar 28, 2024

    @matanlurey
    ContributorAuthor

    OpenGL + Impeller is no longer a priority. Downgrading.

  9. added
    P3Issues that are less important to the Flutter project
    and removed
    P2Important issues not at the top of the work list
    on Mar 28, 2024
  10. changed the title [-]`SurfaceTexture`s created by some critical plugins do not render in Impeller+OpenGL[/-] [+][Impeller] `SurfaceTexture`s created by some critical plugins do not render in OpenGL.[/+] on Jul 3, 2024
  11. github-actions commented on Oct 8, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  12. locked as resolved and limited conversation to collaborators on Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Issues that are less important to the Flutter projecta: pluginsSupport for writing, building, and running plugin packagesblockedIssue is blocked by another issuee: impellerImpeller rendering backend issues and features requestse: openglengineflutter/engine related. See also e: labels.platform-androidAndroid applications specificallyteam-engineOwned by Engine teamtriaged-engineTriaged by Engine team

    Type

    No type

    Projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions