Repository navigation
[engine] Honor targetWidth/targetHeight for animated images - #194615
auto-submit[bot] merged 4 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements resizing for animated (multi-frame) images (such as GIFs, WebPs, and APNGs) to the requested target width and height. Frames are decoded and composited at their intrinsic size to maintain correct blending and disposal behavior, and only the final composited frame is resized to the target size. This change updates MultiFrameCodec and its internal State to accept a target size, introduces a ResizeFrame helper using bilinear filtering, and adapts both Skia and Impeller rendering paths to use the resized output. Comprehensive unit tests have been added to verify the resizing behavior and correctness against downscaled reference frames. I have no feedback to provide.
b-luk
left a comment
There was a problem hiding this comment.
The presubmit checks show a Linux customer_testing failure. That should be resolved if you merge from master.
| : generator_->GetPlayCount() - 1), | ||
| is_impeller_enabled_(UIDartState::Current()->IsImpellerEnabled()) {} | ||
| is_impeller_enabled_(UIDartState::Current()->IsImpellerEnabled()), | ||
| target_size_(target_size.isEmpty() ? generator_->GetInfo().dimensions() |
There was a problem hiding this comment.
When does target_size.isEmpty() happen? It's the default parameter value for the MultiFrameCodec constructor, but I don't think the MultiFrameCodec constructor is ever called with the default value.
When called from Dart, I think it always uses non-zero values for width and height.
Can we get rid of the constructor's default value, and get rid of this isEmpty fallback here?
There was a problem hiding this comment.
Only when a caller passes a size with a 0 dimension. The production caller, ImageDescriptor::instantiateCodec, always passes the size computed in Dart, which is positive except in an integer-division corner case with extreme aspect ratios (e.g. targetWidth: 50 for a 1000x10 image gives targetHeight 0), where the single-frame path can't produce an image either (ResizeRasterImage rejects empty dimensions). The default was only used by two tests (image_decoder_unittests.cc and shell_io_manager_unittests.cc). Removed the default and the fallback; the tests now pass the GIF's size.
| // See https://github.com/flutter/flutter/issues/90804. | ||
| SkBitmap output = bitmap; | ||
| if (output.dimensions() != target_size_) { | ||
| std::optional<SkBitmap> resized = ResizeFrame(bitmap, target_size_); |
There was a problem hiding this comment.
With the single-frame codec, resizing gets capped to max texture size:
Impeller here
flutter/engine/src/flutter/lib/ui/painting/image_decoder_impeller.cc
Lines 365 to 380 in 0590b2b
and Skia here
flutter/engine/src/flutter/lib/ui/painting/image_decoder_skia.cc
Lines 205 to 211 in 0590b2b
Should we do the same for the multi-frame case?
There was a problem hiding this comment.
Yes, done. On Skia, CrossContextTextureFromPixmap now gets limitToMaxTextureSize = true, as in UploadRasterImage. On Impeller, I moved the clamping out of DecompressTexture into ImageDecoderImpeller::ClampToMaxTextureSize() without changing its behavior (the existing DecompressTexture max-texture-size checks still cover it, plus a small direct test), and MultiFrameCodec clamps the target size with it before resizing.
| // `cacheWidth`/`cacheHeight` or `ResizeImage` in the framework). | ||
| // | ||
| // See https://github.com/flutter/flutter/issues/90804. | ||
| SkBitmap output = bitmap; |
There was a problem hiding this comment.
Could we reuse/reassign the existing bitmap and info variables instead of creating new output and output_info variables?
There was a problem hiding this comment.
Done: bitmap and info are reassigned after resizing. lastRequiredFrame_ keeps its reference to the full-size pixels, and the upload code is back to what's on master apart from limitToMaxTextureSize.
|
Thanks for the review! Rebased onto the latest master, which picks up the customer_testing fix from #194968, and addressed the comments. |
This PR is still based on the same original commit. Did you push to github after rebasing? |
ImageDescriptor::instantiateCodec only forwarded the target size to SingleFrameCodec, so animated GIF, WebP and APNG frames were always decoded and uploaded at their intrinsic size, and cacheWidth, cacheHeight and ResizeImage had no effect on them. MultiFrameCodec now receives the target size. Frames are still composited at the intrinsic size, because frame rects, blending and disposal are defined in canvas coordinates and lastRequiredFrame_ is reused as a backdrop. Only the finished frame is resized, with the same sampling as single-frame images, before it is uploaded.
…re size, reuse bitmap/info
f757c0a to
1d3549a
Compare
|
Sorry @b-luk , my mistake the review changes were pushed, but the rebase hadn't gone through. It's now rebased onto the latest master. |
| /// ratio, so that it fits within |max_texture_size|. Each side stays | ||
| /// at least one pixel. A size that already fits is returned as is. | ||
| static SkISize ClampToMaxTextureSize(const SkISize& target_size, | ||
| const impeller::ISize& max_texture_size); |
There was a problem hiding this comment.
I'd just copy the ISize instead of passing it by ref. Since it small it should be faster. I don't think the linter will complain.
There was a problem hiding this comment.
Done, SkISize and impeller::ISize are now passed by value
| // image_decoder_skia.cc and `ResizeOnCpu` in image_decoder_impeller.cc), so | ||
| // `targetWidth` and `targetHeight` behave the same way for every image. | ||
| static std::optional<SkBitmap> ResizeFrame(const SkBitmap& frame, | ||
| const SkISize& target_size) { |
| std::ostringstream ostr; | ||
| ostr << "Could not resize frame " << nextFrameIndex_ << " to " | ||
| << target_size.width() << "x" << target_size.height(); | ||
| std::string decode_error = ostr.str(); |
There was a problem hiding this comment.
Done for target_size. I kept frame as a const reference since copying an SkBitmap bumps its pixel ref count
| resourceContext.get(), // context | ||
| pixmap, // pixmap | ||
| true, // buildMips | ||
| true // limitToMaxTextureSize |
There was a problem hiding this comment.
please use proper argument comments: /*buildMips=*/true (the names will be validated by the linter)
| codec.dispose(); | ||
| }); | ||
|
|
||
| test('Resized animated frames match the downscaled full-size frames', () async { |
There was a problem hiding this comment.
This is overkill, let's remove it.
| /// @brief Creates a codec for an animated (multi-frame) image. | ||
| /// | ||
| /// @param[in] generator The decoder for the encoded image. | ||
| /// @param[in] target_size The size at which frames are delivered to Dart, |
There was a problem hiding this comment.
mention the filtering that is happening when resizing
|
@b-luk can u add CICD label again ? |
|
autosubmit label was removed for flutter/flutter/194615, because - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label. |
|
|
|
The Google testing failure is likely due to the internal Google tests using an engine build based on an older commit compared to your PR. A roll is in progress to update the internal Google engine version. Once that completes, I'll re-run the tests for your PR and hopefully the problem will be resolved. |
|
Oops, sorry @b-luk I just bypassed the failures since they weren't apropos. I don't think this could affect google anyways since their testing infra is on skia. |
Seems fine to me. I suggested waiting for google's next engine roll and then rerunning just in case, because I didn't exhaustively inspect all the failures and there isn't an urgency to get this PR in immediately. But I think it's highly unlikely that this PR breaks google tests, so bypassing is fine with me. |
|
Well done guys |
flutter/flutter@d4c8577...0c2473c 2026-10-28 [email protected] Roll Packages from a8549ae to 13d38e0 (9 revisions) (flutter/flutter#195488) 2026-10-28 [email protected] Use int.oneBitCount in PersistentHashMap and scope 53-bit caps to JS (flutter/flutter#195005) 2026-10-28 [email protected] Roll pub packages (flutter/flutter#195483) 2026-10-28 [email protected] [iPhone Duo] Use the correct UIScreen to report the display size and update when fold/unfold (flutter/flutter#195199) 2026-10-28 [email protected] [fuchsia] Fix portable_ui_test.h includes for newer Fuchsia SDKs (flutter/flutter#195420) 2026-10-28 [email protected] Change default pipeline options to match production draw calls (flutter/flutter#194597) 2026-10-28 [email protected] Roll Fuchsia Test Scripts from 2T18KC7-LDQtmQiLP... to xl1LWBuppzu-BnOzV... (flutter/flutter#195450) 2026-10-27 [email protected] [flutter_tools] Inject dependencies into Darwin build targets, native assets, and Xcode build settings (flutter/flutter#194052) 2026-10-27 [email protected] Revert: Reland square corners fix (flutter/flutter#195436) 2026-10-27 [email protected] [flutter_tools] Add DeviceService app launch and flutter run support for tool extensions (flutter/flutter#194589) 2026-10-27 [email protected] [engine] Honor targetWidth/targetHeight for animated images (flutter/flutter#194615) 2026-10-27 [email protected] [Android][a11y] Set visibleToUser=false for hidden semantics nodes (flutter/flutter#194411) 2026-10-27 [email protected] Fix repo deletion test flake (flutter/flutter#195349) 2026-10-27 [email protected] Roll Packages from 23a05b6 to a8549ae (10 revisions) (flutter/flutter#195434) 2026-10-27 [email protected] [web] Capture pointers to the Flutter view so drags survive platform view iframes (flutter/flutter#195187) 2026-10-27 [email protected] [web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder (flutter/flutter#195118) 2026-10-27 [email protected] Link migration guide in deprecation messages (flutter/flutter#195361) 2026-10-27 [email protected] Pin basic_material_app_macos__compile to macOS 26 (flutter/flutter#195429) 2026-10-27 [email protected] Add benchmark for warmed pipeline usage (flutter/flutter#195001) 2026-10-27 [email protected] [flutter_tools] Add device log streaming support to DeviceService and ExtensionBackedDevice (flutter/flutter#194964) 2026-10-27 [email protected] Roll Dart SDK from 1d16c04dc346 to 65fc979759ca (4 revisions) (flutter/flutter#195383) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
…194615) ## Description `ui.instantiateImageCodec`, `ui.instantiateImageCodecWithSize` and `ImageDescriptor.instantiateCodec` ignore `targetWidth`/`targetHeight` for images with more than one frame. `ImageDescriptor::instantiateCodec` only forwards the target size to `SingleFrameCodec`; `MultiFrameCodec` always decodes, composites and uploads every frame at the intrinsic size of the image. As a result, `cacheWidth`/`cacheHeight` and `ResizeImage` have no effect on animated GIF, WebP and APNG images, which is a common source of high memory usage (for example, grids of animated stickers). This PR forwards the target size to `MultiFrameCodec`: - Frames are still decoded and composited at the intrinsic size. Frame rects, blending and disposal are defined in the coordinate space of the full canvas, and `lastRequiredFrame_` is reused as the backdrop of later frames, so this part is unchanged. - Only the finished frame is resized before it is uploaded, using the same sampling as single-frame images (`SkFilterMode::kLinear`, no mipmaps, as in `ResizeRasterImage` and `ResizeOnCpu`). - All three upload paths (Skia, Impeller on iOS, Impeller elsewhere) upload the resized frame, so the resulting `ui.Image` has the requested size. No Dart API changes are needed: the framework already passes the target size. Like single-frame images, frames are never uploaded larger than the maximum texture size: the Skia path passes limitToMaxTextureSize, and the Impeller path clamps the target size with ImageDecoderImpeller::ClampToMaxTextureSize, which this PR factors out of DecompressTexture. Memory example: a 1000×1000 animated GIF displayed with `cacheWidth: 200` now uploads frames of about 160 KB instead of about 4 MB each (plus mipmaps on the Skia backend). The CPU-side compositing buffers (one frame and the cached backdrop) keep the intrinsic size, exactly as today. ## Alternatives considered Both alternatives were evaluated with an emulation of `MultiFrameCodec::State::GetNextFrameImage` on top of Skia's real GIF (Wuffs) and WebP codecs (skia-python, Skia m144), over 35 files (17 Skia/engine fixtures, 6 synthetic GIFs covering every disposal method, 12 adversarial GIFs), 9 scales and 2 loops each (14,742 frames). 1. **Decode frames directly at a reduced size** with `ImageGenerator::GetScaledDimensions()`, as `SingleFrameCodec` does, keeping the single cached backdrop. Skia's GIF and WebP codecs support arbitrary downscaling (`SkScalingCodec`), but with a single backdrop the output diverges from Skia's own reduced-size decode whenever a frame's required frame is older than the cached frame and a `kRestoreBGColor` clear was applied at the reduced size: the cleared rect is rounded out, while frame draws cover pixels by their centers. 812 of 14,742 frames differed, including 6 frames of Skia's `randPixelsAnim.gif` (1px transparent slivers). 2. **Same as 1, with a dependency-aware cache of backdrops** (derived from `FrameInfo::required_frame`). This matched Skia's reference for all frames, but `heart.webp` (an existing engine fixture) needs 15 backdrops cached at the same time, and the bookkeeping is considerably more complex. This could be a follow-up optimization where reduced-size decoding saves significant CPU memory (WebP). The chosen approach is exact by construction: every delivered frame is today's full-size frame, resized. In the same emulation, the current full-size algorithm matched Skia's reference decode for all 1,638 frames, and the half-size output equalled the rounded 2×2 average of the full-size frames (maximum channel difference 0) on every even-sized fixture. ## Tests Added to `testing/dart/codec_test.dart`: - `Animated gif frames are resized to targetWidth and targetHeight`: `test640x479.gif` decoded at 160×120 for two loops. - `Animated webp frames are resized when only targetWidth is given`: `heart.webp` with `targetWidth: 100` yields 100×100 frames. - `instantiateImageCodecWithSize resizes animated images`: the reproduction from flutter#137668. ``` ./testing/run_tests.py --type=dart --dart-filter=codec_test.dart --variant <host_debug_unopt variant> ``` ## Out of scope - Web (CanvasKit/Skwasm) still ignores `targetWidth`/`targetHeight` for multi-frame images (see flutter-team-archive/engine#38028). - `TargetPixelFormat` is still not forwarded to `MultiFrameCodec`. ## Related issues Fixes flutter#90804 Fixes flutter#137668 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md ## Testing note I could not build the engine locally (not enough disk space), so I have not run the engine tests myself and am relying on presubmit CI. The C++ changes are formatted with clang-format, and codec_test.dart passes dart format and dart analyze.
Description
ui.instantiateImageCodec,ui.instantiateImageCodecWithSizeandImageDescriptor.instantiateCodecignoretargetWidth/targetHeightfor imageswith more than one frame.
ImageDescriptor::instantiateCodeconly forwards thetarget size to
SingleFrameCodec;MultiFrameCodecalways decodes, compositesand uploads every frame at the intrinsic size of the image. As a result,
cacheWidth/cacheHeightandResizeImagehave no effect on animated GIF,WebP and APNG images, which is a common source of high memory usage (for
example, grids of animated stickers).
This PR forwards the target size to
MultiFrameCodec:blending and disposal are defined in the coordinate space of the full canvas,
and
lastRequiredFrame_is reused as the backdrop of later frames, so thispart is unchanged.
sampling as single-frame images (
SkFilterMode::kLinear, no mipmaps, as inResizeRasterImageandResizeOnCpu).resized frame, so the resulting
ui.Imagehas the requested size.No Dart API changes are needed: the framework already passes the target size.
Like single-frame images, frames are never uploaded larger than the maximum texture size: the Skia path passes limitToMaxTextureSize, and the Impeller path clamps the target size with ImageDecoderImpeller::ClampToMaxTextureSize, which this PR factors out of DecompressTexture.
Memory example: a 1000×1000 animated GIF displayed with
cacheWidth: 200nowuploads frames of about 160 KB instead of about 4 MB each (plus mipmaps on the
Skia backend). The CPU-side compositing buffers (one frame and the cached
backdrop) keep the intrinsic size, exactly as today.
Alternatives considered
Both alternatives were evaluated with an emulation of
MultiFrameCodec::State::GetNextFrameImageon top of Skia's real GIF (Wuffs)and WebP codecs (skia-python, Skia m144), over 35 files (17 Skia/engine
fixtures, 6 synthetic GIFs covering every disposal method, 12 adversarial GIFs),
9 scales and 2 loops each (14,742 frames).
ImageGenerator::GetScaledDimensions(), asSingleFrameCodecdoes, keepingthe single cached backdrop. Skia's GIF and WebP codecs support arbitrary
downscaling (
SkScalingCodec), but with a single backdrop the outputdiverges from Skia's own reduced-size decode whenever a frame's required
frame is older than the cached frame and a
kRestoreBGColorclear wasapplied at the reduced size: the cleared rect is rounded out, while frame
draws cover pixels by their centers. 812 of 14,742 frames differed,
including 6 frames of Skia's
randPixelsAnim.gif(1px transparent slivers).FrameInfo::required_frame). This matched Skia's reference for all frames,but
heart.webp(an existing engine fixture) needs 15 backdrops cached atthe same time, and the bookkeeping is considerably more complex. This could
be a follow-up optimization where reduced-size decoding saves significant
CPU memory (WebP).
The chosen approach is exact by construction: every delivered frame is today's
full-size frame, resized. In the same emulation, the current full-size
algorithm matched Skia's reference decode for all 1,638 frames, and the
half-size output equalled the rounded 2×2 average of the full-size frames
(maximum channel difference 0) on every even-sized fixture.
Tests
Added to
testing/dart/codec_test.dart:Animated gif frames are resized to targetWidth and targetHeight:test640x479.gifdecoded at 160×120 for two loops.Animated webp frames are resized when only targetWidth is given:heart.webpwithtargetWidth: 100yields 100×100 frames.instantiateImageCodecWithSize resizes animated images: the reproductionfrom Resizing parameters are not supported for multi-frame images in the engine. #137668.
Out of scope
targetWidth/targetHeightformulti-frame images (see implement targetWidth and targetHeight flutter-team-archive/engine#38028).
TargetPixelFormatis still not forwarded toMultiFrameCodec.Related issues
Fixes #90804
Fixes #137668
Pre-launch Checklist
///).[Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Testing note
I could not build the engine locally (not enough disk
space), so I have not run the engine tests myself and am relying on
presubmit CI. The C++ changes are formatted with clang-format, and
codec_test.dart passes dart format and dart analyze.