Flutter needs a way to handle memory constraints around image decoding.

@chinmaygarde

Internal: b/154531434

Today, it's very easy for a Dart developer to OOM their application by adding images. There are no guards or warnings in place to prevent this from happening, and we place the responsibility squarely on the developer to deal with it (with little-to-no guidance on how to avoid it, besides avoid decoding lots of images at once).

We've made some improvements: #49389 means a very common way of running into this (a list view of images that the user scrolls through rapidly) should not run into this as easily. Fixing #49456 should help some cases as well, such as when a user displays the same image multiple times and it didn't fit in the cache otherwise. #50022 is another step towards helping, since it would at least give more information about the size of dynamically obtained images (e.g. image via network).

Still, none of this addresses the core problem. Here is a strawman proposal; I think it has siginificant shortcomings, but I can't think of anything better right now either:

The engine could track total image memory usage and report it back to the application.
It could also report back total available memory.
The application could use this information to decide whether it should try to evict images from the cache (PROBLEMS: evict all? evict X LRU images? evict specific image?) or try to trim other caches, or perhaps start downsampling images (PROBLEM: new images? existing images? does the engine need new API to allow for this?), or try to just use smaller placeholders (PROBLEM: is this ever a reasonable thing to put on the user?).

/cc @chinmaygarde @alml @gaaclarke @liyuqian

I should also mention we've made improvements in image handling in the engine side as well: pushing decoding off to a worker pool to avoid getting the IO thread so locked up that it can't even release unused images, and being more aggressive about telling Skia to clean up unused image references. Both of these have helped a lot but still leave the fundamental problem.

The application could use [total memory] to decide whether it should try to evict images from the cache

Awhile ago I was looking at a related problem, OOMs when rendering animated images. Something that came up a lot then that still seems relevant now is that it's extremely difficult to have an objective line of when "too much" memory is being used. As I understand it neither iOS or Android have APIs that tell you if you're using a certain portion of available memory, or how much memory is available. They both basically just have an emergency callback when your app is about to go OOM completely that may or may not be called in all cases. Absolute limits that seem reasonable on one device with multiple GB of RAM will be way too high for devices that are more constrained, and vice versa. It's possible to set an absolute upper bound but it will most likely go stale over time, and apps built against that limit would need to be updated to work well on newer devices.

I don't think this really helps here with this problem, but in the end we switched to caching the bare minimum in memory we needed to function in all cases at the tradeoff of more work on the CPU spent decoding frames. We toyed with the idea of letting the developer set the limit but in the end this sort of thing varies on end-user device so it's still not really information the developer would be able to know in a lot of cases either.

I believe there's some way to get total memory on Android - not sure about iOS.

The engine could manually track how much memory it's allocating for images though. Each time we decode an image, we update a count of the memory we know it is/will use.

iOS:

[NSProcessInfo processInfo].physicalMemory`

Android:

ActivityManager actManager = (ActivityManager) context.getSystemService(Context.ACTIVITY_SERVICE);
ActivityManager.MemoryInfo memInfo = new ActivityManager.MemoryInfo();
actManager.getMemoryInfo(memInfo);
long totalMemory = memInfo.totalMem;

It might also be possible to respond to memory pressure events by downsampling our images... But I have a suspicion that might not be fast enough.

Yeah I stand corrected. It's not really recommended as a thing that you should continuously poll on Android but I think just calling it once would still be good enough for us. I thought it was a "development only" API but looking through the dev docs now I'm not seeing anything about that now. The preferred way to handle memory pressure on Android is to wait for low memory callbacks, but I do think that would be too slow like you're mentioning. Definitely we can track how much memory we allocate for images from within the engine itself like you're saying.

I'm not sure that information about total memory usage or available memory can help the application to make informed decisions. I propose to separate 2 concerns here:

Determining how much memory the images can consume (for us a fixed budget would be perfectly enough, we don't need dynamic discovery).
Actually making the images fit. The problem is that images downloaded from Internet can be of any resolution, while we may only need small 100x100 thumbnails on the screen. While the final state is good enough even today (only 100x100 images are kept around), the transient state during download and decompression is unbounded - image decompression requires allocating full-size buffers, and having multiple full-size buffers at the same time severely impacts the application memory footprint.

The original strawman proposal addresses (1), while I expect (2) to be the most important aspect.

I agree - didn't flutter-team-archive/engine#15372 address number 2 though?

@dnfield

@dnfield It works only for JPEGs. @ignatz and I did testing and found that PNG decoding for example couldn't bypass full-res image stage. Apparently this functionality is codec-dependent, and the systematic solution would be proper memory management in Flutter.

Dart now has flags for max old gen heap size. This is supported on iOS, but not Android yet (WIP here: flutter-team-archive/engine#20473). We'll want that on other platforms as well, and then developers could use some out of memory type errors to try to trim caches.

@dnfield

@dnfield I have worked on image memory for several months. there is my changed in engine;

cancel the image from io thread, when the image scroll out the screen or image widget is dispose. it works fine when user enter a view which loading image and exit very quickly. It can cancel the image pending in io thread.
make a decode image pool;
provide mipmaps optional to Dart. In most case, mipmaps should be closed, it can save 30% memory. If dev want this function, can open it via set mipmaps to true.
provide RGB565 on Android, it can save 50% memory, dev can set it in some case.

I found there is a memory cache in dart, I not sure if there is other caches in engine. I can see the memory leak when scroll, the memory is larger than image widget. For example, there are 3 image widget ,every image is 1024*1024, but the gfx memory is 32-36M. I have not found the issue.

This issue is marked P1 but has had no recent status updates.

The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.

Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!

Closing this as we've done everything here except for picking the smaller decode size, but that has its own issue tracking it.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Flutter needs a way to handle memory constraints around image decoding. #50024

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Flutter needs a way to handle memory constraints around image decoding. #50024

Description

Activity

dnfield commented on Feb 3, 2020

mklim commented on Feb 3, 2020

dnfield commented on Feb 3, 2020

dnfield commented on Feb 3, 2020

mklim commented on Feb 3, 2020

alml commented on Feb 4, 2020

dnfield commented on Feb 4, 2020

alml commented on Feb 5, 2020

dnfield commented on Oct 5, 2020

fredlee12345678 commented on Nov 5, 2020

flutter-triage-bot commented on Nov 29, 2023

jonahwilliams commented on Dec 4, 2023

github-actions commented on Dec 18, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions