Repository navigation
Conversation
The PowerVR Rogue GM9446 GL driver on the MediaTek MT6779 has a fixed-size internal job pool and dereferences null when an allocation from that pool fails inside glClear, crashing the raster thread (SIGSEGV in libGLESv2_mtk.so, always preceded by "RM_GrowJobPool" / "Failed to allocate Job" driver logs). The pool is exhausted by many small GL submissions whose retirement falls behind during long low-load sessions (e.g. idling on a screen with an auto-playing carousel): each decoded image costs a tracking fence plus an implicit flush, and every command buffer submission is another kick. On affected devices, this change: - Reports tracking fences as unavailable so image uploads take the WaitUntilCompleted (glFinish) path and retire driver jobs immediately, instead of accumulating a fence per uploaded texture. - Periodically finishes the queue on the raster thread (every 64th submission) so pending jobs are drained before the pool fills. The gate only matches mt6779: other MT67xx SoCs either ship Mali GPUs (different driver) or have no confirmed reports of the job-pool failure signature. Verified on an OPPO CPH2203 (mt6779, Android 13 / API 33): without this change the device crashed within 11-106 minutes in every session, including with the flutter#189534 context-clearing workaround verifiably active; with it, the device ran from the previous afternoon through 9 AM the following morning (~16.5 hours) with zero crashes on system record and zero RM_GrowJobPool precursors during the supervised 2.5-hour soak under the exact reproduction conditions. The context-clearing workaround from flutter#187404 targets a different failure mode (idle-context eglMakeCurrent race) and does not prevent this crash. Fixes flutter#189190
github-actions
Bot
added
engine
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
walley892
left a comment
walley892
left a comment
There was a problem hiding this comment.
This seems like an okay bandaid for the MT6779 for the moment. Some comments
|
Hey @heyi1994, thanks for the PR! I've requested some changes/more experimentation on your end. Please re-request review when you're ready for another pass |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces mitigations for GL driver job-pool exhaustion crashes on specific MediaTek platforms (such as MT6779). It adds platform detection, disables tracking fences to force immediate completion of image uploads, and periodically drains the command queue. Feedback on the changes points out that using a global static atomic counter to track submissions across multiple GL contexts can lead to incorrect behavior, and suggests using a thread-local counter instead to ensure each context is drained independently.
| constexpr uint64_t kJobPoolDrainInterval = 64; | ||
| static std::atomic<uint64_t> submission_count = 0; | ||
| if (submission_count.fetch_add(1, std::memory_order_relaxed) % | ||
| kJobPoolDrainInterval == | ||
| kJobPoolDrainInterval - 1) { | ||
| reactor_->GetProcTable().Finish(); | ||
| } |
There was a problem hiding this comment.
Using a global static std::atomic<uint64_t> for submission_count introduces a correctness issue in multi-context environments (which Flutter uses, e.g., the main raster thread and the background IO/image-decoder thread).
The Problem
- Context Isolation:
glFinish()(called viareactor_->GetProcTable().Finish()) only drains/completes commands for the current GL context. It has no effect on other contexts. - Counter Sharing: Because
submission_countis static and shared globally, if the background thread increments the counter to a multiple of 64, it will triggerFinish()on the background context. This does not drain the pending jobs on the raster thread's context. - Workaround Bypass: As a result, the raster thread can go way past 64 submissions without ever calling
Finish(), potentially leading to the exact job-pool exhaustion and crash we are trying to prevent.
The Solution
Use thread_local instead of static std::atomic. This ensures that each thread (and thus each context/reactor) maintains its own independent submission counter, guaranteeing that Finish() is called every 64 submissions on that specific context. It also allows us to use a plain uint64_t instead of an atomic variable.
constexpr uint64_t kJobPoolDrainInterval = 64;
thread_local uint64_t submission_count = 0;
if (++submission_count % kJobPoolDrainInterval == 0) {
reactor_->GetProcTable().Finish();
}There was a problem hiding this comment.
Good catch — a drain only applies to the current context, so the shared counter could let IO-thread submissions starve the raster context's drain. Fixed with thread_local in the latest commit.
…of Finish. - The periodic-drain submission counter is now thread_local: a drain only applies to the current GL context, so a globally shared counter would let submissions on one context (e.g. the IO thread's) consume the drain interval of another and starve the raster context's drain. - The periodic drain now uses glFlush instead of glFinish, as suggested in review. Validated on the affected device (OPPO CPH2203, mt6779): 3.5 hours in the reproduction scenario with zero RM_GrowJobPool precursors and zero crashes, so the non-blocking flush is sufficient to keep the driver's job pool shallow and the blocking finish is not needed.
|
@heyi1994, it seems like you have a good handle on this issue and its causes on your devices. Would you be able to create a minimal reproduction flutter app? It will help us know whether this issue is chip/driver specific (in which case this PR seems okay to me) or whether the issue is systemic, affecting a wide variety of devices with job pool limitations (in which case we may need a broader fix). Once you have a minimal reproduction app, I can test on a few OpenGL devices |
|
@walley892 Here it is: https://github.com/heyi1994/grow_job_pool No interaction needed — run it in release and leave it in the foreground. 20 animated image tiles (10 GIF + 10 WebP), each in a rounded Please keep the tid when you testBoth threads allocate from the same pool, but only one is fatal — so
The IO thread submits far more often, so it almost always hits the ceiling first and nothing happens. More tiles raises the raster thread's odds of landing in the window; that's why the repro uses 20. adb logcat | grep -E "RM_GrowJobPool|Failed to allocate Job|Fatal signal|libGLESv2_mtk"Chip-specific vs. systemic
The middle row is the informative one and I can't produce it with one device, so whatever you find there settles it. NumbersOPPO CPH2203 (mt6779, PowerVR Rogue GM9446, Android 13),
Verified by Two caveats: these are single runs, and the crash is a race — time-to-crash swings widely (one session crashed at 4m11s, another survived 17 min). The precursor is the repeatable signal and the one I'd compare builds on. Also, GL memory in this repro sits at ~45 MB rather than the ~150–250 MB I quoted from the real app — different app, same conclusion: the pool exhausts nowhere near memory pressure. Knobs are |
|
@heyi1994, thanks for the super clear reproduction steps. I'm away from my test devices, but I'll be able to test this early next week. |
|
Greetings from stale PR triage! 👋 |
Fixes #189190
Problem
The PowerVR Rogue GM9446 GL driver on the MediaTek MT6779 has a fixed-size internal job pool and dereferences null when an allocation from that pool fails inside
glClear, crashing the raster thread (SIGSEGV inlibGLESv2_mtk.soatglClear+564, always preceded within ~0.5s byRM_GrowJobPool/Failed to allocate Jobdriver logs).The pool is exhausted by many small GL submissions whose retirement falls behind during long low-load sessions (e.g. idling on a screen with an auto-playing image carousel): each decoded image costs a tracking fence plus an implicit flush (
image_decoder_impeller.cc→ContextGLES::AddTrackingFence→WaitUntilScheduled), and every command buffer submission is another kick. GL memory is normal (~150–250 MB) when the pool exhausts — this is not memory pressure.This is a different failure mode from the idle-context
eglMakeCurrentrace that the #187404 context-clearing workaround targets: I tested #189534 (which broadens that workaround's gate to cover this device) with a locally built engine, verified vialibflutter.soBuildId in the crash dump that the workaround was active, and the device still crashed twice. Full data in my test report on #189534 and in the issue body.Fix
On affected devices (currently only
mt6779— other MT67xx SoCs either ship Mali GPUs or have no confirmed reports of this failure signature):ContextGLES::AddTrackingFencereports fences as unavailable, so image uploads take theWaitUntilCompleted(glFinish) path and retire driver jobs immediately instead of accumulating a fence per uploaded texture.CommandBufferGLES::OnSubmitCommandsfinishes the queue every 64th submission, draining pending jobs before the pool fills.A finer-grained approach (capping in-flight fences, batching flushes) could replace the
glFinish-based draining if reviewers prefer; I kept this initial version minimal and matching what was validated on hardware.Validation
On an OPPO CPH2203 (mt6779, Android 13 / API 33), same app, same day:
RM_GrowJobPoolprecursors during the supervised 2.5 h soakEvery crash across all builds had the identical signature and the identical driver-log precursor. The
RM_GrowJobPoollogcat line makes validation fast — it appears well before a crash would land, so I can validate any follow-up build on this device same-day without waiting for crashes.Pre-launch Checklist
///).