Repository navigation
Fix/shared thread merger expired lease - #194200
auto-submit[bot] merged 9 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request updates RasterThreadMerger and SharedThreadMerger to remove a caller's lease from the active leases map once it expires, ensuring that subsequent draws by this caller do not incorrectly affect other active leases. A new unit test is added to verify this behavior. The review feedback suggests downgrading a warning log to a verbose log to prevent log spam and potential rendering jank from expired callers, and recommends simplifying the decrement logic using the pre-decrement operator.
gaaclarke
left a comment
There was a problem hiding this comment.
Please update the PR's description
@gaaclarke Description updated. Sorry for missing that. I run All Tests again and all PASSED. |
This PR addresses the lease_term_ref > 0 assertion reported in #153211, rather than a separate Android-only issue.
|
|
Thanks. It's helpful if you mark |
gaaclarke
left a comment
There was a problem hiding this comment.
Okay, code and fix sounds good. I just want the description to be updated to mention what issues it fixes. If none of them are applicable you should file another issue.
|
@flar can you be a second reviewer on this please? |
|
@gaaclarke @flar Thanks for the reviews and approvals! Could you please help with the remaining CI blockers?
Thanks for your help! |
|
autosubmit label was removed for flutter/flutter/194200, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label. |
|
autosubmit label was removed for flutter/flutter/194200, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label. |
|
@gaaclarke @flar The tree is green now, and all checks on this PR have passed. Could you please re-add the |
|
autosubmit label was removed for flutter/flutter/194200, because The base commit of the PR is older than 7 days and can not be merged. Please merge the latest changes from the main into this branch and resubmit the PR. |
|
autosubmit label was removed for flutter/flutter/194200, because - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label. |
|
@gaaclarke @flar Thanks for updating the branch and trying autosubmit again! The latest failed Google testing run on Could you please:
I don't have repository write permission to trigger these actions myself. If another maintainer or the infra team should handle this, please point me to them. Thanks! |
|
@gaaclarke @flar One additional question about the submission process: As an external contributor without repository write permission, I cannot re-add the labels or approve the workflows myself, so each removal leaves me dependent on another maintainer action. Is there a recommended way to resolve this cycle and get the PR through the remaining checks and merged, without repeatedly asking reviewers to reapply the labels? Thanks for your guidance! |
Unfortunately this is the way now. It looks like the bot was removing the CICD label just as @gaaclarke was adding the auto-submit so the two events crossed in the ether and nobody noticed. It now has CICD + autosubmit so hopefully it will go through this time. |
|
autosubmit label was removed for flutter/flutter/194200, because - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label. |
flutter/flutter@c091098...5e805bf 2026-10-30 [email protected] Fix/shared thread merger expired lease (flutter/flutter#194200) 2026-10-30 [email protected] [FGP] Report the real error when applying kotlin-android fails (flutter/flutter#195080) 2026-10-30 [email protected] [Impeller] Minify GLSL shaders (flutter/flutter#195240) 2026-10-30 [email protected] Roll Packages from 0b72e92 to 3742a4e (12 revisions) (flutter/flutter#195686) 2026-10-30 [email protected] Roll Dart SDK from 52c908c00267 to fe609902984b (2 revisions) (flutter/flutter#195679) 2026-10-30 [email protected] Roll Skia from 0789f2ab93b8 to c0c46ee9f938 (1 revision) (flutter/flutter#195671) 2026-10-30 [email protected] Roll Fuchsia Linux SDK from SORPfa9tgFn83M1CR... to x2SqGbVUkOm4d6vk_... (flutter/flutter#195670) 2026-10-30 [email protected] Roll Skia from 4470b8abaad6 to 0789f2ab93b8 (2 revisions) (flutter/flutter#195665) 2026-10-30 [email protected] Roll Dart SDK from 89f7a35480d6 to 52c908c00267 (1 revision) (flutter/flutter#195659) 2026-10-30 [email protected] Roll Skia from 9f4e3117465e to 4470b8abaad6 (4 revisions) (flutter/flutter#195658) 2026-10-30 [email protected] Roll Skia from ec13cd547e98 to 9f4e3117465e (1 revision) (flutter/flutter#195650) 2026-10-30 [email protected] [web] Keep semantics editing state while the text input connection is open (flutter/flutter#195141) 2026-10-30 [email protected] [web_paragraph] Fix multi-span BiDi span order and region direction (flutter/flutter#194980) 2026-10-30 [email protected] [web] Unskip web tests that pass on CanvasKit and Skwasm: accessibility_inspector_test, color_filter_test (flutter/flutter#195541) 2026-10-30 [email protected] Roll Skia from 4ecd22a4a1d0 to ec13cd547e98 (10 revisions) (flutter/flutter#195647) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Fixes flutter#194536. ## Summary Remove expired caller lease records from `SharedThreadMerger` to prevent an engine from aborting when it resumes drawing while another engine keeps the shared raster/platform queues merged. Related to flutter#153211. This PR addresses the `lease_term_ref > 0` assertion reported there, including the corresponding Android scenario. It does not claim to resolve the separate deadlock or backgrounding symptoms discussed in that issue. ## Problem Multiple `RasterThreadMerger` instances can share one `SharedThreadMerger`. The merged state is shared, but each caller has its own lease. The crash occurs in this sequence: 1. Engine A merges with a lease of 1 and consumes it. The queues unmerge, but A's zero-valued entry remains in `lease_term_by_caller_`. 2. Engine B merges the shared queues with an active lease. 3. Engine A resumes drawing. The shared merger reports merged because B's lease is active. 4. A calls `DecrementLease()`, finds its stale zero-valued entry, and aborts on `FML_CHECK(lease_term_ref > 0)`. We reproduced this defect on Android. flutter#153211 reports the same assertion on iOS and includes an [Android report](flutter#153211 (comment)) involving engines in the same `FlutterEngineGroup`. The affected implementation is shared C++ code. However, the iOS application reproduction from flutter#153211 has not been tested with this patch. ## Change - Remove the caller's map entry immediately when its lease reaches zero, under the existing mutex. - Preserve the positive-lease assertion for existing entries. - Allow subsequent draws from an expired caller to use the missing-caller path without consuming another caller's lease. - Use `FML_VLOG(1)` for the missing-caller path, since it can occur during normal rendering. - Combine decrement and expiration checking with `--lease_term_ref`. - Clarify comments describing caller lease lifetime and the shared merge state. The existing all-zero check handles an empty map, so the queues still unmerge when the final active lease expires. This is an internal Engine bug fix with no public API change or migration requirement. ## Production workaround In our Android return-to-bottom scenario, explicitly destroying the outgoing/top Flutter engine in the Activity's `finish()` method, instead of waiting for `onDestroy()`, avoids this crash. We have validated this application-level workaround in production. This changes teardown timing for the transition where the bottom Activity can resume drawing before the top Activity reaches `onDestroy()`. It is a workaround for the affected lifecycle sequence; the stale zero-lease record still needs to be fixed in the Engine. Production validation here applies to the workaround. Patched-Engine Android device results are documented below; iOS application validation remains pending. ## Tests Adds `RasterThreadMerger.ExpiredLeaseCanDecrementWhileAnotherCallerIsMerged`, covering: - Resuming drawing after a caller's lease expires while another caller keeps the queues merged. - Repeated decrements from the expired caller without consuming the other caller's lease. - Unmerging when the remaining lease expires. - Reacquiring and consuming a lease after unmerging. Validated locally on macOS arm64 with `host_debug_unopt_arm64`: - Built `//flutter/fml:fml_unittests`. - New regression test: **1 passed**. - All `RasterThreadMerger` tests: **17 passed**. - Full `fml_unittests`: **204 tests across 38 test suites passed**. - Ran clang-format on changed lines and `git diff --check`. A standalone reproduction using the real FML shared merger and task queue implementations triggers the original assertion before the fix and passes afterward. Patched-Engine Android device results are documented below. iOS application validation, upstream CI and maintainer review remain pending. ### Android device reproduction (unpatched Engine) Confirmed the target SIGABRT on an Android test device running Android 17 with Flutter 3.35.3, Engine `ddf47dd3ff`, debug mode, and Impeller Vulkan. The demo and app-specific crash evidence are attached to flutter#194536: - [lease-zero-demo-sanitized.zip](https://github.com/user-attachments/files/33544786/lease-zero-demo-sanitized.zip) - [zero-lease-crash-sanitized.zip](https://github.com/user-attachments/files/33544788/zero-lease-crash-sanitized.zip) The demo adds a hybrid-composition PlatformView to the bottom engine, removes it and requests 40 frames, then opens a top Activity with a PlatformView in another engine from the same FlutterEngineGroup. Two initial automated cycles did not crash; a subsequent manual cycle in the same process did. The observed abort occurred while opening top again, after top onPostResume and before the return operation. Reproduction is timing-dependent; no success rate is claimed. The captured abort is `shared_thread_merger.cc(59): lease_term_ref > 0`, with `lease_term=0`. This confirms the application-level failure on the unpatched Engine. ### Android device validation (patched Engine, 2026-10-16) Validated the locally built patch on **an Android test device running Android 17 (API 37)** in debug mode. Framework and Engine source both used this PR's head, `bf3e0b302cdb437b932aecc07709f266eb545597`, with `android_debug_unopt_arm64` and matching `host_debug_unopt_arm64` artifacts. - Built with `--no-enable-hcpp` to exercise legacy Hybrid Composition; the APK manifest explicitly sets `EnableHcpp=false`. The `finish()` early-destroy workaround was **not enabled**. - Completed **30/30 ADD -> REMOVE -> OPEN -> return cycles** across three independent app processes, 10 cycles each: 10 with the verbose Engine flag and 20 with normal logging. No zero-lease assertion, native fatal signal, fatal Java exception, Dart exception, or process restart was observed. - Logs record **60 legacy HC PlatformView creations** and **30 returns** in which bottom `onResume()` requested Dart redraw while the top engine was still alive, before top `onDestroy()`. - Verified that the installed APK SHA-256 matches the built APK, and the APK's `libflutter.so` Build ID matches the local Engine: `351451851bb71400213aa0c9764406986940c8c4`. - Reran the rebuilt host `RasterThreadMerger.*` suite: **17/17 passed**, including the new regression. Validation report, event sequences, per-cycle results, and binary identity checks: [engine-validation-sanitized.zip](https://github.com/user-attachments/files/33544787/engine-validation-sanitized.zip) The logs also contain 19 nonfatal `Could not make the resource context current.` messages; the cycles and an additional real-touch responsiveness check still completed. These are recorded in the attached report. The original unpatched device crash used Flutter 3.35.3 / Engine `ddf47dd3ff`. A device A/B comparison at the same framework revision with only this patch removed has not been run. The demo is timing-dependent, so these results provide bounded Android regression evidence. iOS application validation remains pending. The verbose Engine flag does not enable `FML_VLOG(1)`, so these device logs do not demonstrate direct coverage of the expired-caller branch; the deterministic C++ regression covers that behavior. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. The full local FML suite passes; the final test checkbox remains unchecked pending upstream CI. [Contributor Guide]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/master/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md [Flutter Style Guide]: https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/website]: https://github.com/flutter/website [test-exempt]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#tests [breaking change policy]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Data Driven Fixes]: https://github.com/flutter/flutter/blob/master/docs/contributing/Data-driven-Fixes.md --------- Co-authored-by: gaaclarke <[email protected]>
Fixes #194536.
Summary
Remove expired caller lease records from
SharedThreadMergerto prevent an engine from aborting when it resumes drawing while another engine keeps the shared raster/platform queues merged.Related to #153211. This PR addresses the
lease_term_ref > 0assertion reported there, including the corresponding Android scenario. It does not claim to resolve the separate deadlock or backgrounding symptoms discussed in that issue.Problem
Multiple
RasterThreadMergerinstances can share oneSharedThreadMerger. The merged state is shared, but each caller has its own lease.The crash occurs in this sequence:
lease_term_by_caller_.DecrementLease(), finds its stale zero-valued entry, and aborts onFML_CHECK(lease_term_ref > 0).We reproduced this defect on Android. #153211 reports the same assertion on iOS and includes an Android report involving engines in the same
FlutterEngineGroup.The affected implementation is shared C++ code. However, the iOS application reproduction from #153211 has not been tested with this patch.
Change
FML_VLOG(1)for the missing-caller path, since it can occur during normal rendering.--lease_term_ref.The existing all-zero check handles an empty map, so the queues still unmerge when the final active lease expires.
This is an internal Engine bug fix with no public API change or migration requirement.
Production workaround
In our Android return-to-bottom scenario, explicitly destroying the outgoing/top Flutter engine in the Activity's
finish()method, instead of waiting foronDestroy(), avoids this crash. We have validated this application-level workaround in production.This changes teardown timing for the transition where the bottom Activity can resume drawing before the top Activity reaches
onDestroy(). It is a workaround for the affected lifecycle sequence; the stale zero-lease record still needs to be fixed in the Engine. Production validation here applies to the workaround. Patched-Engine Android device results are documented below; iOS application validation remains pending.Tests
Adds
RasterThreadMerger.ExpiredLeaseCanDecrementWhileAnotherCallerIsMerged, covering:Validated locally on macOS arm64 with
host_debug_unopt_arm64://flutter/fml:fml_unittests.RasterThreadMergertests: 17 passed.fml_unittests: 204 tests across 38 test suites passed.git diff --check.A standalone reproduction using the real FML shared merger and task queue implementations triggers the original assertion before the fix and passes afterward.
Patched-Engine Android device results are documented below. iOS application validation, upstream CI and maintainer review remain pending.
Android device reproduction (unpatched Engine)
Confirmed the target SIGABRT on an Android test device running Android 17 with Flutter 3.35.3, Engine
ddf47dd3ff, debug mode, and Impeller Vulkan. The demo and app-specific crash evidence are attached to #194536:The demo adds a hybrid-composition PlatformView to the bottom engine, removes it and requests 40 frames, then opens a top Activity with a PlatformView in another engine from the same FlutterEngineGroup. Two initial automated cycles did not crash; a subsequent manual cycle in the same process did. The observed abort occurred while opening top again, after top onPostResume and before the return operation. Reproduction is timing-dependent; no success rate is claimed.
The captured abort is
shared_thread_merger.cc(59): lease_term_ref > 0, withlease_term=0. This confirms the application-level failure on the unpatched Engine.Android device validation (patched Engine, 2026-10-16)
Validated the locally built patch on an Android test device running Android 17 (API 37) in debug mode. Framework and Engine source both used this PR's head,
bf3e0b302cdb437b932aecc07709f266eb545597, withandroid_debug_unopt_arm64and matchinghost_debug_unopt_arm64artifacts.--no-enable-hcppto exercise legacy Hybrid Composition; the APK manifest explicitly setsEnableHcpp=false. Thefinish()early-destroy workaround was not enabled.onResume()requested Dart redraw while the top engine was still alive, before toponDestroy().libflutter.soBuild ID matches the local Engine:351451851bb71400213aa0c9764406986940c8c4.RasterThreadMerger.*suite: 17/17 passed, including the new regression.Validation report, event sequences, per-cycle results, and binary identity checks: engine-validation-sanitized.zip
The logs also contain 19 nonfatal
Could not make the resource context current.messages; the cycles and an additional real-touch responsiveness check still completed. These are recorded in the attached report.The original unpatched device crash used Flutter 3.35.3 / Engine
ddf47dd3ff. A device A/B comparison at the same framework revision with only this patch removed has not been run. The demo is timing-dependent, so these results provide bounded Android regression evidence. iOS application validation remains pending. The verbose Engine flag does not enableFML_VLOG(1), so these device logs do not demonstrate direct coverage of the expired-caller branch; the deterministic C++ regression covers that behavior.Pre-launch Checklist
///).The full local FML suite passes; the final test checkbox remains unchecked pending upstream CI.