Skip to content

Commit f3b1ee7

Browse files
bpleekmgaaclarke
andauthored
Fix/shared thread merger expired lease (flutter#194200)
Fixes flutter#194536. ## Summary Remove expired caller lease records from `SharedThreadMerger` to prevent an engine from aborting when it resumes drawing while another engine keeps the shared raster/platform queues merged. Related to flutter#153211. This PR addresses the `lease_term_ref > 0` assertion reported there, including the corresponding Android scenario. It does not claim to resolve the separate deadlock or backgrounding symptoms discussed in that issue. ## Problem Multiple `RasterThreadMerger` instances can share one `SharedThreadMerger`. The merged state is shared, but each caller has its own lease. The crash occurs in this sequence: 1. Engine A merges with a lease of 1 and consumes it. The queues unmerge, but A's zero-valued entry remains in `lease_term_by_caller_`. 2. Engine B merges the shared queues with an active lease. 3. Engine A resumes drawing. The shared merger reports merged because B's lease is active. 4. A calls `DecrementLease()`, finds its stale zero-valued entry, and aborts on `FML_CHECK(lease_term_ref > 0)`. We reproduced this defect on Android. flutter#153211 reports the same assertion on iOS and includes an [Android report](flutter#153211 (comment)) involving engines in the same `FlutterEngineGroup`. The affected implementation is shared C++ code. However, the iOS application reproduction from flutter#153211 has not been tested with this patch. ## Change - Remove the caller's map entry immediately when its lease reaches zero, under the existing mutex. - Preserve the positive-lease assertion for existing entries. - Allow subsequent draws from an expired caller to use the missing-caller path without consuming another caller's lease. - Use `FML_VLOG(1)` for the missing-caller path, since it can occur during normal rendering. - Combine decrement and expiration checking with `--lease_term_ref`. - Clarify comments describing caller lease lifetime and the shared merge state. The existing all-zero check handles an empty map, so the queues still unmerge when the final active lease expires. This is an internal Engine bug fix with no public API change or migration requirement. ## Production workaround In our Android return-to-bottom scenario, explicitly destroying the outgoing/top Flutter engine in the Activity's `finish()` method, instead of waiting for `onDestroy()`, avoids this crash. We have validated this application-level workaround in production. This changes teardown timing for the transition where the bottom Activity can resume drawing before the top Activity reaches `onDestroy()`. It is a workaround for the affected lifecycle sequence; the stale zero-lease record still needs to be fixed in the Engine. Production validation here applies to the workaround. Patched-Engine Android device results are documented below; iOS application validation remains pending. ## Tests Adds `RasterThreadMerger.ExpiredLeaseCanDecrementWhileAnotherCallerIsMerged`, covering: - Resuming drawing after a caller's lease expires while another caller keeps the queues merged. - Repeated decrements from the expired caller without consuming the other caller's lease. - Unmerging when the remaining lease expires. - Reacquiring and consuming a lease after unmerging. Validated locally on macOS arm64 with `host_debug_unopt_arm64`: - Built `//flutter/fml:fml_unittests`. - New regression test: **1 passed**. - All `RasterThreadMerger` tests: **17 passed**. - Full `fml_unittests`: **204 tests across 38 test suites passed**. - Ran clang-format on changed lines and `git diff --check`. A standalone reproduction using the real FML shared merger and task queue implementations triggers the original assertion before the fix and passes afterward. Patched-Engine Android device results are documented below. iOS application validation, upstream CI and maintainer review remain pending. ### Android device reproduction (unpatched Engine) Confirmed the target SIGABRT on an Android test device running Android 17 with Flutter 3.35.3, Engine `ddf47dd3ff`, debug mode, and Impeller Vulkan. The demo and app-specific crash evidence are attached to flutter#194536: - [lease-zero-demo-sanitized.zip](https://github.com/user-attachments/files/33544786/lease-zero-demo-sanitized.zip) - [zero-lease-crash-sanitized.zip](https://github.com/user-attachments/files/33544788/zero-lease-crash-sanitized.zip) The demo adds a hybrid-composition PlatformView to the bottom engine, removes it and requests 40 frames, then opens a top Activity with a PlatformView in another engine from the same FlutterEngineGroup. Two initial automated cycles did not crash; a subsequent manual cycle in the same process did. The observed abort occurred while opening top again, after top onPostResume and before the return operation. Reproduction is timing-dependent; no success rate is claimed. The captured abort is `shared_thread_merger.cc(59): lease_term_ref > 0`, with `lease_term=0`. This confirms the application-level failure on the unpatched Engine. ### Android device validation (patched Engine, 2026-10-16) Validated the locally built patch on **an Android test device running Android 17 (API 37)** in debug mode. Framework and Engine source both used this PR's head, `bf3e0b302cdb437b932aecc07709f266eb545597`, with `android_debug_unopt_arm64` and matching `host_debug_unopt_arm64` artifacts. - Built with `--no-enable-hcpp` to exercise legacy Hybrid Composition; the APK manifest explicitly sets `EnableHcpp=false`. The `finish()` early-destroy workaround was **not enabled**. - Completed **30/30 ADD -> REMOVE -> OPEN -> return cycles** across three independent app processes, 10 cycles each: 10 with the verbose Engine flag and 20 with normal logging. No zero-lease assertion, native fatal signal, fatal Java exception, Dart exception, or process restart was observed. - Logs record **60 legacy HC PlatformView creations** and **30 returns** in which bottom `onResume()` requested Dart redraw while the top engine was still alive, before top `onDestroy()`. - Verified that the installed APK SHA-256 matches the built APK, and the APK's `libflutter.so` Build ID matches the local Engine: `351451851bb71400213aa0c9764406986940c8c4`. - Reran the rebuilt host `RasterThreadMerger.*` suite: **17/17 passed**, including the new regression. Validation report, event sequences, per-cycle results, and binary identity checks: [engine-validation-sanitized.zip](https://github.com/user-attachments/files/33544787/engine-validation-sanitized.zip) The logs also contain 19 nonfatal `Could not make the resource context current.` messages; the cycles and an additional real-touch responsiveness check still completed. These are recorded in the attached report. The original unpatched device crash used Flutter 3.35.3 / Engine `ddf47dd3ff`. A device A/B comparison at the same framework revision with only this patch removed has not been run. The demo is timing-dependent, so these results provide bounded Android regression evidence. iOS application validation remains pending. The verbose Engine flag does not enable `FML_VLOG(1)`, so these device logs do not demonstrate direct coverage of the expired-caller branch; the deterministic C++ regression covers that behavior. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. The full local FML suite passes; the final test checkbox remains unchecked pending upstream CI. [Contributor Guide]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/master/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md [Flutter Style Guide]: https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/website]: https://github.com/flutter/website [test-exempt]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#tests [breaking change policy]: https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Data Driven Fixes]: https://github.com/flutter/flutter/blob/master/docs/contributing/Data-driven-Fixes.md --------- Co-authored-by: gaaclarke <[email protected]>
1 parent 0399bce commit f3b1ee7

4 files changed

Lines changed: 57 additions & 18 deletions

File tree

‎engine/src/flutter/fml/raster_thread_merger.h‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,9 @@ class RasterThreadMerger
5353
TaskQueueId platform_id,
5454
TaskQueueId raster_id);
5555

56-
// Un-merges the threads now if current caller is the last merged caller,
57-
// and it resets the lease term to 0, otherwise it will remove
58-
// the caller record and return. The multiple caller records were recorded
59-
// after |MergeWithLease| or |ExtendLeaseTo| method.
56+
// Removes this caller's lease and un-merges the threads if no other caller
57+
// has an active lease. Leases are recorded by |MergeWithLease| or
58+
// |ExtendLeaseTo|.
6059
//
6160
// Must be executed on the raster task runner.
6261
//
@@ -69,16 +68,18 @@ class RasterThreadMerger
6968
// When task queues are statically merged this method becomes no-op.
7069
void ExtendLeaseTo(size_t lease_term);
7170

72-
// Returns |RasterThreadStatus::kUnmergedNow| if this call resulted in
73-
// splitting the raster and platform threads. Reduces the lease term by 1.
71+
// Reduces this caller's active lease by 1 and removes it when it expires.
72+
// A caller without a lease is ignored while other callers keep the threads
73+
// merged. Returns |RasterThreadStatus::kUnmergedNow| only if this call
74+
// un-merges the raster and platform threads after all leases expire.
7475
//
7576
// If the task queues are the same, we consider them statically merged.
7677
// When task queues are statically merged this method becomes no-op.
7778
RasterThreadStatus DecrementLease();
7879

79-
// The method is locked by current instance, and asks the shared instance of
80-
// SharedThreadMerger and the merging state is determined by the
81-
// lease_term_ counter.
80+
// Returns whether the task queues are statically merged or the shared
81+
// merger has an active lease from any caller. This caller need not have
82+
// an active lease for the threads to be merged.
8283
bool IsMerged();
8384

8485
// Waits until the threads are merged.

‎engine/src/flutter/fml/raster_thread_merger_unittests.cc‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -557,6 +557,35 @@ TEST(RasterThreadMerger, MultipleMergersCanMergeSameThreadPair) {
557557
ASSERT_FALSE(raster_thread_merger2->IsMerged());
558558
}
559559

560+
TEST(RasterThreadMerger, ExpiredLeaseCanDecrementWhileAnotherCallerIsMerged) {
561+
TaskQueueWrapper queue1;
562+
TaskQueueWrapper queue2;
563+
const auto merger1 = RasterThreadMerger::CreateOrShareThreadMerger(
564+
nullptr, queue1.GetTaskQueueId(), queue2.GetTaskQueueId());
565+
const auto merger2 = RasterThreadMerger::CreateOrShareThreadMerger(
566+
merger1, queue1.GetTaskQueueId(), queue2.GetTaskQueueId());
567+
568+
merger1->MergeWithLease(1);
569+
ASSERT_EQ(merger1->DecrementLease(), RasterThreadStatus::kUnmergedNow);
570+
571+
// A second engine merges the queues before the first engine resumes drawing.
572+
merger2->MergeWithLease(2);
573+
ASSERT_TRUE(merger1->IsMerged());
574+
ASSERT_EQ(merger1->DecrementLease(), RasterThreadStatus::kRemainsMerged);
575+
ASSERT_EQ(merger1->DecrementLease(), RasterThreadStatus::kRemainsMerged);
576+
577+
// Drawing with an expired lease must not consume the second engine's lease.
578+
ASSERT_EQ(merger2->DecrementLease(), RasterThreadStatus::kRemainsMerged);
579+
ASSERT_EQ(merger1->DecrementLease(), RasterThreadStatus::kRemainsMerged);
580+
ASSERT_EQ(merger2->DecrementLease(), RasterThreadStatus::kUnmergedNow);
581+
ASSERT_FALSE(merger1->IsMerged());
582+
ASSERT_FALSE(merger2->IsMerged());
583+
584+
// The first engine can acquire and consume a new lease normally.
585+
merger1->MergeWithLease(1);
586+
ASSERT_EQ(merger1->DecrementLease(), RasterThreadStatus::kUnmergedNow);
587+
}
588+
560589
TEST(RasterThreadMerger, TheLastCallerOfMultipleMergersCanUnmergeNow) {
561590
TaskQueueWrapper queue1;
562591
TaskQueueWrapper queue2;

‎engine/src/flutter/fml/shared_thread_merger.cc‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -59,15 +59,22 @@ bool SharedThreadMerger::DecrementLease(RasterThreadMergerId caller) {
5959
FML_CHECK(lease_term_ref > 0)
6060
<< "lease_term should always be positive when merged, lease_term="
6161
<< lease_term_ref;
62-
lease_term_ref--;
62+
if (--lease_term_ref == 0) {
63+
// Remove the expired lease so future draws by this caller are ignored
64+
// while another caller keeps the shared queues merged.
65+
lease_term_by_caller_.erase(entry);
66+
}
6367
} else {
64-
FML_LOG(WARNING) << "The caller does not exist when calling "
65-
"DecrementLease(), ignored. This may happens after "
66-
"caller is erased in UnMergeNowIfLastOne(). caller="
67-
<< caller;
68+
// The caller may still draw after its lease expires or is removed while
69+
// another caller keeps the shared queues merged.
70+
FML_VLOG(1) << "The caller does not exist when calling "
71+
"DecrementLease(), ignored. This may happen after "
72+
"the caller's lease expires or the caller is erased in "
73+
"UnMergeNowIfLastOne(). caller="
74+
<< caller;
6875
}
6976
if (IsAllLeaseTermsZeroUnSafe()) {
70-
// Unmerge now because lease_term_ decreased to zero.
77+
// Unmerge only after all callers' leases have expired or been removed.
7178
UnMergeNowUnSafe();
7279
return true;
7380
}

‎engine/src/flutter/fml/shared_thread_merger.h‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -61,9 +61,11 @@ class SharedThreadMerger
6161
std::mutex mutex_;
6262
bool enabled_ = false;
6363

64-
/// The |MergeWithLease| or |ExtendLeaseTo| method will record the caller
65-
/// into this lease_term_by_caller_ map, |UnMergeNowIfLastOne|
66-
/// method will remove the caller from this lease_term_by_caller_.
64+
/// Active leases keyed by caller. |MergeWithLease| and |ExtendLeaseTo|
65+
/// record leases; |DecrementLease| removes them when they reach zero, and
66+
/// |UnMergeNowIfLastOne| removes its caller's lease immediately. A caller
67+
/// without a lease may still draw while another caller keeps the queues
68+
/// merged.
6769
std::map<RasterThreadMergerId, std::atomic_size_t> lease_term_by_caller_;
6870

6971
bool IsAllLeaseTermsZeroUnSafe() const;

0 commit comments

Comments
 (0)