Skip to content

Commit fae3f03

Browse files
matt2eclaudeJitter
authored
test(mobile): keep late callbacks and fixed sleeps from failing tests (#8678)
## Summary Fixes three mobile flakes from the 21–28 Oct 2026 CI runs, in test code only: - In two iOS `RunnerTests` classes, late bridge callbacks no longer fail whichever test is running when they fire. - The waits on a busy test-host main thread now cap at 10 s. - Two Flutter avatar tests poll for the ready frame instead of sleeping a fixed 50 ms. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | `BuzzPushSnapshotEnrichmentTests` (RunnerTests) | 40362671989 (job 121712747802, main), 39681273328#1 (job 119504560518), 40658588384 (job 122715950541, PR #8640) | The 1 s waits ran out while the test host's Flutter app held the main thread, and the test returned and deleted its temp directory. The bridge callback's assertions then ran later and were blamed on whichever test was running (`ProfileSetupRecoveryTests`, `IdentityFilePluginTests`). | Callbacks only record the value and fulfill. Every assertion, including the `isRestricted` checks, runs after the wait, so a late callback writes into a dead local. The seven waits cap at 10 s (`callbackTimeout`). | | `ProfileSetupRecoveryTests` `testStartedEditorWithoutReadyRestoresProfileAndAllowsRetry` | 40585805357 (job 122477274956, PR #8568) | Engine release and editor dismissal must both happen within 2 s, on the main thread the host app keeps busy. | Both predicate waits cap at 10 s. | | Flutter `channel_detail_page_test.dart` "profile sheet shows the poster before autoplay and restores it on tap" and `settings_profile_header_test.dart` "shows the poster until the animated avatar is ready" | 40368315739 (job 121732711851, main) | After the image response is released, both tests slept a fixed 50 ms of real time before `pumpAndSettle`. A slower real decode left no frame scheduled, so the ready key was missing. | A new `WidgetHelpers.pumpUntilFound` pumps in 10 ms real-time steps until the ready key appears, capped at about 5 s. | Local evidence is from the original `flakey-tests` commit (53aaec97c), Xcode 27.1 on the iOS 26.2 simulator: - Enrichment, with `Self.complete` delayed 1.5 s and 4 s: 6 failures before, 107/107 after. With a 12 s delay, every failure lands in its owning test. A mutated error code is still caught. - Profile setup, with the engine release held 4 s: 2 failures before, 4/4 after. A leaked engine still fails. - Avatar tests: under load, 94/96 before and 96/96 after. With a 200 ms image delay, 0/5 before and 5/5 after. A widget that never signals ready still fails. Not included: - The planned `AppDelegate` / `Runner.xcscheme` change that keeps Flutter out of the RunnerTests host. It did not work locally: restored scene sessions and the `UIMainStoryboardFile` fallback, and a normal launch came up black afterwards. - `IdentityFilePluginTests` `testSyntheticDocumentsReadInPlaceAndRejectInvalidOrOversize` (40441255307#1). It is left unchanged and stays open. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs the Flutter tests. - Every file is byte-identical to its final version on `flakey-tests` (67185df6a). - Checked for this PR: `git diff --check`, and a conflict-free merge with all six siblings onto `main`. - #8613's flake-check lane will repeat the two changed Dart test files. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
1 parent d263e2c commit fae3f03

7 files changed

Lines changed: 87 additions & 36 deletions

File tree

‎mobile/ios/RunnerTests/BuzzCommunicationNotificationTests.swift‎

Lines changed: 40 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,12 @@ final class BuzzCommunicationNotificationTests: XCTestCase {
242242
}
243243
}
244244

245+
/// Callbacks only record what they receive and assertions run after the wait,
246+
/// so a callback that lands after its test returned cannot fail a later test.
247+
/// The timeout is a cap; a passing test returns as soon as its callback runs.
245248
final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
249+
private let callbackTimeout: TimeInterval = 10
250+
246251
func testAgeSignalRequestDoesNotDependOnNotificationStorage() {
247252
let delegate = MissingAppGroupDelegate()
248253
var completed = false
@@ -268,28 +273,32 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
268273
BuzzAgeSignalPayload.sharing(exclusiveUpperBound: 18, lowerBound: 13)
269274
}
270275
let minor = expectation(description: "native minor response")
276+
var minorValue: Any?
271277
delegate.handleAgeSignalMethodCall(
272278
FlutterMethodCall(methodName: "requestAgeSignal", arguments: nil),
273279
viewController: UIViewController()
274280
) { value in
275-
XCTAssertEqual((value as? [String: Any])?["status"] as? String, "signal")
276-
XCTAssertEqual((value as? [String: Any])?["ageUpper"] as? Int, 17)
281+
minorValue = value
277282
minor.fulfill()
278283
}
279-
await fulfillment(of: [minor], timeout: 1)
284+
await fulfillment(of: [minor], timeout: callbackTimeout)
285+
XCTAssertEqual((minorValue as? [String: Any])?["status"] as? String, "signal")
286+
XCTAssertEqual((minorValue as? [String: Any])?["ageUpper"] as? Int, 17)
280287

281288
delegate.requestPlatformAgeSignal = { _ in
282289
throw NSError(domain: "InjectedAgeFailure", code: 1)
283290
}
284291
let failed = expectation(description: "native error response")
292+
var failedValue: Any?
285293
delegate.handleAgeSignalMethodCall(
286294
FlutterMethodCall(methodName: "requestAgeSignal", arguments: nil),
287295
viewController: UIViewController()
288296
) { value in
289-
XCTAssertEqual((value as? FlutterError)?.code, "age_signal_unavailable")
297+
failedValue = value
290298
failed.fulfill()
291299
}
292-
await fulfillment(of: [failed], timeout: 1)
300+
await fulfillment(of: [failed], timeout: callbackTimeout)
301+
XCTAssertEqual((failedValue as? FlutterError)?.code, "age_signal_unavailable")
293302
}
294303

295304
func testAllowedNotificationRestoreNeedsNoAppGroupStore() {
@@ -299,13 +308,15 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
299308
keychainAccessGroup: nil
300309
)
301310
let completed = expectation(description: "allowed without storage")
311+
var received: Any?
302312
XCTAssertTrue(bridge.handle(
303313
FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil)
304314
) { value in
305-
XCTAssertNil(value)
315+
received = value
306316
completed.fulfill()
307317
})
308-
wait(for: [completed], timeout: 1)
318+
wait(for: [completed], timeout: callbackTimeout)
319+
XCTAssertNil(received)
309320
}
310321

311322
func testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority() throws {
@@ -335,25 +346,29 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
335346
)
336347
)
337348
let restricted = expectation(description: "restriction despite cleanup failure")
349+
var purgeResult: Any?
338350
XCTAssertTrue(bridge.handle(
339351
FlutterMethodCall(methodName: "purgeAgeRestrictedNotifications", arguments: nil)
340352
) { value in
341-
XCTAssertEqual((value as? FlutterError)?.code, "age_restriction_purge_failed")
342-
XCTAssertTrue(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
353+
purgeResult = value
343354
restricted.fulfill()
344355
})
345-
wait(for: [restricted], timeout: 1)
356+
wait(for: [restricted], timeout: callbackTimeout)
357+
XCTAssertEqual((purgeResult as? FlutterError)?.code, "age_restriction_purge_failed")
358+
XCTAssertTrue(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
346359
XCTAssertEqual(try Data(contentsOf: snapshotURL), original)
347360

348361
let restored = expectation(description: "allowed without restoring snapshot")
362+
var restoreResult: Any?
349363
XCTAssertTrue(bridge.handle(
350364
FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil)
351365
) { value in
352-
XCTAssertNil(value)
353-
XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
366+
restoreResult = value
354367
restored.fulfill()
355368
})
356-
wait(for: [restored], timeout: 1)
369+
wait(for: [restored], timeout: callbackTimeout)
370+
XCTAssertNil(restoreResult)
371+
XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
357372
XCTAssertEqual(deletionCalls, 2)
358373
XCTAssertEqual(try Data(contentsOf: snapshotURL), original)
359374
}
@@ -365,24 +380,28 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
365380
let failure = NSError(domain: "InjectedCleanupFailure", code: 1)
366381
let deletion = BuzzInteractionDeletionDeadline(timeout: 5,
367382
deleteAllInteractions: { $0(failure) }, scheduleTimeout: { _, _ in })
383+
var seedError: Error?
368384
BuzzInteractionCleanupRetry(containerURL: directory, deletion: deletion).request {
369-
XCTAssertNotNil($0)
385+
seedError = $0
370386
}
387+
XCTAssertNotNil(seedError)
371388
let bridge = BuzzPushSnapshotBridge(
372389
appGroupIdentifier: nil,
373390
endpointGrantStore: BuzzPushEndpointGrantKeychainStore(accessGroup: nil),
374391
keychainAccessGroup: nil, containerURL: { directory },
375392
interactionDeletionDeadline: deletion
376393
)
377394
let completed = expectation(description: "retry failure stays allowed")
395+
var received: Any?
378396
XCTAssertTrue(bridge.handle(
379397
FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil)
380398
) { value in
381-
XCTAssertEqual((value as? FlutterError)?.code, "interaction_cleanup_retry_failed")
382-
XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
399+
received = value
383400
completed.fulfill()
384401
})
385-
wait(for: [completed], timeout: 1)
402+
wait(for: [completed], timeout: callbackTimeout)
403+
XCTAssertEqual((received as? FlutterError)?.code, "interaction_cleanup_retry_failed")
404+
XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory))
386405
}
387406

388407
func testStrictAgeGateWriteFailsWhenAppGroupStoreIsUnavailable() {
@@ -392,6 +411,7 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
392411
keychainAccessGroup: nil
393412
)
394413
let completed = expectation(description: "strict write rejected")
414+
var received: Any?
395415

396416
XCTAssertTrue(
397417
bridge.handle(
@@ -405,12 +425,13 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase {
405425
]
406426
)
407427
) { value in
408-
XCTAssertEqual((value as? FlutterError)?.code, "snapshot_sync_unavailable")
428+
received = value
409429
completed.fulfill()
410430
}
411431
)
412432

413-
wait(for: [completed], timeout: 1)
433+
wait(for: [completed], timeout: callbackTimeout)
434+
XCTAssertEqual((received as? FlutterError)?.code, "snapshot_sync_unavailable")
414435
}
415436

416437
func testMetadataAuthorityUsesNewestGrantForMatchingRelay() {

‎mobile/ios/RunnerTests/ProfileSetupRecoveryTests.swift‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -163,8 +163,9 @@ final class ProfileSetupRecoveryTests: XCTestCase {
163163
XCTAssertFalse(avatar.isHidden)
164164
}
165165
// UIKit releases popped controllers at the end of its autorelease turn.
166+
// The cap absorbs a busy main thread; a leaked engine never becomes nil.
166167
let released = XCTNSPredicateExpectation(predicate: NSPredicate { _, _ in engine == nil }, object: nil)
167-
await fulfillment(of: [released], timeout: 2)
168+
await fulfillment(of: [released], timeout: 10)
168169
}
169170
XCTAssertEqual(watchdogs.count, 2)
170171
skip.sendActions(for: .touchUpInside)
@@ -178,7 +179,7 @@ final class ProfileSetupRecoveryTests: XCTestCase {
178179
XCTAssertTrue(notNow.isEnabled)
179180
notNow.sendActions(for: .touchUpInside)
180181
let dismissed = XCTNSPredicateExpectation(predicate: NSPredicate { _, _ in parent.presentedViewController == nil }, object: nil)
181-
await fulfillment(of: [dismissed], timeout: 2)
182+
await fulfillment(of: [dismissed], timeout: 10)
182183
XCTAssertEqual(messenger.completionAction, "skip")
183184
withExtendedLifetime(coordinator) {}
184185
nav.dismiss(animated: false)

‎mobile/test/features/channels/channel_detail_page_test.dart‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ import 'package:buzz/shared/widgets/skeleton.dart';
9090
import 'package:shared_preferences/shared_preferences.dart';
9191

9292
import '../../helpers/thread_ids.dart';
93+
import '../../helpers/widget_helpers.dart';
9394

9495
part 'thread_reply_refresh_cases.dart';
9596
part 'thread_title_capsule_cases.dart';
@@ -10119,17 +10120,13 @@ void main() {
1011910120
);
1012010121

1012110122
animationResponse.complete(http.Response.bytes(_transparentPng, 200));
10122-
await tester.runAsync(
10123-
() => Future<void>.delayed(const Duration(milliseconds: 50)),
10123+
final animationReady = find.byKey(
10124+
const ValueKey('progressive-animated-avatar-animation-ready'),
1012410125
);
10126+
await WidgetHelpers.pumpUntilFound(tester, animationReady);
1012510127
await tester.pumpAndSettle();
1012610128

10127-
expect(
10128-
find.byKey(
10129-
const ValueKey('progressive-animated-avatar-animation-ready'),
10130-
),
10131-
findsOneWidget,
10132-
);
10129+
expect(animationReady, findsOneWidget);
1013310130
expect(
1013410131
find.byKey(const ValueKey('progressive-animated-avatar-poster')),
1013510132
findsNothing,

‎mobile/test/features/profile/settings_profile_header_test.dart‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -128,15 +128,13 @@ void main() {
128128
containsAll([posterUrl, animationUrl]),
129129
);
130130
animationResponse.complete(http.Response.bytes(_transparentPng, 200));
131-
await tester.runAsync(
132-
() => Future<void>.delayed(const Duration(milliseconds: 50)),
131+
final animationReady = find.byKey(
132+
const ValueKey('progressive-animated-avatar-animation-ready'),
133133
);
134+
await WidgetHelpers.pumpUntilFound(tester, animationReady);
134135
await tester.pumpAndSettle();
135136

136-
expect(
137-
find.byKey(const ValueKey('progressive-animated-avatar-animation-ready')),
138-
findsOneWidget,
139-
);
137+
expect(animationReady, findsOneWidget);
140138
expect(
141139
find.byKey(const ValueKey('progressive-animated-avatar-poster')),
142140
findsNothing,

‎mobile/test/helpers/widget_helpers.dart‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import 'package:flutter/material.dart';
2+
import 'package:flutter_test/flutter_test.dart';
23
import 'package:hooks_riverpod/hooks_riverpod.dart';
34
import 'package:hooks_riverpod/misc.dart';
45
import 'package:buzz/shared/theme/theme.dart';
@@ -24,4 +25,16 @@ class WidgetHelpers {
2425
),
2526
);
2627
}
28+
29+
/// Pumps until [finder] matches, letting real async work such as an image
30+
/// decode run between frames. Gives up after about 5 s so the caller's
31+
/// `expect` still reports a real regression.
32+
static Future<void> pumpUntilFound(WidgetTester tester, Finder finder) async {
33+
for (var i = 0; i < 500 && finder.evaluate().isEmpty; i++) {
34+
await tester.runAsync(
35+
() => Future<void>.delayed(const Duration(milliseconds: 10)),
36+
);
37+
await tester.pump();
38+
}
39+
}
2740
}

‎scripts/flake-check.mjs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1115,6 +1115,9 @@ function repeatNode(files, iterations) {
11151115

11161116
// Per-test statuses of a `dart test` JSON report (one event per line), keyed
11171117
// `<file> › <name>`. Hidden entries are the runner's own file-loading steps.
1118+
// A test declared through a helper (every `testWidgets` is) has the helper's
1119+
// `package:` URL as its `url`, so its file comes from its suite, else from
1120+
// `root_url`.
11181121
export function parseDartReport(text, mobileRoot = `${root}mobile/`) {
11191122
const started = new Map();
11201123
const suites = new Map();
@@ -1131,8 +1134,9 @@ export function parseDartReport(text, mobileRoot = `${root}mobile/`) {
11311134
if (event.type !== "testDone" || event.hidden) continue;
11321135
const test = started.get(event.testID);
11331136
if (!test) continue;
1137+
const url = test.root_url ?? test.url;
11341138
const source = suites.get(test.suiteID) ??
1135-
(test.url?.startsWith("file:") ? fileURLToPath(test.url) : test.url ?? "unknown");
1139+
(url?.startsWith("file:") ? fileURLToPath(url) : url ?? "unknown");
11361140
const file = source.replace(mobileRoot, "");
11371141
statuses.set(
11381142
`${file} › ${test.name}`,

‎scripts/flake-check.test.mjs‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -953,6 +953,15 @@ test("reports give per-test statuses per iteration", () => {
953953
type: "testStart",
954954
test: { id: 4, name: "later", url: "file:///m/test/b_test.dart" },
955955
}),
956+
event({
957+
type: "testStart",
958+
test: {
959+
id: 5,
960+
name: "widget renders",
961+
url: "package:flutter_test/src/widget_tester.dart",
962+
root_url: "file:///m/test/b_test.dart",
963+
},
964+
}),
956965
"not json",
957966
event({
958967
type: "testDone",
@@ -975,6 +984,13 @@ test("reports give per-test statuses per iteration", () => {
975984
hidden: false,
976985
skipped: true,
977986
}),
987+
event({
988+
type: "testDone",
989+
testID: 5,
990+
result: "success",
991+
hidden: false,
992+
skipped: false,
993+
}),
978994
event({ type: "done", success: false }),
979995
].join("\n"),
980996
"/m/",
@@ -983,6 +999,7 @@ test("reports give per-test statuses per iteration", () => {
983999
["test/a_test.dart › group renders", "passed"],
9841000
["test/a_test.dart › group fails", "failed"],
9851001
["test/b_test.dart › later", "skipped"],
1002+
["test/b_test.dart › widget renders", "passed"],
9861003
]),
9871004
);
9881005
// Playwright lists a --repeat-each journey once per repetition, in order.

0 commit comments

Comments
 (0)