Skip to content

fix(tui): synchronize reset-progress ReactiveProperties to stop flaky host crash#1525

Merged
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/init-reset-reactiveproperty-race
Jun 30, 2026
Merged

fix(tui): synchronize reset-progress ReactiveProperties to stop flaky host crash#1525
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/init-reset-reactiveproperty-race

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

Fixes the flaky InitExistingInstallViewModelTests.ResetFailure_ShowsErrorAndDoesNotNavigate test-host crash introduced by #1514 ("#1494 stop daemon before reset").

Root cause

#1494 moved the install reset onto a background Task.Run thread that publishes progress by writing four ReactiveProperty instances — StatusMessage, CurrentProgressStep, ProgressMessage, CanQuitProgress.

  • In production the view model is bound to the Termina render loop, so PublishOnLoop marshals every write onto the single loop thread — single-threaded, safe.
  • In a unit test the view model is never bound, so ReactiveViewModel's default InvokeAsync runs the write inline on the background thread, while the test thread concurrently Subscribes/disposes the same property in the WaitFor* helpers.

R3's ReactiveProperty<T> is not thread-safe: its value-set path walks the observer linked list lock-free, while Subscribe/Dispose mutate that list under lock(this). The two race and can corrupt the list into a cycle → the value-set walk never terminates → the reset Task never completes → the xunit host is killed by the watchdog. That matches every symptom: flaky, Ubuntu-only on a given run, no managed stack trace, and the same commit observed both green and red. (ThrowUnobservedTaskExceptions is off, confirming it's the hang-cycle, not an unobserved fault.)

Diagnosed against the dotnet-skills:r3-reactive-extensions concurrency contract: "OnNext must not be called concurrently/re-entrantly from multiple threads … or use SynchronizedReactiveProperty."

Fix

Make the four properties written from the background thread SynchronizedReactiveProperty<T> (R3's built-in lock-based variant — serializes set/subscribe/dispose on one monitor; uncontended in production). The declared type stays ReactiveProperty<T>, so InitExistingInstallPage and every reader is unchanged. CurrentPhase/SelectedIndex are input-thread-only and left as plain ReactiveProperty.

Verification

  • Builds clean; all 16 InitExistingInstall tests pass, including the formerly-flaky ResetFailure case across repeated runs.
  • Slopwatch: 0 issues; copyright headers present.
  • The race only surfaced on specific CI hardware — hundreds of local iterations (incl. heavy contention) didn't reproduce — so CI over time is the real proof. The change is a minimal, root-cause fix that reuses an existing R3 type.

… host crash

netclaw-dev#1494/netclaw-dev#1514 moved the install reset onto a background Task.Run thread that publishes
progress by writing four ReactiveProperty instances (StatusMessage, CurrentProgressStep,
ProgressMessage, CanQuitProgress). In production those writes marshal onto the single
Termina render-loop thread, but an unbound view model (unit tests) runs them inline on the
background thread while the test thread subscribes/disposes the same property.

R3's ReactiveProperty is not thread-safe: its value-set walks the observer linked list
lock-free while Subscribe/Dispose mutate that list under a lock, so the two race and can
corrupt the list into a cycle -> infinite loop -> the reset Task never completes and the
xunit host is killed by the watchdog (flaky 'Test host process crashed', no stack trace,
Ubuntu-only; the same commit was seen both green and red). R3's documented remedy for
shared state written from multiple threads is SynchronizedReactiveProperty, which
serializes set/subscribe/dispose on one monitor (uncontended in production). The declared
type stays ReactiveProperty<T> so the page and all readers are unchanged.

Fixes flaky InitExistingInstallViewModelTests.ResetFailure_ShowsErrorAndDoesNotNavigate.
@Aaronontheweb Aaronontheweb added the tests All issues related to testing, quality assurance, and smoke testing. label Jun 30, 2026
@Aaronontheweb
Aaronontheweb merged commit 850cf8c into netclaw-dev:dev Jun 30, 2026
15 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/init-reset-reactiveproperty-race branch June 30, 2026 14:55
Aaronontheweb added a commit that referenced this pull request Jul 3, 2026
…p 3 (#1573)

RunResetAsync publishes CurrentProgressStep = 3 ("Purge complete") and then
evaluates the Task.Delay(CompletionPause, _timeProvider, ct) that registers the
completion-pause timer. Because RunResetAsync runs off the loop, step 3 is the
signal observers gate on to know deletion finished — but it fired BEFORE the
timer it vouches for existed. On a virtualized clock (FakeTimeProvider in tests,
and any consumer advancing the injected TimeProvider) this is a lost wakeup: an
Advance that lands between "step 3 published" and "timer registered" is dropped,
the delay never fires, and the reset hangs forever awaiting a wakeup that will
never come.

Fix: create the completion-pause timer before publishing step 3, then await the
captured task. This makes "step 3 observed" a happens-before guarantee that the
pause timer already exists, so a single clock advance always fires it
deterministically. On the real clock this shifts timer registration a few
microseconds earlier; the visible "Purge complete" dwell is unchanged.

Evidence: hang-dump forensics of CI run 28676578217 (PR #1568, Netclaw.Cli.Tests
blame-hang after 300s). Test FullReset_AfterBothConfirmations_DeletesEverything
was parked at `await vm.ResetTask!.WaitAsync(ct)`; its CompleteResetAsync helper
had exhausted its advance loop (counter i == 10, all virtual-clock advances
spent) and FakeTimeProvider was left holding exactly one orphaned waiter — the
delay registered after the final Advance. Prior commits 850cf8c (#1525) and
eb01b49 (#1536) band-aided the symptom with advance/yield retry loops; this
removes the race at the source.

The test helper CompleteResetAsync is shared by all three sibling tests
(FullReset_AfterBothConfirmations_DeletesEverything,
SetupOnlyReset_DeletesConfigButKeepsMemoryAndSessions,
DaemonStopFailureResult_ShowsStatusAndContinuesReset); it now waits for step 3
then does exactly one advance, deleting the retry loop that only ever masked the
lost wakeup on loaded runners.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests All issues related to testing, quality assurance, and smoke testing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant