Skip to content

Windows: packed memory-sync sidecar probe times out at 60 s and leaks a locked temp dir (Loro init test) #10911

Description

@code-yeongyu

Summary

On windows-latest, the omo-senpi test "checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm" (packages/omo-senpi/plugin/scripts/build-extension.test.mjs:377) timed out after 60 s. Its temp tree then could not be removed (EBUSY), so the #9766 temp-entry check failed the job a second time. Whether this is CI timing or a real Windows problem in the packed memory-sync runtime is not known yet. The test proves that a user's installed package can load the sync engine, so it is triaged as a possible user-facing Windows sync bug until measured.

Reproduction

  • dev CI run 39773252352 at 125fce2, job senpi-compatibility (windows-latest) (119809817563). That commit changes only a budget number in packages/omo-senpi/src/bundle-size.test.ts, so it is unrelated.
  • The same test passed on windows-latest in dev runs 39762215681 and 39750545847. It is intermittent, not deterministic.
killed 1 dangling process
(fail) checkExtensionCurrent > ... Loro initializes from the packed wasm [60003.55ms]
  ^ this test timed out after 60000ms.
# Unhandled error between tests
403 |       const run = spawnSync("node", [probe], { cwd: parse(process.cwd()).root, encoding: "utf8", shell: process.platform === "win32" })
EBUSY: resource busy or locked, rm '~\AppData\Local\Temp\omo-test-run-<id>\omo-sync-sidecar-pack-<id>'

What the test does

It copies the built omo-memory-sync.js sidecar and its Loro wasm assets into a scratch package and runs npm pack (shell: true on win32). It untars the tarball with tar -xzf, then runs node sync-probe.mjs (shell: true, cwd = the drive root). The probe imports #omo-memory-sync-runtime and awaits checkSyncRuntime(). All three child processes are spawnSync, so the test cannot observe progress, and its timeout cannot stop a hung child. Bun reported killing one dangling process, and the scratch dir stayed locked, which means a child (most likely the node probe) was still alive when the test gave up.

Expected

The packed sidecar initializes Loro and the probe exits well inside the budget on Windows. A probe that hangs is reported with its stage (pack / untar / import / checkSyncRuntime) and its output, and is killed before the scratch dir is removed, so it never leaks a locked temp tree.

Hypotheses to separate (each needs a measurement)

  1. The probe node never exits on win32: checkSyncRuntime() or the sidecar's import leaves a handle open. That would be a real Windows bug in the shipped runtime.
  2. A slow first wasm compile or cold node start under shard load; timing only.
  3. The shell: true launch through cmd.exe with cwd = drive root (or npm pack under cmd) stalls.

Acceptance

  • A measurement on a hosted Windows runner, outside the full shard, names which stage takes the time, and whether the packed runtime ever fails to exit.
  • If it is a real runtime bug, it is fixed in the sidecar with a regression test. The test also gets per-stage async spawns with a kill-on-timeout and a reported stage, so a hang is diagnosable and never leaks the scratch dir.

Related

#9766 (temp-entry check), #10310 / #10814 (other Windows shard flakes; distinct cause).

Activity

  1. code-yeongyu commented on Oct 22, 2026

    @code-yeongyu
    OwnerAuthor

    Reproduction outside the CI shard (run 39788672304): hosted windows-latest, 3 jobs, images 20261007.264.1 (2 jobs) and 20261018.275.1, node v24.21.0, npm 11.19.0, tar = C:\Windows\system32\tar.exe (pwsh, as in CI).

    The packed runtime: 36 packs, 72 node probes, every one sync-sidecar-ok, none hung.

    Stage (async spawn, 45 s cap, per job) Times
    npm pack (shell) 460-843 ms
    tar -xzf 26-51 ms (first run per job 100-295 ms)
    node sync-probe.mjs, shell: true, cwd = drive root (the test's launch) 81-145 ms
    node sync-probe.mjs, direct, cwd = package 76-126 ms
    inside the probe: import("#omo-memory-sync-runtime") / checkSyncRuntime() 15-25 ms / 16-33 ms

    No probe was still alive after it printed (the 5 s STILL_ALIVE handle dump never fired), so the packed sidecar does not hold the process open on Windows. The test itself, alone, 30 runs: 30 passed (16-40 s per run including the build fixture).

    Conclusion: no evidence of a Windows bug in the shipped memory-sync runtime. Loro initializes from the packed wasm in about 40 ms, and the process exits. The 60 s CI failure happened under full-shard load. Because every stage is a blocking spawnSync, it left no record of which stage stalled. Bun later "killed 1 dangling process", and the scratch dir stayed locked (EBUSY).

    Test-side fix (next): run each stage with an async spawn and a per-stage cap. Report the stalled stage with its output, kill it, and wait for its exit before removing the scratch dir. That way a recurrence names its stage and never leaks a locked tree. One more portability finding: run under Git Bash's PATH, tar is GNU tar, which reads C:\... as a remote host and fails (exit 2). The test only passes because CI runs it from pwsh. The fix should pin the Windows tar (or avoid the external tar).

  2. code-yeongyu commented on Oct 22, 2026

    @code-yeongyu
    OwnerAuthor

    Closed by #10917 (merged as ea4341e). The packed memory-sync runtime was measured healthy on hosted Windows: 72/72 probes, Loro init about 40 ms, the test alone 30/30 (run 39788672304). Each test stage (npm pack / tar / node probe) now runs with its own cap, and the test timeout is derived from those caps. A stall kills its process tree and fails naming the stage, and the probe no longer runs through a shell that could orphan a file-holding node. At the merged head CI was 23 pass, 5 skipped, 0 fail, including all Windows legs.

  3. code-yeongyu commented on Oct 22, 2026

    @code-yeongyu
    OwnerAuthor

    Reopening: the test still fails on hosted Windows with #10917's fix present.

    • Where: release-state PR release: v5.1.42 #10930 (v5.1.42) at 7ca0c35, job senpi-compatibility (windows-latest) 119939407113 (6190 pass / 2 fail).
    • git merge-base --is-ancestor ea4341e056 7ca0c35145 exits 0, so test(omo-senpi): bound and name each packed-sidecar stage so a Windows stall is diagnosable and leak-free #10917 (merged as ea4341e) is in that tree.
    • The two failure lines:
      • (fail) checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm [25468.29ms]
      • (fail) (unnamed) [2633.59ms], from the cleanup at packages/omo-senpi/plugin/scripts/build-extension.test.mjs:412: EBUSY: resource busy or locked, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\omo-test-run-6BPOSY\omo-sync-sidecar-pack-Ln9d94'

    The bounded stages from #10917 did not stop the probe from overrunning, and did not stop the work directory from staying locked on this runner. Test-only: the shipped sidecar is unchanged.

  4. code-yeongyu commented on Oct 24, 2026

    @code-yeongyu
    OwnerAuthor

    Seen again on omo#11126 (the senpi 2026.10.23-1 adoption for omo 5.1.45), job senpi-compatibility (windows-latest): https://github.com/code-yeongyu/oh-my-openagent/actions/runs/40056938124/job/120766356446

    (fail) checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm [25477.66ms]
    error: Tests left 1 temp entry behind (#9766). ...:
      omo-sync-sidecar-pack-tOu6l2
    (fail) (unnamed) [3289.55ms]
    

    dev's own push run at the merge base (1a62c298d2, run 40053513652) passed this test on the same leg; #11126 touches no memory-sync code. Release-train shipped omo 5.1.45 citing this issue for that failure (lead ruling), and one rerun --failed of the leg is running as evidence. The Windows desktop app uses the packed sync sidecar, so the desktop pin waits on that rerun.

  5. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    Seen again on dev at ae481df (CI run https://github.com/code-yeongyu/oh-my-openagent/actions/runs/40650695644, job 122707733786, senpi-compatibility (windows-latest)): 7190 pass, 1 fail. The only failure is checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm [49085 ms], which matches this issue. The merge that triggered the run (#11535) only touches macOS-only Rust crates.

  6. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    Owner: bugwatch (Sisyphus Labs), lane bw-win-10911, P1. Still red on dev Windows senpi-compatibility (run 40650695644). Plan: find which step eats the ~49 s and which handle keeps the temp dir locked, fix the root (an unawaited or unkilled child, an unclosed handle, caches under the temp dir), no raised timeouts, proven 10x under load; the full Windows matrix on the post-merge dev push is the final proof.

  7. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    Owner change: this lane (memsync-coord) owns #10911 from now. Plan, root cause first:

    1. The latest failure (job 122707733786) is narrower than the original 60 s timeout. The npm pack stage printed its tarball name (code-yeongyu-omo-senpi-5.1.49.tgz), so the pack itself finished, yet the npm process did not exit within its 25 s cap. Find what npm does after writing the tarball on Windows that can take that long (lifecycle scripts, log or cache maintenance, a lingering child), and why the scratch directory stays locked after the tree kill.
    2. Write a RED test with simulated Windows behavior, since this can't be reproduced locally.
    3. Open a fix PR with Fixes #10911, without raising any timeout.
  8. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    Owner correction: the fix is #11648 (memsync-coord), so bugwatch has stood its lane down. One gap was flagged to memsync-coord: the stage tree kill (windowsTreeKillArgs) can miss the hung node, and #11648 changes only test-support files, so that needs its own fix or a follow-up issue.

  9. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    Follow-up for the gap this PR doesn't close: #11651. The shared windowsTreeKillArgs kills nothing once the root process is gone, and kills only the root when the process listing fails, so other callers can still orphan a hung descendant. This PR removes the shell for the npm pack stage. #11651 fixes the kill itself. Owner: memsync-coord.

  10. added a commit that references this issue on Oct 28, 2026
    421f6d6
  11. code-yeongyu commented on Oct 28, 2026

    @code-yeongyu
    OwnerAuthor

    #11648 merged into dev as 421f6d6.

    The real proof is the next senpi-compatibility (windows-latest) legs on dev. I'll check them. If this test stalls there again, this issue gets reopened with the job log.

  12. code-yeongyu commented on Oct 29, 2026

    @code-yeongyu
    OwnerAuthor

    Confirmed on dev after #11648 (421f6d6): the packed sync-sidecar test passed in 3 of 3 real senpi-compatibility (windows-latest) runs. No npm pack did not finish line appeared in any of them.

    Run Job
    40773273643 123127309438
    40776891403 123150495587
    40790566890 123197719573

    One cancelled run (40760561544) isn't counted. Follow-ups #11716 (Fixes #11651) and #11718 are still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions