Repository navigation
Description
Summary
On windows-latest, the omo-senpi test "checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm" (packages/omo-senpi/plugin/scripts/build-extension.test.mjs:377) timed out after 60 s. Its temp tree then could not be removed (EBUSY), so the #9766 temp-entry check failed the job a second time. Whether this is CI timing or a real Windows problem in the packed memory-sync runtime is not known yet. The test proves that a user's installed package can load the sync engine, so it is triaged as a possible user-facing Windows sync bug until measured.
Reproduction
- dev CI run 39773252352 at 125fce2, job
senpi-compatibility (windows-latest)(119809817563). That commit changes only a budget number inpackages/omo-senpi/src/bundle-size.test.ts, so it is unrelated. - The same test passed on
windows-latestin dev runs 39762215681 and 39750545847. It is intermittent, not deterministic.
killed 1 dangling process
(fail) checkExtensionCurrent > ... Loro initializes from the packed wasm [60003.55ms]
^ this test timed out after 60000ms.
# Unhandled error between tests
403 | const run = spawnSync("node", [probe], { cwd: parse(process.cwd()).root, encoding: "utf8", shell: process.platform === "win32" })
EBUSY: resource busy or locked, rm '~\AppData\Local\Temp\omo-test-run-<id>\omo-sync-sidecar-pack-<id>'
What the test does
It copies the built omo-memory-sync.js sidecar and its Loro wasm assets into a scratch package and runs npm pack (shell: true on win32). It untars the tarball with tar -xzf, then runs node sync-probe.mjs (shell: true, cwd = the drive root). The probe imports #omo-memory-sync-runtime and awaits checkSyncRuntime(). All three child processes are spawnSync, so the test cannot observe progress, and its timeout cannot stop a hung child. Bun reported killing one dangling process, and the scratch dir stayed locked, which means a child (most likely the node probe) was still alive when the test gave up.
Expected
The packed sidecar initializes Loro and the probe exits well inside the budget on Windows. A probe that hangs is reported with its stage (pack / untar / import / checkSyncRuntime) and its output, and is killed before the scratch dir is removed, so it never leaks a locked temp tree.
Hypotheses to separate (each needs a measurement)
- The probe
nodenever exits on win32:checkSyncRuntime()or the sidecar's import leaves a handle open. That would be a real Windows bug in the shipped runtime. - A slow first wasm compile or cold
nodestart under shard load; timing only. - The
shell: truelaunch throughcmd.exewith cwd = drive root (ornpm packunder cmd) stalls.
Acceptance
- A measurement on a hosted Windows runner, outside the full shard, names which stage takes the time, and whether the packed runtime ever fails to exit.
- If it is a real runtime bug, it is fixed in the sidecar with a regression test. The test also gets per-stage async spawns with a kill-on-timeout and a reported stage, so a hang is diagnosable and never leaks the scratch dir.
Related
#9766 (temp-entry check), #10310 / #10814 (other Windows shard flakes; distinct cause).
Activity
measure: run under pwsh like CI (#10911)
code-yeongyu commentedon Oct 22, 2026
Reproduction outside the CI shard (run 39788672304): hosted windows-latest, 3 jobs, images 20261007.264.1 (2 jobs) and 20261018.275.1, node v24.21.0, npm 11.19.0,
tar=C:\Windows\system32\tar.exe(pwsh, as in CI).The packed runtime: 36 packs, 72 node probes, every one
sync-sidecar-ok, none hung.npm pack(shell)tar -xzfnode sync-probe.mjs,shell: true, cwd = drive root (the test's launch)node sync-probe.mjs, direct, cwd = packageimport("#omo-memory-sync-runtime")/checkSyncRuntime()No probe was still alive after it printed (the 5 s
STILL_ALIVEhandle dump never fired), so the packed sidecar does not hold the process open on Windows. The test itself, alone, 30 runs: 30 passed (16-40 s per run including the build fixture).Conclusion: no evidence of a Windows bug in the shipped memory-sync runtime. Loro initializes from the packed wasm in about 40 ms, and the process exits. The 60 s CI failure happened under full-shard load. Because every stage is a blocking
spawnSync, it left no record of which stage stalled. Bun later "killed 1 dangling process", and the scratch dir stayed locked (EBUSY).Test-side fix (next): run each stage with an async spawn and a per-stage cap. Report the stalled stage with its output, kill it, and wait for its exit before removing the scratch dir. That way a recurrence names its stage and never leaks a locked tree. One more portability finding: run under Git Bash's PATH,
taris GNU tar, which readsC:\...as a remote host and fails (exit 2). The test only passes because CI runs it from pwsh. The fix should pin the Windowstar(or avoid the externaltar).code-yeongyu commentedon Oct 22, 2026
Closed by #10917 (merged as ea4341e). The packed memory-sync runtime was measured healthy on hosted Windows: 72/72 probes, Loro init about 40 ms, the test alone 30/30 (run 39788672304). Each test stage (npm pack / tar / node probe) now runs with its own cap, and the test timeout is derived from those caps. A stall kills its process tree and fails naming the stage, and the probe no longer runs through a shell that could orphan a file-holding node. At the merged head CI was 23 pass, 5 skipped, 0 fail, including all Windows legs.
test(omo-senpi): bound and name each packed-sidecar stage, kill its t…
code-yeongyu commentedon Oct 22, 2026
Reopening: the test still fails on hosted Windows with #10917's fix present.
senpi-compatibility (windows-latest)119939407113 (6190 pass / 2 fail).git merge-base --is-ancestor ea4341e056 7ca0c35145exits 0, so test(omo-senpi): bound and name each packed-sidecar stage so a Windows stall is diagnosable and leak-free #10917 (merged as ea4341e) is in that tree.(fail) checkExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm [25468.29ms](fail) (unnamed) [2633.59ms], from the cleanup atpackages/omo-senpi/plugin/scripts/build-extension.test.mjs:412:EBUSY: resource busy or locked, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\omo-test-run-6BPOSY\omo-sync-sidecar-pack-Ln9d94'The bounded stages from #10917 did not stop the probe from overrunning, and did not stop the work directory from staying locked on this runner. Test-only: the shipped sidecar is unchanged.
code-yeongyu commentedon Oct 24, 2026
Seen again on omo#11126 (the senpi 2026.10.23-1 adoption for omo 5.1.45), job
senpi-compatibility (windows-latest): https://github.com/code-yeongyu/oh-my-openagent/actions/runs/40056938124/job/120766356446dev's own push run at the merge base (
1a62c298d2, run 40053513652) passed this test on the same leg; #11126 touches no memory-sync code. Release-train shipped omo 5.1.45 citing this issue for that failure (lead ruling), and onererun --failedof the leg is running as evidence. The Windows desktop app uses the packed sync sidecar, so the desktop pin waits on that rerun.code-yeongyu commentedon Oct 28, 2026
Seen again on dev at ae481df (CI run https://github.com/code-yeongyu/oh-my-openagent/actions/runs/40650695644, job 122707733786,
senpi-compatibility (windows-latest)): 7190 pass, 1 fail. The only failure ischeckExtensionCurrent > #given the sync sidecar packed with npm pack #when the installed package imports #omo-memory-sync-runtime under Node #then Loro initializes from the packed wasm[49085 ms], which matches this issue. The merge that triggered the run (#11535) only touches macOS-only Rust crates.code-yeongyu commentedon Oct 28, 2026
Owner: bugwatch (Sisyphus Labs), lane bw-win-10911, P1. Still red on dev Windows senpi-compatibility (run 40650695644). Plan: find which step eats the ~49 s and which handle keeps the temp dir locked, fix the root (an unawaited or unkilled child, an unclosed handle, caches under the temp dir), no raised timeouts, proven 10x under load; the full Windows matrix on the post-merge dev push is the final proof.
code-yeongyu commentedon Oct 28, 2026
Owner change: this lane (memsync-coord) owns #10911 from now. Plan, root cause first:
npm packstage printed its tarball name (code-yeongyu-omo-senpi-5.1.49.tgz), so the pack itself finished, yet the npm process did not exit within its 25 s cap. Find what npm does after writing the tarball on Windows that can take that long (lifecycle scripts, log or cache maintenance, a lingering child), and why the scratch directory stays locked after the tree kill.Fixes #10911, without raising any timeout.code-yeongyu commentedon Oct 28, 2026
Owner correction: the fix is #11648 (memsync-coord), so bugwatch has stood its lane down. One gap was flagged to memsync-coord: the stage tree kill (windowsTreeKillArgs) can miss the hung node, and #11648 changes only test-support files, so that needs its own fix or a follow-up issue.
code-yeongyu commentedon Oct 28, 2026
Follow-up for the gap this PR doesn't close: #11651. The shared
windowsTreeKillArgskills nothing once the root process is gone, and kills only the root when the process listing fails, so other callers can still orphan a hung descendant. This PR removes the shell for the npm pack stage. #11651 fixes the kill itself. Owner: memsync-coord.test(omo-senpi): resolve npm-cli.js past shims and spawn the node tha…
Merge pull request #11648 from code-yeongyu/fix/10911-npm-pack-exit
code-yeongyu commentedon Oct 28, 2026
#11648 merged into dev as 421f6d6.
process.exit()can hang on Node 24 (Hang at shutdown nodejs/node#64274), after npm has already printed the tarball name. This is measured on Node v24.22.1 locally. The Windows part (thecmd.exeworking-directory lock and the missed tree kill) is inferred.node --single-threaded npm-cli.js packfrom the filesystem root.The real proof is the next
senpi-compatibility (windows-latest)legs on dev. I'll check them. If this test stalls there again, this issue gets reopened with the job log.test(omo-senpi): run npm pack single-threaded, unshimmed and outside …
code-yeongyu commentedon Oct 29, 2026
Confirmed on dev after #11648 (421f6d6): the packed sync-sidecar test passed in 3 of 3 real
senpi-compatibility (windows-latest)runs. Nonpm pack did not finishline appeared in any of them.One cancelled run (40760561544) isn't counted. Follow-ups #11716 (Fixes #11651) and #11718 are still open.