Repository navigation
ci(windows): eliminate hidden failures in the non-blocking baseline #2624
Description
English
Problem
The Windows baseline currently reports an overall successful workflow while individual continue-on-error steps emit failure annotations. This makes the PR check green without giving us a clean Windows baseline.
PR #2613 exposed the current set in run 31323029613. The latest comparable main run, 31322689039, reproduces the deterministic failures.
Observed failures
1. Windows skip inventory is stale
Both the PR and main report:
Windows test skip inventory is stale; run `npm run windows:inventory:write`
The generated inventory must be refreshed and reviewed against the current tree.
2. Runtime PTY cleanup leaves SQLite locked
Both the PR and main run the two selected PTY tests successfully, then fail suite cleanup:
EBUSY: resource busy or locked, unlink '...\maka-shell-run-...\runtime.sqlite'
The test or production owner still holds the SQLite file when the temporary root is removed. Windows does not permit unlinking an open database.
3. Root-authority initialization race fixture resolves too early
Both the PR and main fail:
rejects replacement before opening the temporary marker
initialization race fixture reported resolved while waiting for marker_open_pending
This is a regression in an area previously tracked as completed by #2438. The fixture must preserve the intended cut on Windows instead of resolving before the marker-open observation.
4. Intermittent SQLite file-lock failures
The latest PR run also observed two failures that did not reproduce in the prior PR run or the compared main run:
EBUSY: resource busy or locked, unlink '...\workspaces\default\memory.sqlite'
EPERM: operation not permitted, open '...\memory.sqlite-shm'
The first occurred during Desktop real-window smoke cleanup. The second occurred in the concurrent Long-term Memory CAS test. Their one-run appearance suggests timing-sensitive ownership or cleanup rather than a deterministic semantic failure, but they still need repeated evidence before being classified as flakes.
Why this matters
The lane is intentionally non-blocking, but a successful outer job currently hides real failing steps. This prevents the baseline from showing whether Windows lifecycle correctness is improving or regressing.
The goal is not to mask Windows file locking with broad retry loops. Every Store, Host, and fixture should release its exact owner before temporary-root cleanup.
Desired outcome
-
npm run windows:inventoryis clean on the merge tree. - Runtime PTY tests close the SQLite owner before temporary-root removal.
- The root-authority race fixture reliably reaches
marker_open_pendingbefore replacement is evaluated. - Desktop smoke closes Runtime Host and Memory owners before deleting the fixture root.
- Concurrent Long-term Memory CAS does not race database sidecar permission setup.
- At least three consecutive
windows-latestbaseline runs complete without failure annotations for these groups. - The baseline summary makes every failed inner step visible even while the job remains non-blocking.
Suggested work split
- Refresh the generated Windows skip inventory.
- Fix deterministic owner/fixture lifecycle failures: Runtime PTY and root authority.
- Reproduce the Desktop and Long-term Memory file-lock failures under a repeated Windows stress gate.
- Keep the fixes at the ownership boundary; do not add catch-and-retry cleanup that can hide a live handle.
- Re-evaluate whether the lane can become blocking after the tracked baseline is clean.
Evidence and context
- Windows roadmap: roadmap(windows): make Windows a supported platform #2142
- Runtime Host PR: refactor(runtime-host): establish durable host architecture #2613
- Current PR baseline: run 31323029613
- Previous PR baseline: run 31322725976
- Comparable
mainbaseline: run 31322689039 - Earlier root-authority work: fix(storage): serialize root marker repair on Windows #2438
- Earlier inventory refresh: docs(windows): refresh test skip inventory #2434
简体中文
问题
Windows baseline 当前会在多个 continue-on-error step 产生 failure annotations，但整个 workflow 仍显示成功。因此 PR check 是绿色的，Windows baseline 本身却并不干净。
PR #2613 的 run 31323029613 暴露了当前问题。最新可比较的 main run 31322689039 可以复现其中的确定性失败。
已观察到的失败
1. Windows skip inventory 已过期
PR 与 main 都报告：
Windows test skip inventory is stale; run `npm run windows:inventory:write`
需要根据当前代码树重新生成并审查 inventory。
2. Runtime PTY cleanup 遗留被锁定的 SQLite 文件
PR 与 main 中两个目标 PTY tests 本身都成功，但 suite cleanup 随后失败：
EBUSY: resource busy or locked, unlink '...\maka-shell-run-...\runtime.sqlite'
删除临时 root 时，test 或 production owner 仍持有 SQLite 文件。Windows 不允许 unlink 已打开的数据库。
3. Root-authority initialization race fixture 过早结束
PR 与 main 都失败于：
rejects replacement before opening the temporary marker
initialization race fixture reported resolved while waiting for marker_open_pending
该领域此前在 #2438 中被标记为已完成，因此这是一个回归。Fixture 必须在 Windows 上保留预期的 execution cut，不能在观察到 marker open pending 之前结束。
4. 偶发 SQLite 文件锁失败
最新 PR run 还出现了两个没有在前一轮 PR run 或对照 main run 中复现的问题：
EBUSY: resource busy or locked, unlink '...\workspaces\default\memory.sqlite'
EPERM: operation not permitted, open '...\memory.sqlite-shm'
前者发生在 Desktop real-window smoke cleanup，后者发生在 concurrent Long-term Memory CAS test。它们只出现一次，更像 timing-sensitive ownership 或 cleanup 问题，而不是确定性的语义错误；但在获得重复证据前，也不能直接归类为 flake。
为什么需要处理
该 lane 目前有意保持 non-blocking，但成功的外层 job 会隐藏实际失败的 steps，导致 baseline 无法准确表达 Windows lifecycle correctness 是改善还是回退。
目标不是通过宽泛 retry loop 掩盖 Windows 文件锁。每个 Store、Host 与 fixture 都应在临时 root cleanup 前释放自己的准确 owner。
目标结果
- Merge tree 上的
npm run windows:inventory保持干净。 - Runtime PTY tests 在删除临时 root 前关闭 SQLite owner。
- Root-authority race fixture 能稳定到达
marker_open_pending，再判断 replacement。 - Desktop smoke 在删除 fixture root 前关闭 Runtime Host 与 Memory owners。
- Concurrent Long-term Memory CAS 不再与 database sidecar permission setup 竞争。
- 至少连续三次
windows-latestbaseline 不再出现这些 failure annotations。 - Baseline summary 在 job 仍为 non-blocking 时也明确显示每个失败的 inner step。
建议拆分
- 刷新生成的 Windows skip inventory。
- 修复确定性的 owner/fixture lifecycle 问题：Runtime PTY 与 root authority。
- 在重复 Windows stress gate 下复现 Desktop 与 Long-term Memory 文件锁问题。
- 在 ownership boundary 修复问题；不要增加可能掩盖 live handle 的 catch-and-retry cleanup。
- Baseline 清零后，再评估是否将该 lane 改为 blocking。
证据与上下文
- Windows roadmap：roadmap(windows): make Windows a supported platform #2142
- Runtime Host PR：refactor(runtime-host): establish durable host architecture #2613
- 当前 PR baseline：run 31323029613
- 前一轮 PR baseline：run 31322725976
- 可比较的
mainbaseline：run 31322689039 - 之前的 root-authority 工作：fix(storage): serialize root marker repair on Windows #2438
- 之前的 inventory refresh：docs(windows): refresh test skip inventory #2434
Activity
bugkiller-damo commentedon Aug 14, 2026
Local Windows reproduction of
shell-run-managerfailuresI ran the
@maka/runtimeshell-run-manager tests locally on Windows and found a stable set of failures that seem related to the Windows baseline cleanup problems tracked here.Environment
main@2f76c3109Command
cd packages/runtime node --test dist/__tests__/shell-run-manager.test.jsSummary
Failures observed in three consecutive runs
keeps the default pipe path separated, durable, redacted, and observedexpected 'completed', got 'failed'uses the shared explicit PowerShell pipe planspawn /bin/echo ENOENT/bin/echodoes not exist on Windowspreserves the caller environment through a PowerShell PTYexpected 'completed', got 'failed'publishes raw PTY deltas and exposes a bounded replay snapshotTimed out waiting for ShellRun statejoins finalization when a real PTY exits before a queued control cutexpected 'completed', got 'running'fails closed before terminal protocol replies can form an unbounded native write queueTimed out waiting for ShellRun stateEBUSY: resource busy or locked, unlink '...\maka-shell-run-XXXX\runtime.sqlite'Additional note
The
#2978test case (redacts a secret across a soft wrap and the scrollback/screen boundary) passed in all three runs, so its flakiness appears separate from these stable Windows failures.The
/bin/echohard-coding looks like the easiest deterministic fix; the PTY state/timeouts and the SQLite cleanup issue likely need ownership-boundary fixes as described in this issue.Let me know if more local reproduction logs would help.
简体中文：
我在 Windows 本地跑了
@maka/runtime的shell-run-manager测试，发现一组稳定失败。其中顶层 hook 的EBUSY: runtime.sqlite与本 issue 的第 2 项观察完全对应；此外还有 6 个测试用例失败，包括一个硬编码/bin/echo导致的ENOENT。#2978的 secret redaction 用例在三次运行中均通过，因此它的 flaky 问题与这些稳定失败无关。需要的话我可以提供更多日志。kyletser commentedon Aug 23, 2026
I'm working on the deterministic Runtime PTY cleanup slice (item 2), scoped to releasing the exact SQLite owner before the temporary root is removed on Windows. I'll reproduce it against current main, add regression coverage for the ownership boundary, and keep the PR separate from the inventory and root-authority fixture work so the issue's proposed split remains reviewable.
liugddx commentedon Aug 25, 2026
Progress update (2026-08-25)
#3789 now advances the authoritative Windows baseline/required-recovery slice from current
main:windows_recoveryon every PR andmainpush;windows_recoveryas a required context in.asf.yamland protects the workflow/config contract;marker_open_pendingreliably;fsutil file queryStreamswith a bounded, structured, long-path-safe Windows PowerShell ADS query and tightens the named-stream test;Local Windows / Node 24 evidence:
The latest full local storage run is 903 pass / 5 fail / 44 skip. Three failures are local symlink privilege gaps; one Git cleanup EBUSY remains outside #3789. The fifth newer-schema EBUSY is fixed by #3789.
Current-main blockers discovered while validating the new automatic recovery check:
queryTurn/stopTurn/startTurnaliases; current-main CI and fix(windows): make recovery evidence authoritative #3789windows_recoveryfail during build before recovery tests run.Runtime PTY cleanup remains assigned to the contributor who claimed it above, so #3789 does not duplicate that work. After #3788/#3790 restore a green main and #3789 passes exact-head Windows recovery, the remaining issue exit criteria are the Git/PTY ownership slices, symlink-capability inventory, and three consecutive clean Windows baseline runs.
liugddx commentedon Aug 25, 2026
Bootstrap repair update: #3796 is now exact-head green at \cfda25912\ (required test, 7m55s). It converges every known version-30 session metadata shape to schema 31, closes failed schema-open handles, and migrates the three stale Runtime Host queue test aliases. This clears #3788/#3790 once reviewed and merged; #3789 can then rebase and rerun automatic \windows_recovery\ through the real recovery steps.
kyletser commentedon Aug 25, 2026
Runtime PTY cleanup update: the slice I claimed above was completed and merged in #3644. It tracks every SQLite ShellRun store opened by the suite and closes those leases before removing the temporary roots.
I revalidated current main at c4971c9 on Windows with Node 24.14.1:
The deterministic Runtime PTY SQLite cleanup slice is therefore complete. The remaining full-file Windows PTY behavior failures are separate slices.
liugddx commentedon Aug 25, 2026
#3789 exact-head closure: rebased head \9047c2eb3\ passed hosted test in 17m28s and windows_recovery in 3m23s. The native lane completed Local IPC, SQLite crash, Runtime continuation, Runtime Host owner-death, and managed workspace crash recovery. The required-context workflow is therefore reportable on every PR/main head; remaining work in this issue is Runtime PTY/Git ownership cleanup, symlink-capability inventory, and three consecutive clean scheduled baselines after merge.
liugddx commentedon Aug 26, 2026
Post-#3789 baseline closure kickoff
#3789 merged as
32a1db0eb; its merge-commitwindows_recoveryrun is green and now includes the strict hosted Windows ADS evidence (3 tests / 3 pass / 0 skip).The remaining #2624 exit criterion is a current full Windows baseline, including inner-step outcomes rather than the non-blocking outer job conclusion, plus three consecutive clean runs.
I attempted to dispatch
windows-baseline.ymlwithfull_storage=trueinapache/maka, but GitHub returned HTTP 403 because this account lacks repository Actions-admin permission. Please trigger three full-storage runs on currentmain, or confirm that the next three scheduled runs should be used as authoritative evidence.To keep technical work moving, I started the same active workflow in my fork on the byte-identical Apache main commit:
apache/main@32a1db0ebci/windows-baseline-main-32a1db0ebFork runs are diagnostic evidence only; they do not replace Apache-owned scheduled/admin-triggered evidence. I will inspect every inner step and artifact, split any deterministic residual by root cause, and post the three-run matrix here.
liugddx commentedon Aug 26, 2026
Baseline round 1/3 result and focused fix
Fork run 32936082313 executed
windows-baseline.ymlwithfull_storage=trueon byte-identicalapache/main@32a1db0eb.Inner-step matrix:
[]);Both Storage executions failed only
rejects replacement before opening the temporary marker: the fixture reportedresolvedbeforemarker_open_pending. The run is therefore not a clean baseline despite the non-blocking outer job reporting success.Focused fix PR: #3872.
windows_recoverywith strict 1/1/0 evidence and repository-control coverage.Rounds 2/3 are paused until #3872 reaches green/final disposition; repeating the known-red baseline would not add evidence.
liugddx commentedon Aug 26, 2026
Three clean baseline runs after #3872
The deterministic root-marker race was fixed in #3872, and the same full baseline workflow was run three times on the exact branch head
6dae318b97c0a758a715539b50e490e39b962490:[][][]The first post-#3789 run had one deterministic root-race failure in both focused and full Storage. After #3872, the root race is also a required
windows_recoverystep with strict1 test / 1 pass / 0 skipevidence.The runs above are fork diagnostics on byte-identical
apache/main@32a1db0eb, not Apache-owned scheduled runs. I attempted upstream manual dispatch and GitHub returned HTTP 403 because this account lacks Actions-admin permission. ASF/PPMC should now trigger or authorize three equivalent runs onapache/main; the fork matrix establishes the technical result but does not replace organization-owned evidence.Remaining #2624 administration action: apply and verify
windows_recoveryas an effective required check in the GitHub ruleset. Once that is confirmed and the three Apache-owned runs are clean, this baseline item can be closed.liugddx commentedon Aug 26, 2026
#3872 is now merged as part of main@235a12d77; its post-merge windows_recovery run is green. I am refreshing the three-run full baseline evidence on this latest Apache main commit before closing the remaining baseline item.
liugddx commentedon Aug 26, 2026
Post-#3872 status
#3872 is merged as
235a12d77. Its post-mergewindows_recoveryrun is green and directly executes the root initialization race gate.On the byte-identical
apache/main@235a12d77, three fullwindows-baseline.ymlruns withfull_storage=trueare clean:Each run reports focused Storage
88 pass / 0 fail / 12 skip, full Storage950 pass / 0 fail / 38 skip, successful install/build/smoke, PTY, PowerShell UTF-8, storage path/lock and residual-process steps ([]).The technical baseline work is therefore complete. These three runs were executed in the maintainer fork because this account cannot dispatch the Apache workflow (HTTP 403, Actions-admin permission required). Remaining acceptance actions are administrative:
apache/mainand records the inner-step/artifact results.windows_recoveryas an effective required check in the GitHub ruleset.Once those organization-owned checks are recorded, #2624 can be closed. Authenticode #3414 remains the separate Phase 3 blocker for changing Windows from Preview to Supported.