Skip to content

ci: run extension host tests on Windows in a merge-only job - #16619

Merged
jonvanausdeln merged 8 commits into
mainfrom
jonv/windows-ext-host-tests-a096f3
Oct 15, 2026
Merged

jonvanausdeln merged 8 commits into
mainfrom
jonv/windows-ext-host-tests-a096f3

Conversation

@jonvanausdeln

@jonvanausdeln jonvanausdeln commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Moves Windows extension host tests into their own merge-only workflow job, and fixes what that turned up on Windows. Fixes #8853.

Today the only Windows ext-host coverage is positron-r, run on shard 1 of the Windows e2e job (added in #8816). That step is soft-fail: continue-on-error, always exit 0. Its Slack alert in test-merge.yml reads needs.e2e-windows-electron.outputs.extensions_failed, but #11014 renamed that job to e2e-windows, so the alert has never fired since. Meanwhile positron-r is failing 4 tests on Windows on main (e.g. merge run 38429410345, shard 1: 204 passing, 4 failing).

Workflow changes

  • New test-ext-host-windows.yml, called from test-merge.yml and test-full-suite.yml (new run_ext_host_tests_windows input). It is not called from the PR workflow. It runs the Electron driver via scripts\test-integration.bat and fails the job when tests fail.
    • The web (Chromium) driver is not run: Positron has no web build on Windows.
    • The Remote driver is not run: Windows remote (WSL) is covered by e2e tests.
    • Restores the Windows build caches but does not save them; the e2e-windows job in the same merge run owns the save.
    • Sets VSCODE_SKIP_PRELAUNCH=1 on the test step, as upstream's Windows CI does. Prelaunch already ran, and otherwise the agent host E2E runner calls spawnSync('npm.cmd') without shell: true, which Node rejects with EINVAL.
  • test-e2e-windows-run.yml: removes the positron-r step, its log save/upload steps, and the extensions_failed output. Removes the dead slack-notify-win-extensions job from test-merge.yml. PRs tagged @:win no longer run positron-r ext-host tests on Windows; that coverage moves to merge.
  • Insights filter_jobs regex in both callers now matches ext-host-win.
  • test-full-suite.yml is now at the 25-input limit for workflow_dispatch.

scripts/test-integration.bat

  • Adds the 13 Positron extension suites that test-integration.sh runs, and skips colorize as the .sh does (Test: fix MS colorize integration tests #7349). The Positron suites are skipped when --suite is given, since they are not selectable by name.
  • Resolves TMP/TEMP to the long path, the same line upstream already has in test-remote-integration.bat and test-web-integration.bat. On CI TMP is C:\Users\RUNNER~1\..., and under Node 24.16+ fs.watch on an 8.3 short-name path drops every event (Potential regression with libuv in Node 24.16.0 nodejs/node#63638; plain Node asserts in libuv fs-event.c).
  • Runs the Positron suites with --config pointing at .vscode-test.js through a lowercase drive letter. The extension host loads an extension through c:\, while vscode-test required the test files through C:\. Node keys its module cache by path, so a test importing extension modules got a second, never-initialized copy (confirmed with a probe: both require.cache keys present while the extension was active). This is what broke positron-code-cells' context and decoration tests and positron-r's RSessionManager.

Product bugs found (Windows only)

  • next-edit-suggestions: exact-filename keys in nextEditSuggestions.enabled (e.g. "Dockerfile": false) never matched on Windows, because the basename was taken from document.fileName at the last / only. It now comes from path.posix.basename(document.uri.path), as isFileExcludedFromAI in the same file already does, with a Windows-path test so Linux CI covers it.
  • positron-data-driver-duckdb: on Windows DuckDB reports a lock conflict as a sharing violation (... File is already open in <process> (PID n)), so users got the raw error instead of the "another session has it locked" explanation. The pattern now also matches DuckDB's own File is already open in text (the OS part of the message is localized).

Test fixes

  • positron-r: rootSignature builds expectations with path.join (faking process.platform does not change path); session-dispose stubs RSessionManager.instance instead of depending on activation order, which also keeps its fake sessions out of the live manager.
  • authentication: snowflake path expectation uses path.join. Tests that start the provider catalog watcher remove their temp dir with an async fs.promises.rm retry: Windows releases a closed watcher's directory handle on a later event loop turn, and rmSync's own retries block the loop.
  • positron-data-driver-odbc: resolveUnixConfigPaths uses path.posix. The module takes the platform from an injected host and already uses path.posix/path.win32 explicitly elsewhere; this was the one spot using the host path. No change on Unix.
  • CSS / HTML language server tests (upstream files, Positron markers): their entry points globbed path.posix.join(testRoot, ...); on Windows testRoot has backslashes, which glob reads as escapes, so both suites ran 0 tests and passed. Now glob relative to testRoot, and fail if no test files match. Windows runs the same 9 CSS / 71 HTML tests as Linux. Upstream has the same code.
  • positron-data-driver-sqlite / duckdb: teardowns remove temp dirs with an async retry. disconnect() kills the worker process without waiting for it to exit, so the database file is briefly still open, and Windows will not delete an open file. (Making disconnect() await worker exit would be the product-side fix; left out of scope here.)

Unrelated

Release Notes

New Features

  • N/A

Bug Fixes

  • Next Edit Suggestions: per-filename entries in nextEditSuggestions.enabled now apply on Windows.
  • DuckDB data connections: opening a database another session has locked now explains the conflict on Windows too.

Validation Steps

  • Locally on Windows, all 13 Positron suites run the way the .bat now runs them: 12 pass. positron-r's R-session tests could not start an R kernel from this machine's agent-launched shell (a known local environment issue with Ark's ZeroMQ poller); main's Windows e2e job shows those tests starting R sessions fine on the CI runner.
  • The originally failing tests were reproduced locally first and fail without the fixes.
  • Full suite runs on this branch with only run_ext_host_tests_windows:
    • Run 1 (38450231603): failed before any tests, on the agent host E2E npm.cmd EINVAL.
    • Run 2 (38455222586): node.js integration, agent host E2E, and every upstream ext-host suite passed on Windows; stopped at authentication.
    • Run 3 (38466440126): green. Every suite passed, including positron-r (208 passing, with R sessions on the CI runner). CSS/HTML reported 0 tests, which led to the fix above.
    • Run 4 (38473148320): green on 05908324ee, now including 9 CSS / 71 HTML server tests. Review follow-ups after it (7dcca78d53: empty-glob guard, NES basename, job timeout 150, no id-token) were verified locally.
    • Run 5 (38478932934): green on 7dcca78d53 (the PR head). All 33 sections of test-integration.bat ran with no failures; the job took about 1 hour.

Adds test-ext-host-windows.yml, called from test-merge.yml and
test-full-suite.yml (not the PR workflow). It runs the Electron ext-host
driver through test-integration.bat; the web driver is skipped (no web
build on Windows) and so is Remote (WSL remote is covered by e2e).

Previously the only Windows ext-host coverage was positron-r on shard 1
of the Windows e2e job, as a soft-fail whose Slack alert referenced a
renamed job and never fired. It was failing 4 tests on main. That step
and the dead alert are removed.

- test-integration.bat: add the Positron extension suites run by
  test-integration.sh, and skip colorize as the .sh does (#7349).
- positron-r rootSignature test: build expectations with path.join, since
  faking process.platform does not change the path module.
- positron-r session-dispose test: stub RSessionManager.instance. On
  Windows the extension host loads the extension via a c:\ path and the
  tests via C:\, so they get separate module instances and the test's
  copy is never initialized.

Closes #8853
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Tests 🚀
This PR will run tests tagged with: @:critical @:win @:assistant @:connections

Why these tags?
Tag Source
@:critical Always runs (required)
@:win PR description
@:assistant Changed files
@:connections Changed files

More on automatic tags from changed files.

readme  valid tags

The agent host E2E runner spawns npm.cmd without shell: true when
prelaunch is not skipped, which Node rejects with EINVAL on Windows.
Prelaunch already ran in an earlier step, and upstream's Windows CI sets
VSCODE_SKIP_PRELAUNCH=1 on its test steps the same way.
- test-integration.bat: resolve TMP/TEMP to the long path, as upstream's
  test-remote-integration.bat and test-web-integration.bat already do.
  On CI TMP is C:\Users\RUNNER~1\..., and under Node 24.16+ fs.watch on an
  8.3 short-name path drops every event (nodejs/node#63638), so the
  providerCatalog watcher test never saw its file edit.
- authentication tests: remove watched temp dirs with an async
  fs.promises.rm retry. Windows releases a closed watcher's directory
  handle on a later event loop turn, so an immediate rmSync fails with
  ENOTEMPTY/EPERM, and rmSync's own retries block the loop.
- snowflake test: build the expected path with path.join.
- test-integration.bat: pass vscode-test a lowercase-drive config path.
  The extension host loads extensions through c:\ and vscode-test
  required the tests through C:\, so Node loaded each module twice and
  tests reading extension state (positron-code-cells contexts and
  decorations) saw a never-initialized copy.
- next-edit-suggestions: exact-filename keys in nextEditSuggestions.enabled
  never matched on Windows, because the basename was taken at the last
  '/' only. Split on either separator; add a Windows-path test.
- positron-data-driver-duckdb: recognize DuckDB's Windows lock conflict
  ("File is already open in ...") so it gets the explanatory message.
- positron-data-driver-odbc: resolve unixODBC config paths with path.posix,
  since the platform is injected and these are always unix paths.
- sqlite/duckdb driver tests: remove temp dirs with an async retry; the
  worker process still has the database open just after disconnect, and
  Windows will not delete an open file.
- positron-r session-dispose test: reword the stub's rationale.
Their test entry points globbed path.posix.join(testRoot, ...). On
Windows testRoot has backslashes, which glob reads as escapes, so no
files matched and both suites passed with 0 tests. Glob relative to
testRoot instead. Windows now runs the same 9 CSS and 71 HTML tests as
Linux.
- css/html server tests: fail when the glob matches no test files, so a
  pattern regression cannot pass silently with 0 tests again.
- next-edit-suggestions: take the basename from document.uri.path with
  path.posix, like isFileExcludedFromAI in the same file, instead of
  splitting fileName on both separators.
- test-ext-host-windows.yml: timeout 150 minutes (as in the Windows e2e
  workflow) so it covers the npm install retry; drop id-token: write,
  since no step uses OIDC.
@jonvanausdeln
jonvanausdeln marked this pull request as ready for review October 14, 2026 16:06
@jonvanausdeln
jonvanausdeln requested a review from lionel- October 14, 2026 17:00
@jonvanausdeln
jonvanausdeln merged commit 0849854 into main Oct 15, 2026
67 of 69 checks passed
@jonvanausdeln
jonvanausdeln deleted the jonv/windows-ext-host-tests-a096f3 branch October 15, 2026 18:23
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move windows integration tests into separate yml and workflow job

3 participants