Skip to content

Commit b78a59b

Browse files
authored
ci: test on Windows, keep coverage and the extension zip, and check the extension panel (#468)
- A Windows job runs the package tests; scans now report file names with forward slashes on every platform. - A coverage job keeps v8 coverage of the package and panel tests. - The build job keeps the extension zip as an artifact. - axe runs on the extension panel page in every connection state. - A headless harness smoke-tests the NativeScript, Angular Native and Capacitor overlays. - A weekly e2e loads the unpacked extension and opens its DevTools panel. Refs #371, #373
1 parent cb0b9c1 commit b78a59b

32 files changed

Lines changed: 1184 additions & 102 deletions

‎.github/workflows/ci.yml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,20 @@ jobs:
6161
exit 1
6262
fi
6363
64+
# The zip step of `pnpm extension:zip`, on the bundle built above. Without
65+
# PANGULAR_EXTENSION_KEY it holds no key.pem, so the artifact is only for loading and
66+
# checking by hand: nothing here uploads it to the Chrome Web Store.
67+
- name: Zip the Chrome extension
68+
run: node scripts/extension-zip.mjs
69+
70+
- name: Keep the extension zip
71+
uses: actions/upload-artifact@v4
72+
with:
73+
name: chrome-extension
74+
path: dist/pangular-inspector-extension.zip
75+
if-no-files-found: error
76+
retention-days: 14
77+
6478
- name: Smoke-test the CLI
6579
run: node bin.mjs --help
6680

@@ -81,6 +95,61 @@ jobs:
8195
- name: Test devtools package
8296
run: pnpm test:devtools
8397

98+
coverage:
99+
name: Test coverage
100+
# A report to read, not a gate: there is no threshold, and the build job is what fails a pull
101+
# request on a broken test. v8 instrumentation slows every test, so the run gives each one
102+
# 20 seconds instead of the default 5, which the slowest agent-tool tests went over.
103+
runs-on: ubuntu-latest
104+
timeout-minutes: 25
105+
continue-on-error: true
106+
steps:
107+
- uses: actions/checkout@v7
108+
with:
109+
persist-credentials: false
110+
111+
- uses: ./.github/actions/setup
112+
113+
- name: Package tests with coverage
114+
run: pnpm test:devtools --coverage --testTimeout=20000
115+
116+
- name: Panel tests with coverage
117+
if: ${{ !cancelled() }}
118+
run: pnpm test:panel --coverage --testTimeout=20000
119+
120+
# HTML under devtools/ and panel/, plus coverage-summary.json and lcov.info for each.
121+
- name: Keep the coverage report
122+
if: ${{ !cancelled() }}
123+
uses: actions/upload-artifact@v4
124+
with:
125+
name: coverage
126+
path: dist/coverage
127+
if-no-files-found: ignore
128+
retention-days: 14
129+
130+
windows:
131+
name: Package tests on Windows
132+
# The CLI, the hub and the source scans run on developers' Windows machines too, where paths
133+
# use `\` and a drive letter.
134+
runs-on: windows-latest
135+
timeout-minutes: 30
136+
defaults:
137+
run:
138+
shell: bash
139+
env:
140+
# pnpm runs package scripts with cmd.exe on Windows, and the root `prepare` script is
141+
# written for a POSIX shell. Git Bash, not the WSL `bash` that comes first on PATH.
142+
npm_config_script_shell: C:\Program Files\Git\bin\bash.exe
143+
steps:
144+
- uses: actions/checkout@v7
145+
with:
146+
persist-credentials: false
147+
148+
- uses: ./.github/actions/setup
149+
150+
- name: Test devtools package
151+
run: pnpm test:devtools
152+
84153
axe:
85154
runs-on: ubuntu-latest
86155
timeout-minutes: 15
@@ -97,6 +166,12 @@ jobs:
97166
- name: Check the panel with axe
98167
run: pnpm test:axe
99168

169+
# The committed extension/ui in extension/panel.html, reading the report test:axe wrote,
170+
# and every status the panel bridge shows before it connects.
171+
- name: Check the extension panel with axe
172+
if: ${{ !cancelled() }}
173+
run: pnpm test:axe:extension
174+
100175
# The e2e jobs below are slow, so they run only when the package, the panel, the SSR demo, the
101176
# tools docs (e2e:mcp reads them) or the scripts and fixtures they use change. The engines-floor
102177
# job runs only when what the packed package is built from changes. A diff that cannot be

‎.github/workflows/e2e.yml‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
name: E2E
22

3-
# The slow end-to-end checks that do not fit a pull request: every panel view in a real browser,
4-
# multi-tab and lifecycle resilience, and the install matrix from a packed tarball. Pull requests
3+
# The slow end-to-end checks that do not fit a pull request: every panel view and the Chrome
4+
# extension panel in a real browser, multi-tab and lifecycle resilience, and the install matrix
5+
# from a packed tarball. Pull requests
56
# run the security and MCP checks (ci.yml), and a release can run a slice of the install matrix on
67
# its tarball. See apps/docs/src/content/contributing/development.md.
78
on:
@@ -54,12 +55,20 @@ jobs:
5455
- name: Check every panel view
5556
run: pnpm e2e:panel --base http://localhost:4000 --out dist/e2e-panel
5657

58+
# The unpacked extension in headless Chromium, with DevTools open on the demo: the panel
59+
# finds the server, loads extension/ui for the page and lists the App component.
60+
- name: Check the Chrome extension panel
61+
if: ${{ !cancelled() }}
62+
run: pnpm e2e:extension --base http://localhost:4000 --out dist/e2e-extension
63+
5764
- name: Keep the screenshots and results
5865
if: always()
5966
uses: actions/upload-artifact@v4
6067
with:
6168
name: e2e-panel
62-
path: dist/e2e-panel
69+
path: |
70+
dist/e2e-panel
71+
dist/e2e-extension
6372
if-no-files-found: ignore
6473

6574
resilience:

‎app/vitest.config.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,5 +21,14 @@ export default defineConfig({
2121
include: ['src/**/*.test.ts'],
2222
setupFiles: ['src/test-setup.ts'],
2323
execArgv: ['--no-experimental-webstorage'],
24+
// Used only with `--coverage` (`pnpm test:coverage`): a report, with no threshold to meet.
25+
coverage: {
26+
provider: 'v8',
27+
include: ['src/**/*.ts'],
28+
exclude: ['src/**/__tests__/**', 'src/**/*.test.ts', 'src/test-setup.ts'],
29+
reporter: ['text-summary', 'html', 'json-summary', 'lcov'],
30+
reportOnFailure: true,
31+
reportsDirectory: '../dist/coverage/panel',
32+
},
2433
},
2534
});

‎apps/docs/src/content/contributing/chrome-extension.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ This builds the devtools UI (`pnpm devtools:build`), then replaces `extension/ui
5353
<code>extension/ui</code> is committed. If you change <code>app/</code>, run <code>pnpm extension:build</code> and commit the result. CI builds the extension and fails when <code>extension/ui</code> is stale.
5454
</ngmd-callout>
5555

56+
If you change `panel.html`, `panel-bridge.js` or a status message, run `pnpm test:axe`, then `pnpm test:axe:extension`. It loads `panel.html` with a stand-in for the `chrome` APIs and runs axe on each status the panel shows and on the connected panel. See [Run the checks](./development.md#run-the-checks).
57+
5658
## Load in Chrome
5759

5860
<ngmd-workflow>
@@ -80,6 +82,8 @@ pnpm extension:zip
8082

8183
This runs `extension:build`, then writes `dist/pangular-inspector-extension.zip`. The zip leaves out `.DS_Store` files and drops `key` from the manifest, because the Chrome Web Store refuses a manifest with a `key`.
8284

85+
CI builds the same zip, without a key, on every pull request and push to `main`. Download it from the run's `chrome-extension` artifact to try a branch without building it.
86+
8387
For the first upload of a new store item, give the script the private key so the store keeps the ID `dcogniffeelebaolkkfbopmjcblhblfk`:
8488

8589
```bash

‎apps/docs/src/content/contributing/development.md‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -151,15 +151,23 @@ pnpm exec nx affected -t test build # Test and build affected projects
151151
pnpm test:devtools # Devtools package tests (Vitest)
152152
pnpm test:panel # Devtools UI tests (Vitest)
153153
pnpm test:axe # axe check of every panel page (needs Chromium)
154+
pnpm test:axe:extension # axe check of the extension panel (after pnpm test:axe)
155+
pnpm test:coverage # Package and panel tests with v8 coverage, into dist/coverage
154156
pnpm extension:build # Chrome extension
155157
pnpm skills:check # Agent skills and roles in .claude/
156158
pnpm commit:check # Commit messages on your branch
157159
```
158160

159161
`pnpm typecheck` runs `ngc` on `app/tsconfig.json` and `examples/analog/tsconfig.app.json`, so template errors fail it. `app/tsconfig.json` turns on `strictTemplates`.
160162

163+
The mobile overlays (NativeScript, Angular Native and the Capacitor collector) have no browser or device in the package tests. `packages/devtools/src/__tests__/native-harness.ts` stands in for one in plain Node.js: a fake `connectDevframe` that records each session's calls, the globals of a native runtime (a `WebSocket`, a `fetch` for `__connection.json` and no DOM), NativeScript views, Angular Native nodes and a Capacitor WebView with its bridge. `native-overlays-smoke.test.ts` starts each overlay in it and checks the first report and the clean-up. Point a test file's `devframe/client` mock at the harness with `vi.mock('devframe/client', async () => (await import('./native-harness.ts')).devframeModule())`.
164+
161165
`pnpm test:panel` runs the tests in `app/src/__tests__` in jsdom, with the Analog Angular plugin compiling the components. `pnpm test:axe` builds the package, writes a static report of Angular Travel to `dist/panel-axe`, serves it, and runs axe on every tab and on each hub view (`?view=ngrx`, `analog`, `nativescript`, `capacitor`) in both dark and light color schemes. It also checks each tab's states: a filter that matches nothing in every search box, a project whose source scans come back empty, a server that answers every call with an error, and a server that has not answered yet. Seeded pages stand in for a live app on the Capacitor, Errors, Plugins, Queries and Forms tabs, and for long lists on Components and **SSR & HTTP**. It fails on any violation, page error or state that does not show. Run `pnpm exec playwright install chromium` once before the first run.
162166

167+
`pnpm test:axe:extension` checks the Chrome extension's own page. It serves `extension/` over http with a stand-in for the `chrome` APIs, opens `panel.html` and runs axe on every status the panel bridge shows before it connects: a page that doesn't answer or can't be read, a page not served over http, a host the extension may not reach yet, no devtools server and a server that refuses. Then it serves the report from `pnpm test:axe` on the same origin and checks the connected panel, the committed `extension/ui` in the panel's frame. Both color schemes, at 1280 and 360 px. Run `pnpm test:axe` first, or pass another report folder to `node scripts/extension-axe.mjs`.
168+
169+
`pnpm test:coverage` runs `pnpm test:devtools --coverage`, then `pnpm test:panel --coverage`, each with `--testTimeout=20000`: v8 instrumentation slows every test, and the slowest ones went over the default 5 seconds. The v8 provider writes an HTML report, `coverage-summary.json` and `lcov.info` to `dist/coverage/devtools` and `dist/coverage/panel`, also when a test fails. There is no threshold: the report shows what the tests reach and nothing fails on a low number. Open `dist/coverage/devtools/index.html` to read it.
170+
163171
`pnpm skills:check` validates the frontmatter of every skill and role and checks that the files and links they mention exist. `pnpm commit:check` checks every commit on your branch that is not on `main` (it compares with `upstream/main`, then `origin/main`, then `main`).
164172

165173
### What CI runs
@@ -182,12 +190,15 @@ pnpm commit:check # Commit messages on your branch
182190
<ngmd-step title="Build the extension and check it is committed">
183191
<code>pnpm extension:build</code>. The job fails when <code>extension/ui</code> differs from the committed copy.
184192
</ngmd-step>
193+
<ngmd-step title="Zip the extension">
194+
The zip step of <code>pnpm extension:zip</code>, kept for 14 days as the <code>chrome-extension</code> artifact of the run. It has no <code>key.pem</code> and nothing uploads it to the Chrome Web Store.
195+
</ngmd-step>
185196
<ngmd-step title="Smoke-test the CLI">
186197
<code>node bin.mjs --help</code>.
187198
</ngmd-step>
188199
</ngmd-workflow>
189200

190-
A separate `axe` job in the same workflow installs Chromium and runs `pnpm test:axe`. A `node-22` job runs `pnpm test:devtools` on the newest Node.js 22, the oldest major `engines.node` (`>=22.12.0`) allows.
201+
A separate `axe` job in the same workflow installs Chromium and runs `pnpm test:axe`, then `pnpm test:axe:extension`. A `coverage` job (**Test coverage**) runs both test suites with `--coverage` and keeps `dist/coverage` as the `coverage` artifact for 14 days. It is marked `continue-on-error`, so it never blocks a pull request: the build job is the test gate. A `node-22` job runs `pnpm test:devtools` on the newest Node.js 22, the oldest major `engines.node` (`>=22.12.0`) allows. A `windows` job (**Package tests on Windows**) runs `pnpm test:devtools` on `windows-latest`, so a path built with `/` or a file name reported with `\` fails there. It runs package scripts with Git Bash, because pnpm uses `cmd.exe` on Windows and the root `prepare` script needs a POSIX shell. In a test, build expected paths with `join()` or `resolve()` from `node:path`, and in a source scan, report file names with `relativeFile()` from `packages/devtools/src/rpc/source-scan.ts`.
191202

192203
The source tests can't run on Node.js 22.12 itself: Node.js runs `.ts` files only from 22.18, and the build tools need newer versions too. So when a pull request or push changes `packages/devtools`, `app`, the root manifests, the lockfile, `.nvmrc` or the CI setup, the `engines-floor` job (**Packed package on Node.js 22.12**) packs the package on the `.nvmrc` version, switches to Node.js 22.12.0 and installs the tarball into an empty project with npm. It then runs `npx pangular --version`, imports the `cli`, `hub`, `ssr` and `vite` entry points, and loads the `hub` entry with `require()`. When `engines.node` changes, change the version in that job too.
193204

@@ -197,13 +208,16 @@ Two more jobs run when a pull request or push changes `packages/devtools`, `app`
197208

198209
The `e2e:*` scripts drive real apps in Playwright's Chromium. Run `pnpm exec playwright install chromium` once first.
199210

200-
| Script | What it checks | Needs | Runs |
201-
| --------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
202-
| `pnpm e2e:security` | Secrets planted in a fixture app never reach the panel, agents, logs or the static report; the access gates hold. | `pnpm devtools:build-pkg`. Ports 5231 to 5236. | Pull requests |
203-
| `pnpm e2e:mcp` | Every agent tool, resource and prompt over HTTP and stdio, with bad arguments and config variants. | Builds what it needs. Ports 5150 to 5154. | Pull requests |
204-
| `pnpm e2e:panel` | Every panel view in both themes at 1280 and 360 px: axe, overflow and page errors. | A running app, such as the SSR demo on 4000 (`--base`) started with `PANGULAR_AUTH=false`, or `--auth-log` with its log. | Weekly |
205-
| `pnpm e2e:resilience` | `multitab`, `expiry`, `lifecycle` and `stress`: tabs that close, crash or hide, server restarts and a large app. | `pnpm devtools:build-pkg` and the development build. Port 5401. | Weekly |
206-
| `pnpm e2e:install` | A packed tarball installed into fresh apps per setup, package manager and Angular version, plus the package audit. | Network for npm. A local registry on 4873 and the ports after it. | Weekly, and a few cells on every release |
211+
| Script | What it checks | Needs | Runs |
212+
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
213+
| `pnpm e2e:security` | Secrets planted in a fixture app never reach the panel, agents, logs or the static report; the access gates hold. | `pnpm devtools:build-pkg`. Ports 5231 to 5236. | Pull requests |
214+
| `pnpm e2e:mcp` | Every agent tool, resource and prompt over HTTP and stdio, with bad arguments and config variants. | Builds what it needs. Ports 5150 to 5154. | Pull requests |
215+
| `pnpm e2e:panel` | Every panel view in both themes at 1280 and 360 px: axe, overflow and page errors. | A running app, such as the SSR demo on 4000 (`--base`) started with `PANGULAR_AUTH=false`, or `--auth-log` with its log. | Weekly |
216+
| `pnpm e2e:extension` | The unpacked extension in headless Chromium with DevTools open on the app: the panel finds the server, loads `extension/ui` for the page and lists the `App` component. | A running app, such as the SSR demo on 4000 (`--base`) started with `PANGULAR_AUTH=false`. | Weekly |
217+
| `pnpm e2e:resilience` | `multitab`, `expiry`, `lifecycle` and `stress`: tabs that close, crash or hide, server restarts and a large app. | `pnpm devtools:build-pkg` and the development build. Port 5401. | Weekly |
218+
| `pnpm e2e:install` | A packed tarball installed into fresh apps per setup, package manager and Angular version, plus the package audit. | Network for npm. A local registry on 4873 and the ports after it. | Weekly, and a few cells on every release |
219+
220+
`pnpm e2e:extension` loads `extension/` with `--load-extension` in Playwright's Chromium in the new headless mode, which runs extensions, and opens DevTools on each tab with `--auto-open-devtools-for-tabs`. Playwright has no handle on DevTools, so the script selects the panel through the DevTools front end and reads `panel.html` over the Chrome DevTools Protocol. Set `HEADED=1` to watch it. The weekly `Panel views` job runs it after `pnpm e2e:panel`, against the same demo.
207221

208222
`.github/workflows/e2e.yml` runs the weekly ones on Mondays and opens an issue when one fails. In `e2e:resilience`, a check that fails on an open issue is marked as a known failure with the issue number: it prints `XFAIL` and does not fail the run, and it prints `XPASS` once it passes, so the mark can go. The `soak` check of `e2e:resilience` runs by hand only.
209223

0 commit comments

Comments
 (0)