Skip to content

Commit df28bfb

Browse files
committed
fix(ci): update mutation workflow assertions and strengthen validation coverage
1 parent 7cc493e commit df28bfb

7 files changed

Lines changed: 240 additions & 63 deletions

File tree

‎.github/workflows/code-qa.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -411,7 +411,7 @@ jobs:
411411

412412
validation-gate:
413413
name: validation-gate (code-qa)
414-
# Always-running required gate for change-based validation. It
414+
# Always-running audit gate for change-based validation. It
415415
# rejects a missing or failed classification, validation jobs that
416416
# did not run when full validation was required, jobs that ran when
417417
# a documentation-only skip was expected, and a classifier replay

‎docs/ci-change-based-validation.md‎

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Change-based validation with a required gate
1+
# Change-based validation with an audit gate
22

33
Every pull request update previously ran the full validation suite even when the
44
change could not affect product behavior. This document describes the
@@ -18,10 +18,13 @@ their `needs` graph:
1818
`pull_request`, and the category is exactly `docs-only`. The job-level
1919
condition reports `skipped`, and GitHub treats a skipped required check as
2020
successful. The required unit-test matrix jobs start both platform checks
21-
and gate their steps instead, so both required contexts are reported.
21+
and gate their steps instead, because GitHub evaluates job conditions before
22+
expanding a matrix. Skipping the matrix job itself may leave its named
23+
required contexts unreported.
2224
The required `compile` job also starts every time and fails unless the
2325
always-running formatting and validation-policy tests passed; only its
24-
expensive steps are gated. No ruleset change is needed for these checks.
26+
expensive steps are gated. This preserves the existing check names without a ruleset change; live
27+
verification is still pending.
2528
- Every other outcome runs the job unchanged: category `full`, a failed
2629
classification, a missing classification, or any non-`pull_request` event
2730
(merge queue, pushes, scheduled runs).
@@ -79,6 +82,28 @@ The allowlist is deliberately narrow:
7982
deliberately unconditional because a repository invariant test requires
8083
the knip job to run without conditions or allowed failure.
8184

85+
## Documentation input audit
86+
87+
The audit at `687f8489d` checked references to the allowlisted paths and their
88+
file names across tracked source, tests, build scripts, and CI configuration.
89+
It also checked package build entry points, test discovery, asset copying,
90+
Turbo inputs, and VSIX packaging rules.
91+
92+
| Surface | Evidence | Result |
93+
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
94+
| Extension bundling | `src/esbuild.mjs` calls `copyPaths()` with an explicit source list. `packages/build/src/esbuild.ts` copies only those configured paths. | Root `README.md` and `CHANGELOG.md` are package inputs and remain outside the allowlist. No allowlisted documentation is copied. |
95+
| VSIX packaging | `src/.vscodeignore` excludes everything, then includes the built extension, webview, assets, and release metadata. | Root `docs/`, `.github/`, and the four allowlisted root files are not packaged. |
96+
| Webview and package builds | `webview-ui/vite.config.ts`, `apps/cli/tsup.config.ts`, and `packages/types/tsup.config.ts` select code entry points and assets. | No imports, raw-file loads, or asset rules select allowlisted documentation. |
97+
| Tests and models | Vitest configurations select code tests. Searches covered `docs/`, current documentation basenames, Markdown imports, and filesystem readers. | Documentation references are comments, temporary repository files, mocked paths, or workspace examples. Tests do not read the checked-in allowlisted documents. |
98+
| Turbo and Knip | Root and package Turbo files define package-local inputs. `knip.json` selects code entry points and projects. | No explicit input reaches the root documentation tree. Default package inputs can include package-local Markdown, which remains outside the allowlist. |
99+
| Release validation | `.github/workflows/release-validation.yml` builds the extension and compares root `README.md` with the copied package README. | Release inputs retain full validation. The release workflow is unchanged. |
100+
101+
Generic runtime tools can read Markdown in a user's workspace. This does not
102+
make repository documentation a build or test input. The audit found no reason
103+
to narrow the current allowlist further. This is source-level evidence for the
104+
current tree, not a guarantee about future consumers. A new documentation
105+
reader must update the classification policy and its regression tests.
106+
82107
## Fail-safe behavior
83108

84109
| Input | Classification |
@@ -182,3 +207,20 @@ through the existing required `compile` context.
182207

183208
The repeated gate jobs remain explicit to keep workflow needs and check names
184209
visible. Extracting a shared gate action is deferred to a separate refactor.
210+
211+
## Current verification and open work
212+
213+
No live docs-only PR run has confirmed the two platform check contexts yet.
214+
Local workflow tests verify their names and wiring; this does not establish
215+
GitHub's live check-reporting behavior.
216+
217+
`format-check` and all `validation-gate` jobs are not required checks in the
218+
current ruleset. Formatting and policy-test failures block via required
219+
`compile`. Making audit-gate failures block directly still requires a separate
220+
ruleset update. The gate reads job results and cannot observe step-level gating
221+
inside `compile` or `unit-test`; workflow tests pin those step conditions.
222+
223+
All label changes currently start all five workflows and may cancel in-flight
224+
PR validation, including mutation testing. This runner-time cost is accepted
225+
for now to keep override event handling simple and consistent. A shared gate
226+
action and broader Stryker integration remain follow-ups.

‎scripts/change-classification.mjs‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -253,14 +253,13 @@ function escapeSummaryCell(value) {
253253
}
254254

255255
function escapeSummaryPath(value) {
256-
return String(value)
257-
.replaceAll("&", "&")
258-
.replaceAll("<", "&lt;")
259-
.replaceAll(">", "&gt;")
260-
.replaceAll("`", "&#96;")
261-
.replaceAll("|", "&#124;")
262-
.replaceAll("\r", "&#13;")
263-
.replaceAll("\n", "&#10;")
256+
const characters = Array.from(String(value))
257+
// Encode punctuation before Markdown parsing; cap individual paths in summaries.
258+
const text = characters.slice(0, 500).join("")
259+
return (
260+
text.replace(/[^\p{L}\p{N} /-]/gu, (character) => `&#${character.codePointAt(0)};`) +
261+
(characters.length > 500 ? "…" : "")
262+
)
264263
}
265264

266265
export function formatSummary(result, { eventName, mergeBase, headSha } = {}) {

‎scripts/change-classification.test.mjs‎

Lines changed: 143 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ describe("ci command", () => {
311311
assert.equal(outputs.get("changed-files"), "1")
312312
const summary = fs.readFileSync(stepSummary, "utf8")
313313
assert.match(summary, /docs-only/)
314-
assert.ok(summary.includes("docs/pr.md"))
314+
assert.ok(summary.includes("docs/pr&#46;md"))
315315
} finally {
316316
fs.rmSync(tempRepo, { recursive: true, force: true })
317317
}
@@ -483,7 +483,7 @@ describe("validation policy changes", () => {
483483
it("escapes filenames in classification summaries", () => {
484484
const filename = "docs/`<details>|\nforged.md"
485485
const summary = formatSummary(classifyEntries([{ status: "M", path: filename }]))
486-
assert.ok(summary.includes("docs/&#96;&lt;details&gt;&#124;&#10;forged.md"))
486+
assert.ok(summary.includes("docs/&#96;&#60;details&#62;&#124;&#10;forged&#46;md"))
487487
assert.ok(!summary.includes(filename))
488488
})
489489

@@ -513,18 +513,61 @@ it("runs the gate CLI against a docs-only repository and writes its summary", ()
513513
git(tempRepo, ["add", "."])
514514
git(tempRepo, ["commit", "-m", "docs"])
515515
const head = git(tempRepo, ["rev-parse", "HEAD"]).trim()
516-
for (const [actual, status] of [
517-
["skipped", 0],
518-
["success", 1],
519-
]) {
520-
const summary = path.join(tempRepo, `summary-${actual}.md`)
516+
const cases = [
517+
{ name: "docs skip", actual: "skipped", status: 0 },
518+
{ name: "unexpected run", actual: "success", status: 1 },
519+
{ name: "label", actual: "success", category: "full", reason: "override-label", label: "true", status: 0 },
520+
{
521+
name: "variable",
522+
actual: "success",
523+
category: "full",
524+
reason: "override-variable",
525+
variable: "true",
526+
status: 0,
527+
},
528+
{
529+
name: "label wins",
530+
actual: "success",
531+
category: "full",
532+
reason: "override-label",
533+
label: "true",
534+
variable: "true",
535+
status: 0,
536+
},
537+
{
538+
name: "draft",
539+
actual: "skipped",
540+
rule: "pr-non-draft",
541+
draft: "true",
542+
category: "full",
543+
reason: "override-label",
544+
label: "true",
545+
status: 0,
546+
},
547+
{
548+
name: "ready",
549+
actual: "success",
550+
rule: "pr-non-draft",
551+
category: "full",
552+
reason: "override-label",
553+
label: "true",
554+
status: 0,
555+
},
556+
{ name: "missing event", actual: "skipped", event: "", error: /EVENT_NAME is required/, status: 1 },
557+
{ name: "missing watches", actual: "skipped", watch: "", error: /WATCHED_JOBS is empty/, status: 1 },
558+
{ name: "missing needs", actual: "skipped", needs: "{}", error: /missing/, status: 1 },
559+
{ name: "missing base", actual: "skipped", base: "", error: /--base and --head are required/, status: 1 },
560+
]
561+
for (const testCase of cases) {
562+
const { actual, status } = testCase
563+
const summary = path.join(tempRepo, `summary-${testCase.name}.md`)
521564
const result = spawnSync(
522565
process.execPath,
523566
[
524567
path.join(tempRepo, "scripts/change-gate.mjs"),
525568
"ci",
526569
"--base",
527-
base,
570+
testCase.base ?? base,
528571
"--head",
529572
head,
530573
"--skip-metrics",
@@ -533,22 +576,32 @@ it("runs the gate CLI against a docs-only repository and writes its summary", ()
533576
encoding: "utf8",
534577
env: {
535578
...process.env,
536-
EVENT_NAME: "pull_request",
537-
WATCHED_JOBS: "expensive:always:gated",
538-
NEEDS_JSON: JSON.stringify({
539-
classify: {
540-
result: "success",
541-
outputs: { category: "docs-only", "reason-code": "allowlisted-documentation" },
542-
},
543-
expensive: { result: actual },
544-
}),
579+
EVENT_NAME: testCase.event ?? "pull_request",
580+
WATCHED_JOBS: testCase.watch ?? `expensive:${testCase.rule ?? "always"}:gated`,
581+
NEEDS_JSON:
582+
testCase.needs ??
583+
JSON.stringify({
584+
classify: {
585+
result: "success",
586+
outputs: {
587+
category: testCase.category ?? "docs-only",
588+
"reason-code": testCase.reason ?? "allowlisted-documentation",
589+
},
590+
},
591+
expensive: { result: actual },
592+
}),
545593
GITHUB_STEP_SUMMARY: summary,
546-
FORCE_FULL_LABEL: "false",
547-
FORCE_FULL_VARIABLE: "false",
594+
FORCE_FULL_LABEL: testCase.label ?? "false",
595+
PR_DRAFT: testCase.draft ?? "false",
596+
FORCE_FULL_VARIABLE: testCase.variable ?? "false",
548597
},
549598
},
550599
)
551-
assert.equal(result.status, status, result.stderr)
600+
assert.equal(result.status, status, `${testCase.name}: ${result.stderr}`)
601+
if (testCase.error) {
602+
assert.match(result.stderr, testCase.error)
603+
continue
604+
}
552605
assert.match(fs.readFileSync(summary, "utf8"), /Validation gate/)
553606
assert.match(fs.readFileSync(summary, "utf8"), /Runner-minute metrics were unavailable/)
554607
}
@@ -596,3 +649,73 @@ it("uses the PR merge base and detects documentation renames", () => {
596649
fs.rmSync(tempRepo, { recursive: true, force: true })
597650
}
598651
})
652+
653+
for (const filename of ["docs/space name.md", "docs/日本語.md", "docs/line\nbreak.md", "docs/UPPER.MD"]) {
654+
it(`classifies the real git path ${JSON.stringify(filename)}`, () => {
655+
const tempRepo = createTempRepository()
656+
try {
657+
const base = git(tempRepo, ["rev-parse", "HEAD"]).trim()
658+
fs.writeFileSync(path.join(tempRepo, filename), "# documentation")
659+
git(tempRepo, ["add", "."])
660+
git(tempRepo, ["commit", "-m", "add unusual path"])
661+
const head = git(tempRepo, ["rev-parse", "HEAD"]).trim()
662+
const { entries } = changedEntries(tempRepo, base, head)
663+
assert.deepEqual(entries, [{ status: "A", path: filename }])
664+
assert.equal(classifyEntries(entries).category, filename.endsWith(".MD") ? "full" : "docs-only")
665+
} finally {
666+
fs.rmSync(tempRepo, { recursive: true, force: true })
667+
}
668+
})
669+
}
670+
671+
it("requires full validation for a real documentation-to-source rename", () => {
672+
const tempRepo = createTempRepository()
673+
try {
674+
const base = git(tempRepo, ["rev-parse", "HEAD"]).trim()
675+
fs.mkdirSync(path.join(tempRepo, "src"))
676+
git(tempRepo, ["mv", "docs/base.md", "src/source.md"])
677+
git(tempRepo, ["commit", "-m", "move documentation into source"])
678+
const head = git(tempRepo, ["rev-parse", "HEAD"]).trim()
679+
const { entries } = changedEntries(tempRepo, base, head)
680+
assert.deepEqual(entries, [{ status: "R", oldPath: "docs/base.md", path: "src/source.md" }])
681+
assert.equal(classifyEntries(entries).category, "full")
682+
} finally {
683+
fs.rmSync(tempRepo, { recursive: true, force: true })
684+
}
685+
})
686+
687+
it("fails closed when a shallow clone lacks the base commit", () => {
688+
const tempRepo = createTempRepository()
689+
const clone = fs.mkdtempSync(path.join(os.tmpdir(), "classification-shallow-"))
690+
try {
691+
const base = git(tempRepo, ["rev-parse", "HEAD"]).trim()
692+
fs.writeFileSync(path.join(tempRepo, "docs/base.md"), "updated")
693+
git(tempRepo, ["add", "."])
694+
git(tempRepo, ["commit", "-m", "docs"])
695+
const head = git(tempRepo, ["rev-parse", "HEAD"]).trim()
696+
git(tempRepo, ["clone", "--depth=1", `file://${tempRepo}`, clone])
697+
assert.equal(git(clone, ["rev-parse", "--is-shallow-repository"]).trim(), "true")
698+
assert.throws(() => changedEntries(clone, base, head), /Could not resolve base commit/)
699+
assert.equal(validationPolicyChanged(clone, base, head), true)
700+
} finally {
701+
fs.rmSync(tempRepo, { recursive: true, force: true })
702+
fs.rmSync(clone, { recursive: true, force: true })
703+
}
704+
})
705+
706+
it("renders formatting and autolink punctuation as literal filename characters", () => {
707+
const filename = "docs/[link](https://example.com)*_~#@!\\.md"
708+
const summary = formatSummary(classifyEntries([{ status: "M", path: filename }]))
709+
const line = summary.split("\n").find((line) => line.startsWith("- docs/"))
710+
assert.equal(
711+
line,
712+
"- docs/&#91;link&#93;&#40;https&#58;//example&#46;com&#41;&#42;&#95;&#126;&#35;&#64;&#33;&#92;&#46;md",
713+
)
714+
})
715+
716+
it("caps long filename summaries", () => {
717+
const filename = `docs/${"a".repeat(600)}.md`
718+
const summary = formatSummary(classifyEntries([{ status: "M", path: filename }]))
719+
const line = summary.split("\n").find((line) => line.startsWith("- docs/"))
720+
assert.equal(line, `- docs/${"a".repeat(495)}…`)
721+
})

‎scripts/change-gate.test.mjs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -429,6 +429,10 @@ describe("change-based validation workflow wiring", () => {
429429
assert.equal(workflow.on.pull_request["paths-ignore"], undefined)
430430
assert.equal(Boolean(workflow.on.merge_group), expectation.hasMergeGroup)
431431
const jobs = workflow.jobs
432+
assert.deepEqual(
433+
Object.keys(jobs).sort(),
434+
["classify", "validation-gate", ...expectation.jobs.map((job) => job.id)].sort(),
435+
)
432436
const gate = jobs["validation-gate"]
433437
assert.equal(gate.if, "always()")
434438
assert.deepEqual([...gate.needs].sort(), ["classify", ...expectation.jobs.map((job) => job.id)].sort())
@@ -471,14 +475,19 @@ describe("change-based validation workflow wiring", () => {
471475
}
472476
it("keeps required compile and matrix checks while gating expensive steps", () => {
473477
const { jobs } = parse(fs.readFileSync(path.join(repositoryRoot, ".github/workflows/code-qa.yml"), "utf8"))
478+
assert.equal(jobs.compile.name, undefined)
479+
assert.equal(jobs["unit-test"].name, "platform-unit-test (${{ matrix.name }})")
474480
assert.deepEqual(jobs.compile.needs, ["classify", "format-check"])
475481
assert.equal(jobs.compile.steps[0].env.FORMAT_RESULT, "${{ needs.format-check.result }}")
476482
assert.equal(jobs.compile.steps[0].run, 'test "$FORMAT_RESULT" = success')
477483
assert.ok(jobs["format-check"].steps.some((step) => step.run === "pnpm format:check"))
478484
assert.ok(jobs["format-check"].steps.some((step) => step.run === "pnpm test:change-gate"))
479485
for (const id of ["compile", "unit-test"]) {
480486
assert.equal(jobs[id].if, "${{ !cancelled() }}")
481-
for (const term of GATING_TERMS.slice(1)) assert.ok(jobs[id].env.RUN_VALIDATION.includes(term))
487+
assert.equal(
488+
jobs[id].env.RUN_VALIDATION,
489+
"${{ needs.classify.result != 'success' || github.event_name != 'pull_request' || needs.classify.outputs.category != 'docs-only' }}",
490+
)
482491
for (const step of jobs[id].steps.slice(1))
483492
assert.ok(step.if.includes("env.RUN_VALIDATION == 'true'"), step.name)
484493
}

0 commit comments

Comments
 (0)