Skip to content

Commit 90b5008

Browse files
quality-delta: consume the merged artifact freshness status from #1423
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 2f49256 commit 90b5008

4 files changed

Lines changed: 59 additions & 12 deletions

File tree

‎scripts/quality-delta/README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,12 @@ every row plus the notes for inputs that were unavailable.
5454
amendment). If the baseline's `schemaVersion` differs from head's, every baseline-derived row is
5555
dropped and the comment says so; PR-local rows still show. To resume diffing, add a migration note
5656
under "Schema migrations" below and bump the reader accordingly.
57-
- **Stale inputs are labelled, not laundered.** repo-health flags a read artifact (coverage, size)
58-
as `stale` when a source file is newer than it. Rows derived from a stale artifact render as
59-
`JS gzip (stale artifact)` rather than silently claiming to describe the head commit.
57+
- **Stale inputs are labelled, not laundered.** repo-health proves a read artifact's freshness
58+
(`provenance.inputs.*.status`) only from a producing commit the artifact stamps in its own bytes.
59+
A row whose artifact is `stale` renders as `JS gzip (stale artifact)` rather than silently claiming
60+
to describe the head commit. `unknown` — today's honest answer for every artifact, since no
61+
producer stamps a commit — gets one job-summary note instead of a caveat on every size and coverage
62+
row, which is the comment fatigue this comment exists to remove.
6063
- **Missing inputs cost rows, not runs.** No baseline, no size artifact, no coverage JSON — each
6164
costs the rows it would have produced plus a job-summary note. Fork PRs (read-only token, no
6265
secrets) therefore degrade to job-summary output: the comment step is skipped, never failed.

‎scripts/quality-delta/fixtures.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,14 @@ export function snapshotFixture(overrides: SnapshotOverrides = {}): RepoHealthSn
3838
coverageSummary: {
3939
path: 'coverage/coverage-summary.json',
4040
sha256: 'cccccccccccc',
41-
producerInputs: ['src', 'test', 'vitest.config.ts'],
42-
stale: false,
41+
producerCommit: commit,
42+
status: 'fresh',
4343
},
4444
sizeReport: {
4545
path: '.tmp/size-report.json',
4646
sha256: 'ssssssssssss',
47-
producerInputs: ['src', 'package.json'],
48-
stale: overrides.sizeStale ?? false,
47+
producerCommit: overrides.sizeStale === true ? 'f'.repeat(40) : commit,
48+
status: overrides.sizeStale === true ? 'stale' : 'fresh',
4949
},
5050
},
5151
},

‎scripts/quality-delta/model.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@ function commentOf(overrides: Partial<DeltaSources> = {}): string {
2020
return renderComment(computeQualityDelta(sources(overrides)), CONTEXT);
2121
}
2222

23+
function summaryOf(overrides: Partial<DeltaSources> = {}): string {
24+
return renderJobSummary(computeQualityDelta(sources(overrides)), CONTEXT);
25+
}
26+
2327
test('an unchanged tree collapses the whole comment to one line', () => {
2428
expect(commentOf()).toBe(`${NO_DELTAS_LINE}\n`);
2529
});
@@ -57,6 +61,27 @@ test('a metric read from a stale artifact is marked rather than silently attribu
5761
expect(commentOf({ head })).toContain('| JS gzip (stale artifact) |');
5862
});
5963

64+
test('an unproven (`unknown`) artifact caveats the job summary, not every row', () => {
65+
// #1423 reports `unknown` for every artifact whose producer stamps no commit, i.e. all of them
66+
// today; marking each row would caveat size and coverage forever.
67+
const head = snapshotFixture({ jsGzipBytes: 900_000 });
68+
const sizeReport = head.provenance.inputs.sizeReport!;
69+
const unknown = {
70+
...head,
71+
provenance: {
72+
...head.provenance,
73+
inputs: {
74+
...head.provenance.inputs,
75+
sizeReport: { ...sizeReport, producerCommit: null, status: 'unknown' as const },
76+
},
77+
},
78+
};
79+
const comment = commentOf({ head: unknown });
80+
expect(comment).toContain('| JS gzip |');
81+
expect(comment).not.toContain('stale artifact');
82+
expect(summaryOf({ head: unknown })).toContain('freshness as `unknown`');
83+
});
84+
6085
test('PR-local coverage rows are scored against a bound, not a baseline', () => {
6186
// 95% is comfortably above the gate threshold + margin, so it earns no row.
6287
expect(commentOf({ changedCoverage: changedCoverageFixture() })).toBe(`${NO_DELTAS_LINE}\n`);

‎scripts/quality-delta/model.ts‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
//
2222
// All I/O — reading files, resolving links, posting the comment — belongs to run.ts.
2323

24-
import type { RepoHealthSnapshot } from '../repo-health/model.ts';
24+
import type { ArtifactStatus, RepoHealthSnapshot } from '../repo-health/model.ts';
2525
import {
2626
MAX_COMMENT_LINES,
2727
QUALITY_DELTA_METRICS,
@@ -75,11 +75,21 @@ function formatSigned(value: number, format: MetricFormat): string {
7575
return `${sign}${formatValue(Math.abs(value), format)}`;
7676
}
7777

78-
function staleInputFlag(snapshot: RepoHealthSnapshot, spec: MetricSpec): boolean {
78+
function artifactStatus(snapshot: RepoHealthSnapshot, spec: MetricSpec): ArtifactStatus | null {
7979
const inputs = snapshot.provenance.inputs;
80-
if (spec.staleInput === 'sizeReport') return inputs.sizeReport?.stale === true;
81-
if (spec.staleInput === 'coverageSummary') return inputs.coverageSummary?.stale === true;
82-
return false;
80+
if (spec.staleInput === 'sizeReport') return inputs.sizeReport?.status ?? null;
81+
if (spec.staleInput === 'coverageSummary') return inputs.coverageSummary?.status ?? null;
82+
return null;
83+
}
84+
85+
/**
86+
* Only a PROVEN stale artifact marks its row. #1423 reports `unknown` whenever the producer stamps
87+
* no commit, which is every artifact today, so marking that too would put a caveat on every size and
88+
* coverage row forever — the comment fatigue this issue exists to remove. `unknown` gets one job
89+
* summary note instead.
90+
*/
91+
function staleInputFlag(snapshot: RepoHealthSnapshot, spec: MetricSpec): boolean {
92+
return artifactStatus(snapshot, spec) === 'stale';
8393
}
8494

8595
function deltaRow(spec: MetricSpec, sources: DeltaSources): QualityDeltaRow | null {
@@ -155,6 +165,15 @@ function missingInputNotes(sources: DeltaSources): string[] {
155165
'Slow-test report unavailable (the vitest reporter writes it during the coverage run).',
156166
);
157167
}
168+
const unverified = QUALITY_DELTA_METRICS.filter(
169+
(spec) => artifactStatus(sources.head, spec) === 'unknown',
170+
);
171+
if (unverified.length > 0) {
172+
notes.push(
173+
'Size and coverage are read artifacts whose producers stamp no commit, so repo-health reports ' +
174+
'their freshness as `unknown` (#1423): treat those rows as not provably current with the head tree.',
175+
);
176+
}
158177
return notes;
159178
}
160179

0 commit comments

Comments
 (0)