ci(coverage): merge per-integration coverage in All Green before upload#9086
Merged
Conversation
Each test cell uploaded its own report to Codecov, so a commit sent ~430 uploads. Codecov silently parks uploads past its ~150-per-commit ceiling in `started` and never merges them, so roughly 40 reports' worth of coverage was dropped from every commit. The Datadog coverage upload was a separate, broken path: `upload-coverage-artifact` probed for files with `find -maxdepth 1`, but the LCOV lives one level deeper at `coverage/node-<version>/lcov.info`, so the check found nothing, no `coverage-*` artifact was ever produced, and `datadog-ci coverage upload` reported nothing while passing green. All Green already downloads every `coverage-*` artifact to drive the Datadog upload, so it is the one place that sees the whole commit's coverage. It now folds those per-cell reports into one merged LCOV per integration and uploads the groups to both backends: 1. `upload-coverage-artifact` recurses for `lcov.info`, fixing the Datadog starvation, and names each artifact `coverage-<flag>__<job>-<index>` so matrix cells that share a flag (cypress varies `spec` outside its flag) stop clobbering each other under one name. 2. `merge-coverage-groups.mjs` groups cells by integration — stripping Node.js and library versions, which are noise for "which integration regressed" — and merges each group's LCOV by summing hit counters. The producers already wrote each report through the patched `getLineCoverage`, so the text merge needs no istanbul dependency in All Green's sparse checkout. ~430 cells collapse to ~100 readable groups (`apm-integrations-next`, `serverless-aws-sdk-s3`), well under the ceiling. 3. The per-cell Codecov upload is gone; All Green uploads one report per group to Codecov (CLI directly, to avoid re-validating the uploader ~100 times) and Datadog, both flagged with the integration name. `master-coverage` still rides every Codecov upload on PRs targeting master so the `codecov/patch` gate fires; reruns are de-duplicated to the newest run so a stale rerun's counters are not double-counted.
Contributor
Overall package sizeSelf size: 6.36 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
BenchmarksBenchmark execution time: 2026-06-26 15:58:34 Comparing candidate commit 08fcc11 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2263 metrics, 20 unstable metrics.
|
The first cut merged each group's LCOV locally, reimplementing istanbul's coverage-map merge as ~350 lines of LCOV text parsing. Both Codecov and Datadog already merge same-flag uploads server-side — that is how they combined the ~430 per-cell reports before this change — so the local merge bought nothing and carried the drift risk of a hand-rolled parser that has to round-trip every field the backends read. Drop it: All Green now sorts each cell's already-patched `lcov.info` into its integration's directory and uploads the directory under one flag. Reports pass through byte-for-byte, so the `getLineCoverage` patch the producers baked in survives without needing a patched istanbul in All Green's sparse checkout, and a future LCOV field needs no parser change. Also drop the unused `CODECOV_TOKEN` env — the repo uploads tokenless and a token would have to be an org secret that fork PRs cannot read.
…ives Routing only lcov to Codecov collapsed branch coverage from ~10000 to ~30 and method coverage from ~6250 to ~3: Codecov's lcov parser ingests line hits but not istanbul's BRDA/FN records, so every file reported branches=0 (proxy.js: 44 branches on master, 0 here) while lines and overall coverage looked right. The old per-cell upload avoided this by letting codecov-action search the coverage directory, which picked up istanbul's coverage-final.json — its NodeProcessor reads the full branch and function maps. Restore that: every cell now also emits coverage-final.json (the unit nyc config gained the json reporter; the integration harness already wrote one), group-coverage sorts each cell's reports into coverage-upload/<group>/lcov and .../json, and All Green uploads the JSON to Codecov and the lcov to Datadog — Datadog does not ingest istanbul JSON. Codecov's directory search matches the JSON because the copied name keeps the `coverage-` prefix.
BridgeAR
marked this pull request as ready for review
June 26, 2026 14:40
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c98789b6b9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![datadog-prod-us1-5[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The rerun deduplication in planCoverageGroups used JavaScript string comparison (runId > previous) instead of numeric comparison. When two workflow run IDs straddle a digit-length boundary — '9' vs '10' — the lexicographic order inverts, causing the older run's stale coverage to survive. Fixed to Number(runId) > Number(previous) with a regression test.
📊 Validated against 10 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit c98789b · What is Autotest? · Any feedback? Reach out in #autotest
…cally
Both backends reject any flag that fails Codecov's `^[\w\.\-]{1,45}$`, so the
two grouping bugs below silently dropped coverage for busy areas instead of
erroring.
1. Bucket flags joined member libraries with `+`, an invalid flag character;
they now join with `_`, and a name that would overrun 45 characters falls
back to a numbered bucket that stays valid.
2. The newest-run selection compared run ids as strings, so a rerun crossing a
power-of-ten boundary (`'9' > '10'`) kept the older, stale failed run. The
compare is now numeric.
Both upload steps walked the ~100 integration groups in a serial for loop, so the runner sat idle waiting on one latency-bound request at a time while the rest of the job was done. Each group uploads independently, so a 10-wide xargs fan-out hides that latency. The Codecov create-commit/create-report setup stays serial ahead of the fan-out, since only the do-upload calls are independent.
This was referenced Jun 26, 2026
Closed
Merged
Merged
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
…ad (#9086) Each test cell uploaded its own report to Codecov, so a commit sent ~430 uploads. Codecov silently parks uploads past its ~150-per-commit ceiling in `started` and never merges them, so roughly 40 reports' worth of coverage was dropped from every commit. The Datadog coverage upload was separately broken: `upload-coverage-artifact` probed for files with `find -maxdepth 1`, but the report lives one level deeper at `coverage/node-<version>/`, so the check found nothing, no `coverage-*` artifact was produced, and `datadog-ci coverage upload` reported nothing while passing green. All Green already downloads every `coverage-*` artifact to drive the Datadog upload, so it is the one place that sees a whole commit's coverage. It now groups the per-cell reports by integration and uploads ~100 groups to both backends instead of ~430 per-cell reports: 1. `upload-coverage-artifact` recurses for the report files and names each artifact `coverage-<flag>__<job>-<index>` so matrix cells that share a flag (cypress varies `spec` outside its flag) stop clobbering each other. 2. `scripts/group-coverage.mjs` sorts each cell's report into its integration's directory, stripping Node.js and library versions, which are noise for "which integration regressed". Reports are not merged locally — both backends merge same-flag uploads server-side — so each report passes through byte-for-byte and the harness needs no istanbul dependency in All Green's sparse checkout. ~430 cells collapse to ~100 groups. 3. Each cell emits both lcov and istanbul JSON: Codecov reads branch and function coverage from the JSON (its lcov parser ingests only line hits), Datadog reads the lcov and does not ingest the JSON. All Green uploads each format to the backend that reads it, one group per integration, flagged with the integration name. `master-coverage` still rides every Codecov upload on PRs targeting master so the `codecov/patch` gate fires; reruns de-duplicate to the newest run so a stale rerun's counters are not double-counted.
rochdev
pushed a commit
that referenced
this pull request
Jul 2, 2026
…ad (#9086) Each test cell uploaded its own report to Codecov, so a commit sent ~430 uploads. Codecov silently parks uploads past its ~150-per-commit ceiling in `started` and never merges them, so roughly 40 reports' worth of coverage was dropped from every commit. The Datadog coverage upload was separately broken: `upload-coverage-artifact` probed for files with `find -maxdepth 1`, but the report lives one level deeper at `coverage/node-<version>/`, so the check found nothing, no `coverage-*` artifact was produced, and `datadog-ci coverage upload` reported nothing while passing green. All Green already downloads every `coverage-*` artifact to drive the Datadog upload, so it is the one place that sees a whole commit's coverage. It now groups the per-cell reports by integration and uploads ~100 groups to both backends instead of ~430 per-cell reports: 1. `upload-coverage-artifact` recurses for the report files and names each artifact `coverage-<flag>__<job>-<index>` so matrix cells that share a flag (cypress varies `spec` outside its flag) stop clobbering each other. 2. `scripts/group-coverage.mjs` sorts each cell's report into its integration's directory, stripping Node.js and library versions, which are noise for "which integration regressed". Reports are not merged locally — both backends merge same-flag uploads server-side — so each report passes through byte-for-byte and the harness needs no istanbul dependency in All Green's sparse checkout. ~430 cells collapse to ~100 groups. 3. Each cell emits both lcov and istanbul JSON: Codecov reads branch and function coverage from the JSON (its lcov parser ingests only line hits), Datadog reads the lcov and does not ingest the JSON. All Green uploads each format to the backend that reads it, one group per integration, flagged with the integration name. `master-coverage` still rides every Codecov upload on PRs targeting master so the `codecov/patch` gate fires; reruns de-duplicate to the newest run so a stale rerun's counters are not double-counted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each test cell uploads its own report to Codecov, so a commit sends ~430 uploads. Codecov silently parks uploads past its ~150-per-commit ceiling in
startedand never merges them — roughly 40 reports' worth of coverage was dropped from every commit (see #9073, where this surfaced as "40 started, 393 merged"). Separately, the Datadog coverage upload was broken:upload-coverage-artifactprobed for files withfind -maxdepth 1, but the LCOV lives one level deeper atcoverage/node-<version>/lcov.info, so nocoverage-*artifact was ever produced anddatadog-ci coverage uploadreported nothing while passing green.All Green already downloads every
coverage-*artifact to drive the Datadog upload, so it is the one place that sees a commit's whole coverage. It now folds the per-cell reports into one merged LCOV per integration and uploads the groups to both backends:upload-coverage-artifactrecurses forlcov.info(fixing the Datadog starvation) and names each artifactcoverage-<flag>__<job>-<index>so matrix cells that share a flag (cypress variesspecoutside its flag) stop clobbering each other.scripts/merge-coverage-groups.mjsgroups cells by integration — stripping Node.js and library versions, which are noise for "which integration regressed" — and merges each group's LCOV by summing hit counters. Producers already wrote each report through the patchedgetLineCoverage, so the text merge needs no istanbul dependency in All Green's sparse checkout. ~430 cells collapse to ~100 readable groups (apm-integrations-next,serverless-aws-sdk-s3).Why
Fewer uploads is the only thing that actually fixes the Codecov ceiling, and the grouping is anchored on the workflow/flag rather than a hand-maintained map, so a new test job lands in its integration automatically. The merge runs with zero GitHub REST calls and no
actions/cacheuse, so it adds no rate-limit pressure — and removing ~430 codecov-action invocations (each a tokenless validation Codecov runs against GitHub) reduces it.master-coveragestill rides every Codecov upload on PRs targeting master so thecodecov/patchgate fires; reruns de-duplicate to the newest run so a stale rerun's counters are not double-counted.Test plan
scripts/merge-coverage-groups.spec.mjs(run bytest:scripts) covers integration stripping, lcov-merge summing, rerun de-dup, multi-version cells, and flag bucketing.MERGED(nostarted) anddatadog-ci coverage uploadreports a non-zero file count per group.CODECOV_TOKENsecret is configured; without it the CLI falls back to tokenless upload (unchanged from today's behavior).