Skip to content

fix(ci): flag codecov uploads, enable carryforward, close coverage gaps#28028

Merged
yuneng-berri merged 8 commits into
litellm_internal_stagingfrom
litellm_/priceless-archimedes-e39aab
May 16, 2026
Merged

fix(ci): flag codecov uploads, enable carryforward, close coverage gaps#28028
yuneng-berri merged 8 commits into
litellm_internal_stagingfrom
litellm_/priceless-archimedes-e39aab

Conversation

@yuneng-berri

@yuneng-berri yuneng-berri commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Coverage reporting on main/litellm_internal_staging was unstable and the
largest, most-imported modules were missing from Codecov entirely.

  • Flag every upload + enable carryforward. GHA and CircleCI uploads were
    unflagged. When a commit received the push-triggered workflows more than
    once (re-runs, or branches cut at the same SHA), Codecov accumulated many
    overlapping flagless sessions and its per-commit merge dropped the biggest
    ubiquitously-imported files (router.py, proxy_server.py, main.py,
    utils.py, cost_calculator.py) even though the uploaded XMLs contained
    them. flag_management.default_rules.carryforward: true + per-upload flags
    make a re-upload replace its prior session instead of conflicting.
  • proxy-legacy now routes through _test-unit-base, so the full
    tests/proxy_unit_tests/** suite (incl. comprehensive test_proxy_server*.py)
    is measured and uploaded with per-group flags (was plain pytest, no --cov).
  • CircleCI: add --cov + combine + upload-coverage requires for
    litellm_router_unit_testing (tests/router_unit_tests) and
    auth_ui_unit_tests (tests/proxy_admin_ui_tests) — neither was measured
    anywhere. Also combine the agent / google_generate_content_endpoint /
    litellm_utils datafiles that were produced and required but missing from
    the combine list.
  • Remove the redundant GHA test-unit-caching-redis workflow. It had a
    workflow_call signature mismatch and startup_failured on every push
    (never ran). The same three Redis test files already run on CircleCI
    redis_caching_unit_tests with coverage that is already combined/uploaded.

Test plan

  • Pushed the branch; Codecov for the head commit now reports
    router.py 66%, proxy_server.py 32%, plus main.py/utils.py/
    cost_calculator.py (previously absent); 0 flagless uploads, flags applied.
  • A/B: a flagless commit with the same upload composition dropped these
    files (single-file API 404); the flagged commit retains them.
  • Duplicate-run: re-ran proxy-db at the same SHA — files stayed present
    (flag replace, not conflict-accumulate).
  • Unit Tests: Proxy DB Operations and Unit Tests: Security GHA runs
    succeed on the final commit (no regression from the shared-base change).

Note

Low Risk
Low risk: CI/config-only changes that affect coverage upload/merge behavior and job wiring, with potential for misconfigured flags or missing workspace artifacts causing coverage reporting regressions but no production runtime impact.

Overview
Stabilizes coverage reporting by flagging every Codecov upload (GitHub Actions and CircleCI) and enabling codecov.yaml flag_management carryforward so reruns/overlapping sessions don’t cause large files to be dropped from merges.

Extends CircleCI unit-test jobs (auth_ui_unit_tests, litellm_router_unit_testing) to generate and persist per-job coverage artifacts, and updates the upload-coverage combine step to include these plus previously omitted coverage datafiles before uploading with the circleci flag. Also removes the redundant/broken GHA test-unit-caching-redis workflow.

Reviewed by Cursor Bugbot for commit ab876b4. Bugbot is set up for automated code reviews on this repo. Configure here.

Coverage uploads from GHA and CircleCI were unflagged. Commits that
receive the push-triggered workflows more than once (re-runs, or branches
cut at the same SHA) accumulated many overlapping flagless sessions, and
Codecov's per-commit merge dropped the largest, ubiquitously-imported
files (router.py, proxy_server.py, main.py, utils.py, cost_calculator.py)
from the report even though the uploaded XMLs contained them.

- codecov.yaml: flag_management.default_rules.carryforward: true
- GHA reusable bases: tag each upload with its workflow/shard name
- CircleCI: tag the combined upload "circleci"; also combine the
  agent / google_generate_content_endpoint / litellm_utils datafiles
  that were produced and required but missing from the combine list
…aching-redis

- test-unit-proxy-legacy: route through _test-unit-base so the full
  proxy_unit_tests suite (incl. comprehensive test_proxy_server*.py) is
  measured and uploaded with per-group flags (was plain pytest, no --cov)
- _test-unit-services-base: declare the enable-redis input + the six
  secrets test-unit-caching-redis passes; that workflow had a workflow_call
  signature mismatch and startup_failed on every push (never ran).
  Changes are additive/optional - proxy-db and security callers unchanged
- circleci: add --cov + persist + combine + upload-coverage requires for
  litellm_router_unit_testing (tests/router_unit_tests) and
  auth_ui_unit_tests (tests/proxy_admin_ui_tests); neither was covered
  anywhere. Redundant -k subset jobs left as-is (local_testing covers them)
CircleCI redis_caching_unit_tests already runs the exact same files
(tests/local_testing/test_dual_cache.py, test_redis_batch_optimizations.py,
test_router_utils.py) with --cov, and that datafile is already combined
and uploaded. The GHA test-unit-caching-redis workflow was redundant and
had never run (workflow_call signature mismatch -> startup_failure on
every push).

- Delete .github/workflows/test-unit-caching-redis.yml
- Revert _test-unit-services-base.yml to the flag-fix state (drop the
  enable-redis input / secrets / env wiring added only to prop up the
  GHA Redis workflow); the verified per-upload flags line is kept
- The only single-star "litellm_*" branch glob lived in the deleted
  file; no other single-star globs exist, so none remain to widen
@yuneng-berri yuneng-berri requested a review from a team May 15, 2026 23:22
@codecov

codecov Bot commented May 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes Codecov coverage reporting by flagging every upload, enabling carryforward, and wiring previously unmeasured test suites into the coverage pipeline. It resolves the root cause where flagless, overlapping CI sessions caused Codecov to drop the largest files from its merge.

  • Flag + carryforward: Adds per-upload flags to GHA base workflows and the CircleCI upload, and sets flag_management.default_rules.carryforward: true in codecov.yaml so re-uploads replace prior sessions rather than accumulating conflicts.
  • New coverage sources: Enables --cov on CircleCI's auth_ui_unit_tests and litellm_router_unit_testing jobs (neither was measured before), and adds the previously-generated-but-uncombined agent, google_generate_content_endpoint, and litellm_utils data files to the combine step.
  • proxy-legacy refactor: Rewrites test-unit-proxy-legacy.yml to delegate to _test-unit-base.yml, so each matrix shard now collects and uploads coverage with a unique per-shard flag; adds the required id-token:write and pull-requests:write permissions for OIDC and PR comments.
  • Broken workflow removed: Deletes test-unit-caching-redis.yml, which had a workflow_call signature mismatch (enable-redis and secrets not accepted by the callee) and was causing startup_failure on every push; those tests already run on CircleCI with coverage.

Confidence Score: 4/5

Safe to merge; changes are limited to CI/coverage configuration with no impact on production code paths.

The refactoring is well-motivated and the mechanics are sound. The two observations — coverage upload being gated on newly added jobs and global carryforward without a mode restriction — are operational trade-offs rather than correctness defects. The deleted workflow was already broken, and the proxy-legacy rewrite preserves equivalent test parameters.

.circleci/config.yml around the upload-coverage requires list, and codecov.yaml for the carryforward configuration.

Important Files Changed

Filename Overview
.circleci/config.yml Adds --cov flags + persist_to_workspace to auth_ui_unit_tests and litellm_router_unit_testing; adds previously-missing agent/google/litellm_utils coverage files to the combine step; adds circleci flag to Codecov upload; adds both new jobs to upload-coverage requires.
.github/workflows/_test-unit-base.yml Adds flags: inputs.artifact-name to the Codecov upload step — a one-line, low-risk change that was the missing piece for flag-per-shard uploads.
.github/workflows/_test-unit-services-base.yml Adds flags: inputs.artifact-name to the Codecov upload step, mirroring the same change in _test-unit-base.yml.
.github/workflows/test-unit-caching-redis.yml File deleted. The workflow had a broken workflow_call signature (passed enable-redis and secrets that _test-unit-services-base.yml does not accept), causing startup_failure on every push. The same tests run on CircleCI redis_caching_unit_tests with coverage.
.github/workflows/test-unit-proxy-legacy.yml Replaces a hand-rolled job with 9 reusable-workflow shards via _test-unit-base.yml; adds id-token:write and pull-requests:write permissions; each shard now uploads coverage with a unique artifact-name flag.
codecov.yaml Adds flag_management.default_rules.carryforward:true so that re-uploads replace their prior session and flags carry forward from a parent commit when not re-uploaded.

Comments Outside Diff (1)

  1. .circleci/config.yml, line 2675-2696 (link)

    P2 Coverage upload silently blocked when new jobs fail

    upload-coverage now lists litellm_router_unit_testing and auth_ui_unit_tests in its requires list. In CircleCI, if any job in requires fails, all downstream jobs are skipped — so a single flaky test in either of these newly-coverage-enabled jobs will block the entire coverage upload for that commit. Previously, a failure in those jobs left coverage unaffected (they weren't wired in). Adding a when: on_success variant is the default, but you could also consider splitting the requires list so that upload-coverage runs whenever all coverage files are ready, using an intermediate fan-in job with when: always if resilience is needed.

Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile

Comment thread codecov.yaml
Comment on lines +12 to +14
flag_management:
default_rules:
carryforward: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Global carryforward without carryforward_mode may obscure coverage regressions

flag_management.default_rules.carryforward: true without a carryforward_mode key defaults to carrying forward the full prior session for any flag that wasn't uploaded in the current push. This means a flag whose job was skipped (e.g., due to path filtering or a CI outage) will silently reuse its last-known session, making per-commit coverage look stable even when an entire test suite was never run. Setting carryforward_mode: patch limits carryforward to patch-only coverage checks and avoids inflating the project-level metric with stale data from a potentially obsolete session.

…heck names

Routing proxy-legacy through the reusable workflow renamed each check from
the bare matrix name (e.g. "proxy-response-and-misc") to
"proxy-response-and-misc / Run tests". Those bare names are required status
checks in branch protection, so the old contexts never reported and PRs sat
"Expected — Waiting for status to be reported" indefinitely.

Restore the original standalone matrix job (job name == matrix name, so the
required contexts report again) and add coverage in place: --cov on pytest
plus an OIDC Codecov upload flagged proxy-legacy-<group>. Net effect of the
gap-#2 fix is preserved (flagged coverage for tests/proxy_unit_tests/**)
without changing any check name.
@veria-ai

veria-ai Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

PR overview

CI coverage upload flagging and carryforward

This PR updates CircleCI/GitHub Actions coverage artifact handling and Codecov flag configuration, and removes a trusted-branch Redis unit-test workflow. I checked the changed upload paths, OIDC Codecov usage, workflow permissions, and the removed secret-using Redis workflow; I did not find a security issue.

Security review

  • No new security issues were flagged in the latest review.
  • No review issues remain open on this pull request.

Risk: 1/10

Comment thread .github/workflows/test-unit-proxy-legacy.yml Outdated
tests/proxy_unit_tests/** is already fully covered by test-unit-proxy-db
(its shard-coverage guard fails CI if any file in that dir is unassigned),
which this PR already flags + carryforwards. Adding --cov and id-token:write
to the legacy pull_request job was redundant and put OIDC on a job that runs
untrusted PR code. Restore the file to the base version verbatim so this PR
no longer touches proxy-legacy at all (also restores its original required
check names). Retiring proxy-legacy in favor of proxy-db on pull_request is
a separate effort that needs a branch-protection change.
@yuneng-berri yuneng-berri enabled auto-merge (squash) May 16, 2026 17:54
@yuneng-berri yuneng-berri merged commit 62dca9e into litellm_internal_staging May 16, 2026
116 checks passed
@yuneng-berri yuneng-berri deleted the litellm_/priceless-archimedes-e39aab branch May 16, 2026 17:56
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…ps (BerriAI#28028)

* fix(ci): flag codecov uploads and enable carryforward

Coverage uploads from GHA and CircleCI were unflagged. Commits that
receive the push-triggered workflows more than once (re-runs, or branches
cut at the same SHA) accumulated many overlapping flagless sessions, and
Codecov's per-commit merge dropped the largest, ubiquitously-imported
files (router.py, proxy_server.py, main.py, utils.py, cost_calculator.py)
from the report even though the uploaded XMLs contained them.

- codecov.yaml: flag_management.default_rules.carryforward: true
- GHA reusable bases: tag each upload with its workflow/shard name
- CircleCI: tag the combined upload "circleci"; also combine the
  agent / google_generate_content_endpoint / litellm_utils datafiles
  that were produced and required but missing from the combine list

* fix(ci): close coverage gaps in proxy-legacy, router-unit, auth-ui, caching-redis

- test-unit-proxy-legacy: route through _test-unit-base so the full
  proxy_unit_tests suite (incl. comprehensive test_proxy_server*.py) is
  measured and uploaded with per-group flags (was plain pytest, no --cov)
- _test-unit-services-base: declare the enable-redis input + the six
  secrets test-unit-caching-redis passes; that workflow had a workflow_call
  signature mismatch and startup_failed on every push (never ran).
  Changes are additive/optional - proxy-db and security callers unchanged
- circleci: add --cov + persist + combine + upload-coverage requires for
  litellm_router_unit_testing (tests/router_unit_tests) and
  auth_ui_unit_tests (tests/proxy_admin_ui_tests); neither was covered
  anywhere. Redundant -k subset jobs left as-is (local_testing covers them)

* fix(ci): remove dead GHA Redis workflow; keep Redis on CircleCI only

CircleCI redis_caching_unit_tests already runs the exact same files
(tests/local_testing/test_dual_cache.py, test_redis_batch_optimizations.py,
test_router_utils.py) with --cov, and that datafile is already combined
and uploaded. The GHA test-unit-caching-redis workflow was redundant and
had never run (workflow_call signature mismatch -> startup_failure on
every push).

- Delete .github/workflows/test-unit-caching-redis.yml
- Revert _test-unit-services-base.yml to the flag-fix state (drop the
  enable-redis input / secrets / env wiring added only to prop up the
  GHA Redis workflow); the verified per-upload flags line is kept
- The only single-star "litellm_*" branch glob lived in the deleted
  file; no other single-star globs exist, so none remain to widen

* fix(ci): keep proxy-legacy as a standalone job to preserve required check names

Routing proxy-legacy through the reusable workflow renamed each check from
the bare matrix name (e.g. "proxy-response-and-misc") to
"proxy-response-and-misc / Run tests". Those bare names are required status
checks in branch protection, so the old contexts never reported and PRs sat
"Expected — Waiting for status to be reported" indefinitely.

Restore the original standalone matrix job (job name == matrix name, so the
required contexts report again) and add coverage in place: --cov on pytest
plus an OIDC Codecov upload flagged proxy-legacy-<group>. Net effect of the
gap-BerriAI#2 fix is preserved (flagged coverage for tests/proxy_unit_tests/**)
without changing any check name.

* revert(ci): drop all proxy-legacy changes from this PR

tests/proxy_unit_tests/** is already fully covered by test-unit-proxy-db
(its shard-coverage guard fails CI if any file in that dir is unassigned),
which this PR already flags + carryforwards. Adding --cov and id-token:write
to the legacy pull_request job was redundant and put OIDC on a job that runs
untrusted PR code. Restore the file to the base version verbatim so this PR
no longer touches proxy-legacy at all (also restores its original required
check names). Retiring proxy-legacy in favor of proxy-db on pull_request is
a separate effort that needs a branch-protection change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants