fix(ci): flag codecov uploads, enable carryforward, close coverage gaps#28028
Conversation
Coverage uploads from GHA and CircleCI were unflagged. Commits that receive the push-triggered workflows more than once (re-runs, or branches cut at the same SHA) accumulated many overlapping flagless sessions, and Codecov's per-commit merge dropped the largest, ubiquitously-imported files (router.py, proxy_server.py, main.py, utils.py, cost_calculator.py) from the report even though the uploaded XMLs contained them. - codecov.yaml: flag_management.default_rules.carryforward: true - GHA reusable bases: tag each upload with its workflow/shard name - CircleCI: tag the combined upload "circleci"; also combine the agent / google_generate_content_endpoint / litellm_utils datafiles that were produced and required but missing from the combine list
…aching-redis - test-unit-proxy-legacy: route through _test-unit-base so the full proxy_unit_tests suite (incl. comprehensive test_proxy_server*.py) is measured and uploaded with per-group flags (was plain pytest, no --cov) - _test-unit-services-base: declare the enable-redis input + the six secrets test-unit-caching-redis passes; that workflow had a workflow_call signature mismatch and startup_failed on every push (never ran). Changes are additive/optional - proxy-db and security callers unchanged - circleci: add --cov + persist + combine + upload-coverage requires for litellm_router_unit_testing (tests/router_unit_tests) and auth_ui_unit_tests (tests/proxy_admin_ui_tests); neither was covered anywhere. Redundant -k subset jobs left as-is (local_testing covers them)
CircleCI redis_caching_unit_tests already runs the exact same files (tests/local_testing/test_dual_cache.py, test_redis_batch_optimizations.py, test_router_utils.py) with --cov, and that datafile is already combined and uploaded. The GHA test-unit-caching-redis workflow was redundant and had never run (workflow_call signature mismatch -> startup_failure on every push). - Delete .github/workflows/test-unit-caching-redis.yml - Revert _test-unit-services-base.yml to the flag-fix state (drop the enable-redis input / secrets / env wiring added only to prop up the GHA Redis workflow); the verified per-upload flags line is kept - The only single-star "litellm_*" branch glob lived in the deleted file; no other single-star globs exist, so none remain to widen
…itellm_/priceless-archimedes-e39aab
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fixes Codecov coverage reporting by flagging every upload, enabling carryforward, and wiring previously unmeasured test suites into the coverage pipeline. It resolves the root cause where flagless, overlapping CI sessions caused Codecov to drop the largest files from its merge.
Confidence Score: 4/5Safe to merge; changes are limited to CI/coverage configuration with no impact on production code paths. The refactoring is well-motivated and the mechanics are sound. The two observations — coverage upload being gated on newly added jobs and global carryforward without a mode restriction — are operational trade-offs rather than correctness defects. The deleted workflow was already broken, and the proxy-legacy rewrite preserves equivalent test parameters. .circleci/config.yml around the upload-coverage requires list, and codecov.yaml for the carryforward configuration.
|
| Filename | Overview |
|---|---|
| .circleci/config.yml | Adds --cov flags + persist_to_workspace to auth_ui_unit_tests and litellm_router_unit_testing; adds previously-missing agent/google/litellm_utils coverage files to the combine step; adds circleci flag to Codecov upload; adds both new jobs to upload-coverage requires. |
| .github/workflows/_test-unit-base.yml | Adds flags: inputs.artifact-name to the Codecov upload step — a one-line, low-risk change that was the missing piece for flag-per-shard uploads. |
| .github/workflows/_test-unit-services-base.yml | Adds flags: inputs.artifact-name to the Codecov upload step, mirroring the same change in _test-unit-base.yml. |
| .github/workflows/test-unit-caching-redis.yml | File deleted. The workflow had a broken workflow_call signature (passed enable-redis and secrets that _test-unit-services-base.yml does not accept), causing startup_failure on every push. The same tests run on CircleCI redis_caching_unit_tests with coverage. |
| .github/workflows/test-unit-proxy-legacy.yml | Replaces a hand-rolled job with 9 reusable-workflow shards via _test-unit-base.yml; adds id-token:write and pull-requests:write permissions; each shard now uploads coverage with a unique artifact-name flag. |
| codecov.yaml | Adds flag_management.default_rules.carryforward:true so that re-uploads replace their prior session and flags carry forward from a parent commit when not re-uploaded. |
Comments Outside Diff (1)
-
.circleci/config.yml, line 2675-2696 (link)Coverage upload silently blocked when new jobs fail
upload-coveragenow listslitellm_router_unit_testingandauth_ui_unit_testsin itsrequireslist. In CircleCI, if any job inrequiresfails, all downstream jobs are skipped — so a single flaky test in either of these newly-coverage-enabled jobs will block the entire coverage upload for that commit. Previously, a failure in those jobs left coverage unaffected (they weren't wired in). Adding awhen: on_successvariant is the default, but you could also consider splitting therequireslist so thatupload-coverageruns whenever all coverage files are ready, using an intermediate fan-in job withwhen: alwaysif resilience is needed.
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
| flag_management: | ||
| default_rules: | ||
| carryforward: true |
There was a problem hiding this comment.
Global carryforward without
carryforward_mode may obscure coverage regressions
flag_management.default_rules.carryforward: true without a carryforward_mode key defaults to carrying forward the full prior session for any flag that wasn't uploaded in the current push. This means a flag whose job was skipped (e.g., due to path filtering or a CI outage) will silently reuse its last-known session, making per-commit coverage look stable even when an entire test suite was never run. Setting carryforward_mode: patch limits carryforward to patch-only coverage checks and avoids inflating the project-level metric with stale data from a potentially obsolete session.
…heck names Routing proxy-legacy through the reusable workflow renamed each check from the bare matrix name (e.g. "proxy-response-and-misc") to "proxy-response-and-misc / Run tests". Those bare names are required status checks in branch protection, so the old contexts never reported and PRs sat "Expected — Waiting for status to be reported" indefinitely. Restore the original standalone matrix job (job name == matrix name, so the required contexts report again) and add coverage in place: --cov on pytest plus an OIDC Codecov upload flagged proxy-legacy-<group>. Net effect of the gap-#2 fix is preserved (flagged coverage for tests/proxy_unit_tests/**) without changing any check name.
…itellm_/priceless-archimedes-e39aab
PR overviewCI coverage upload flagging and carryforwardThis PR updates CircleCI/GitHub Actions coverage artifact handling and Codecov flag configuration, and removes a trusted-branch Redis unit-test workflow. I checked the changed upload paths, OIDC Codecov usage, workflow permissions, and the removed secret-using Redis workflow; I did not find a security issue. Security review
Risk: 1/10 |
tests/proxy_unit_tests/** is already fully covered by test-unit-proxy-db (its shard-coverage guard fails CI if any file in that dir is unassigned), which this PR already flags + carryforwards. Adding --cov and id-token:write to the legacy pull_request job was redundant and put OIDC on a job that runs untrusted PR code. Restore the file to the base version verbatim so this PR no longer touches proxy-legacy at all (also restores its original required check names). Retiring proxy-legacy in favor of proxy-db on pull_request is a separate effort that needs a branch-protection change.
…itellm_/priceless-archimedes-e39aab
…ps (BerriAI#28028) * fix(ci): flag codecov uploads and enable carryforward Coverage uploads from GHA and CircleCI were unflagged. Commits that receive the push-triggered workflows more than once (re-runs, or branches cut at the same SHA) accumulated many overlapping flagless sessions, and Codecov's per-commit merge dropped the largest, ubiquitously-imported files (router.py, proxy_server.py, main.py, utils.py, cost_calculator.py) from the report even though the uploaded XMLs contained them. - codecov.yaml: flag_management.default_rules.carryforward: true - GHA reusable bases: tag each upload with its workflow/shard name - CircleCI: tag the combined upload "circleci"; also combine the agent / google_generate_content_endpoint / litellm_utils datafiles that were produced and required but missing from the combine list * fix(ci): close coverage gaps in proxy-legacy, router-unit, auth-ui, caching-redis - test-unit-proxy-legacy: route through _test-unit-base so the full proxy_unit_tests suite (incl. comprehensive test_proxy_server*.py) is measured and uploaded with per-group flags (was plain pytest, no --cov) - _test-unit-services-base: declare the enable-redis input + the six secrets test-unit-caching-redis passes; that workflow had a workflow_call signature mismatch and startup_failed on every push (never ran). Changes are additive/optional - proxy-db and security callers unchanged - circleci: add --cov + persist + combine + upload-coverage requires for litellm_router_unit_testing (tests/router_unit_tests) and auth_ui_unit_tests (tests/proxy_admin_ui_tests); neither was covered anywhere. Redundant -k subset jobs left as-is (local_testing covers them) * fix(ci): remove dead GHA Redis workflow; keep Redis on CircleCI only CircleCI redis_caching_unit_tests already runs the exact same files (tests/local_testing/test_dual_cache.py, test_redis_batch_optimizations.py, test_router_utils.py) with --cov, and that datafile is already combined and uploaded. The GHA test-unit-caching-redis workflow was redundant and had never run (workflow_call signature mismatch -> startup_failure on every push). - Delete .github/workflows/test-unit-caching-redis.yml - Revert _test-unit-services-base.yml to the flag-fix state (drop the enable-redis input / secrets / env wiring added only to prop up the GHA Redis workflow); the verified per-upload flags line is kept - The only single-star "litellm_*" branch glob lived in the deleted file; no other single-star globs exist, so none remain to widen * fix(ci): keep proxy-legacy as a standalone job to preserve required check names Routing proxy-legacy through the reusable workflow renamed each check from the bare matrix name (e.g. "proxy-response-and-misc") to "proxy-response-and-misc / Run tests". Those bare names are required status checks in branch protection, so the old contexts never reported and PRs sat "Expected — Waiting for status to be reported" indefinitely. Restore the original standalone matrix job (job name == matrix name, so the required contexts report again) and add coverage in place: --cov on pytest plus an OIDC Codecov upload flagged proxy-legacy-<group>. Net effect of the gap-BerriAI#2 fix is preserved (flagged coverage for tests/proxy_unit_tests/**) without changing any check name. * revert(ci): drop all proxy-legacy changes from this PR tests/proxy_unit_tests/** is already fully covered by test-unit-proxy-db (its shard-coverage guard fails CI if any file in that dir is unassigned), which this PR already flags + carryforwards. Adding --cov and id-token:write to the legacy pull_request job was redundant and put OIDC on a job that runs untrusted PR code. Restore the file to the base version verbatim so this PR no longer touches proxy-legacy at all (also restores its original required check names). Retiring proxy-legacy in favor of proxy-db on pull_request is a separate effort that needs a branch-protection change.
Summary
Coverage reporting on
main/litellm_internal_stagingwas unstable and thelargest, most-imported modules were missing from Codecov entirely.
unflagged. When a commit received the push-triggered workflows more than
once (re-runs, or branches cut at the same SHA), Codecov accumulated many
overlapping flagless sessions and its per-commit merge dropped the biggest
ubiquitously-imported files (
router.py,proxy_server.py,main.py,utils.py,cost_calculator.py) even though the uploaded XMLs containedthem.
flag_management.default_rules.carryforward: true+ per-upload flagsmake a re-upload replace its prior session instead of conflicting.
_test-unit-base, so the fulltests/proxy_unit_tests/**suite (incl. comprehensivetest_proxy_server*.py)is measured and uploaded with per-group flags (was plain
pytest, no--cov).--cov+ combine +upload-coveragerequires forlitellm_router_unit_testing(tests/router_unit_tests) andauth_ui_unit_tests(tests/proxy_admin_ui_tests) — neither was measuredanywhere. Also combine the
agent/google_generate_content_endpoint/litellm_utilsdatafiles that were produced and required but missing fromthe combine list.
test-unit-caching-redisworkflow. It had aworkflow_callsignature mismatch andstartup_failured on every push(never ran). The same three Redis test files already run on CircleCI
redis_caching_unit_testswith coverage that is already combined/uploaded.Test plan
router.py66%,proxy_server.py32%, plusmain.py/utils.py/cost_calculator.py(previously absent); 0 flagless uploads, flags applied.files (single-file API 404); the flagged commit retains them.
proxy-dbat the same SHA — files stayed present(flag replace, not conflict-accumulate).
Unit Tests: Proxy DB OperationsandUnit Tests: SecurityGHA runssucceed on the final commit (no regression from the shared-base change).
Note
Low Risk
Low risk: CI/config-only changes that affect coverage upload/merge behavior and job wiring, with potential for misconfigured flags or missing workspace artifacts causing coverage reporting regressions but no production runtime impact.
Overview
Stabilizes coverage reporting by flagging every Codecov upload (GitHub Actions and CircleCI) and enabling
codecov.yamlflag_managementcarryforward so reruns/overlapping sessions don’t cause large files to be dropped from merges.Extends CircleCI unit-test jobs (
auth_ui_unit_tests,litellm_router_unit_testing) to generate and persist per-job coverage artifacts, and updates theupload-coveragecombine step to include these plus previously omitted coverage datafiles before uploading with thecircleciflag. Also removes the redundant/broken GHAtest-unit-caching-redisworkflow.Reviewed by Cursor Bugbot for commit ab876b4. Bugbot is set up for automated code reviews on this repo. Configure here.