fix: avoid leaking every Router instance via _cached_get_model_group_info - #41582

TLDR

Problem this solves:

• Router leaks every instance once it calls a cached helper method
• Happens even after calling Router's own documented cleanup, discard()
• cache_clear() on that helper also wipes every other Router's cache

How it solves it:

• Moves the cache from the class to the instance, matching a sibling method
• Old Router instances are now collectible once nothing else references them
• cache_clear() now only clears the calling Router's own entries

User Flow

This is an SDK-level object lifecycle bug, not something visible in a single request/response pair, so it's written from the seat of a developer embedding litellm.Router in their own long-running process (a proxy wrapper, a multi-tenant service creating one Router per tenant, or a config-reload path that rebuilds the Router).

Before: memory grows without bound as old Router instances pile up

1. The developer builds a Router with a model list and serves at least one request through it (any request touches the same cached helper that set_response_headers calls on every request)
2. They call router.discard(), the method Router itself documents as its cleanup step, then drop every other reference to the Router
3. They inspect the process (a heap snapshot, or just gc.get_referrers on the Router object) and find it is still reachable, held by the cached helper's own internal cache
4. Every Router they ever constructed this way stays alive for the life of the process, even though discard() was called correctly

After: a discarded Router with no other references is actually freed

1. Same steps: build a Router, serve a request through it, call discard(), drop the reference
2. The Router is now collectible; a heap snapshot no longer shows it, and gc.get_referrers returns nothing
3. Only Routers still in active use hold memory; ones a developer has already cleaned up do not accumulate

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is a memory lifecycle bug: there's no request or response that looks different before and after, since the leak only shows up in what stays reachable on the heap after a Router is discarded. A live-proxy curl session would show identical output either way. The honest proof here is a direct reproduction against the Router object itself, run at each commit with uv run python3 -c "...", not a pytest suite.

Before (4b368bf)

Router leaks after discard()

1. Ran: uv run python3 -c "import litellm, weakref, gc; r = litellm.Router(model_list=[{'model_name': 'gpt-4', 'litellm_params': {'model': 'gpt-4', 'api_key': 'fake'}}]); r._cached_get_model_group_info('gpt-4'); r.discard(); ref = weakref.ref(r); del r; gc.collect(); print(ref() is None)"
2. Output: False, the Router is still reachable after discard() and every other reference are dropped

After (7f1296f)

Router leaks after discard()

1. Ran the same command
2. Output: True, the Router is collected once discard() has run and nothing else references it

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Low

• Proof of fix uses a direct object-lifecycle reproduction instead of a live proxy session, since the bug has no request/response signature to curl against

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR