Relevant issues

Investigation of a flaky vertex pass-through e2e test (the one that polls /spend/logs?request_id=<x-litellm-call-id> for a costed row and often times out at 240s even though the write should take seconds)

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This is a logging-durability fix; the cleanest live proof is to drive a real pass-through call through a proxy on localhost:4000 and confirm the SpendLogs row lands, hitting a real provider so it costs real money. With a Gemini key configured for the /gemini pass-through route:

1. Start the proxy

python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log

2. Send a native Gemini pass-through call and capture the call id

CALL_ID=$(curl -sS -D /dev/stderr -o /tmp/resp.json \
  -X POST "http://localhost:4000/gemini/v1beta/models/gemini-2.5-flash:generateContent" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"Say hello in one word"}]}]}' \
  2>&1 1>/dev/null | tr -d '\r' | awk -F': ' 'tolower($1)=="x-litellm-call-id"{print $2}')
echo "call_id=$CALL_ID"

3. A couple seconds later, confirm a costed row exists for that call id

curl -sS "http://localhost:4000/spend/logs?request_id=$CALL_ID" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.[0] | {request_id, call_type, model, spend, status}'

Expect call_type: "pass_through_endpoint", the gemini model, spend > 0, and status: "success". Repeat the loop a few dozen times in a for loop to confirm the row shows up every time rather than intermittently going missing.

Type

🐛 Bug Fix

Changes

Pass-through success logging was scheduled with a bare asyncio.create_task(...) whose return value is discarded, in three places: the non-streaming HTTP path (pass_through_endpoints.py), the streaming finally-block (streaming_handler.py), and the vertex live websocket path. The asyncio event loop keeps only a weak reference to such tasks, so under GC or load pressure the task can be collected before it finishes writing the SpendLogs row. The request then returns 2xx to the caller while its spend log silently never gets written. This is the most likely cause of the flaky vertex pass-through e2e test, and a rare but real source of unbilled pass-through spend in prod.

Worth being precise about the blast radius: for pass-through, the budget/cost increment rides in the same dispatched success-logging callback as the SpendLogs row (it dispatches to _PROXY_track_cost_callback -> db_spend_update_writer.update_database, which both writes the row and bumps key/team/user spend). So a dropped task loses both the audit row and the aggregate spend; the cost never even reaches the durable spend queue. Normal (non-pass-through) completions are unaffected because their callback is awaited inline and reliably enters that durable queue; pass-through is exactly the path that lacked that guarantee.

The fix routes those coroutines through GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue(...), the same async logging worker the SDK completion path already uses (litellm/utils.py, caching_handler.py). The worker keeps a strong reference to each in-flight task in its _running_tasks set and drains on shutdown via flush/stop/clear_queue plus an atexit handler, so the spend write can no longer be dropped mid-flight.

Tradeoffs

This deliberately reuses the existing best-effort worker rather than building a new durable-delivery path, so the pros and cons are worth being explicit about.

Pros. It fixes the GC-drop without changing the non-blocking behavior pass-through wants, since the caller's response is still returned before logging runs. It inherits real shutdown draining instead of losing in-flight tasks on process exit. It bounds concurrency and applies a per-coroutine timeout, so a stuck log can't pile up unbounded. And it follows an established in-repo pattern (the same _running_tasks + add_done_callback(discard) shape used by LoggingWorker and the akto guardrail) rather than hand-rolling another bespoke task set that would need its own shutdown drain to be correct.

Cons. The logging worker is explicitly best-effort with a bounded queue, so under sustained overload it aggressively clears and can still drop logs; this turns a guaranteed-drop-under-GC into a much-less-likely drop-under-overload, but it is not at-least-once delivery. It also swallows per-task exceptions internally, so this does not fix the separate issue where a failing cost-calc / payload-build in the pass-through handlers logs-and-returns and the row never gets queued; that silent-swallow path is worth a follow-up. And pass-through spend logging now shares the process-global worker's queue and concurrency budget with all other async callback logging, so their backpressure is coupled.

Truly durable at-least-once spend logging (persist-before-return outbox + reconciler keyed on request_id) would be a larger, separate change; this PR is the minimal fix for the actual flake and the GC-drop behind it.

Tests

Added two regression tests to test_streaming_handler_interrupt.py asserting that chunk_processor hands its logging coroutine to GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue (rather than a bare task), on both normal completion and client-disconnect. Both fail against the pre-fix asyncio.create_task code (ensure_initialized_and_enqueue called 0 times) and pass after the change.