fix(aiohttp): reset the upstream connection when a stream closes early, so aborted Bedrock Invoke streams stop by princejain756 · Pull Request #50657 · BerriAI/litellm

princejain756 · 2026-11-09T01:27:06Z

Handwritten 1-3 sentence description

When AnthropicMessagesStreamingIterator._abort_upstream_stream() aborts an abandoned /v1/messages stream on bedrock/invoke/*, self.httpx_response was None because AmazonAnthropicClaudeMessagesConfig.get_async_streaming_response_iterator never passed httpx_response into the iterator, and AWSEventStreamDecoder.aiter_bytes() did not close its underlying byte iterator on early exit. Passing httpx_response through and closing byte_iterator in a finally block ensures httpx_response.aclose() closes the upstream Bedrock HTTP stream when a client disconnects.

Details

TLDR

Problem this solves:

Aborted Bedrock Invoke /v1/messages streams keep reading from Bedrock
Bedrock's CloudWatch counts about 1,500 output tokens per aborted Sonnet 4.5 call

How it solves it:

Each stream layer closes its source when it is closed early
The Bedrock HTTP response closes when its byte stream closes
The aiohttp transport resets the socket instead of a slow TLS close
The shared close helpers move into a new stream_close module

User Flow

Before: a client that drops a long streaming answer leaves Bedrock generating, and CloudWatch counts about 1,500 output tokens for that call

The proxy admin sets ANTHROPIC_MESSAGES_MAX_DETACHED_STREAM_DRAINS=0 and serves bedrock/invoke/eu.anthropic.claude-sonnet-4-5-20250929-v1:0
A developer sends POST https://litellm-domain/v1/messages with "stream": true, max_tokens 16384 and a prompt for a long answer
Their client disconnects 7 seconds after the first content_block_delta
Bedrock's CloudWatch OutputTokenCount for that call reads 1,400 to 1,600 tokens

After: the same disconnect closes the Bedrock stream at once, and CloudWatch counts 277 to 314 output tokens

The proxy admin sets ANTHROPIC_MESSAGES_MAX_DETACHED_STREAM_DRAINS=0 and serves bedrock/invoke/eu.anthropic.claude-sonnet-4-5-20250929-v1:0
A developer sends POST https://litellm-domain/v1/messages with "stream": true, max_tokens 16384 and a prompt for a long answer
Their client disconnects 7 seconds after the first content_block_delta
Bedrock's CloudWatch OutputTokenCount for that call reads 277 to 314 tokens

Relevant issues

Fixes #50514

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Screenshots / Proof of Fix

Last updated: 3dab9ff

Both legs build the repo's litellm/deploy/docker/Dockerfile (Python 3.13.18, uvloop 0.22.1) and run 2 workers against real Bedrock in eu-central-1. Before is the merge base dc378de and After is the tip 3dab9ff, whose installed aiohttp_transport.py names _abort_unreleased_connection 3 times to Before's 0

Case	Before	After
1. Sonnet 4.5 CloudWatch output tokens per abort	1569, 1564, 1462	314, 287, 277
2. Sonnet 5.5 socket after disconnect	reads 28 s more	closed at once
3. Sonnet 4.5 full stream, 3 calls	200, end_turn, 126 tokens	200, end_turn, 126 tokens
4. 8 other routes, socket 5 s after abort	8 still reading	0 open

Shared setup: images, config.yaml, env vars, docker run

Each image is built from its own commit's tree

git archive dc378de207 | tar -x -C ctx_base
git archive 3dab9ffab4 | tar -x -C ctx_head
docker build -f ctx_base/litellm/deploy/docker/Dockerfile -t litellm-e6dbfc97-base:local ctx_base
docker build -f ctx_head/litellm/deploy/docker/Dockerfile -t litellm-e6dbfc97-head:local ctx_head

Tree check inside each container

== work-e6dbfc97-abbase
Python 3.13.18
/app/.venv/lib/python3.13/site-packages/litellm/__init__.py
head_symbol_count=0
uvloop 0.22.1
== work-e6dbfc97-abhead
Python 3.13.18
/app/.venv/lib/python3.13/site-packages/litellm/__init__.py
head_symbol_count=3
uvloop 0.22.1

Cases 1 and 3 run on a pair of proxies with no database and this config.yaml

model_list:
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: bedrock/invoke/eu.anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: eu-central-1

Env vars, passed with --env-file: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, LITELLM_MASTER_KEY, ANTHROPIC_MESSAGES_MAX_DETACHED_STREAM_DRAINS=0

docker run -d --name work-e6dbfc97-c1base -p 127.0.0.1:29385:4000 --env-file $E -v $Q/c1base/config.yaml:/app/config.yaml litellm-e6dbfc97-base:local --config /app/config.yaml --port 4000 --num_workers 2
docker run -d --name work-e6dbfc97-c1head -p 127.0.0.1:23368:4000 --env-file $E -v $Q/c1head/config.yaml:/app/config.yaml litellm-e6dbfc97-head:local --config /app/config.yaml --port 4000 --num_workers 2

Cases 2 and 4 run on a second pair of proxies, each with its own Postgres, using this config.yaml. The -rec models go through a recorder that logs and forwards each upstream request

model_list:
  - model_name: claude-sonnet-5-5
    litellm_params:
      model: bedrock/invoke/eu.anthropic.claude-sonnet-5-5
      aws_region_name: eu-central-1
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: bedrock/invoke/eu.anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: eu-central-1
  - model_name: claude-sonnet-5-5-converse
    litellm_params:
      model: bedrock/converse/eu.anthropic.claude-sonnet-5-5
      aws_region_name: eu-central-1
  - model_name: deepseek-v3
    litellm_params:
      model: bedrock/converse/deepseek.v3.2
      aws_region_name: us-east-1
  - model_name: llama4-maverick
    litellm_params:
      model: bedrock/converse/us.meta.llama4-maverick-17b-instruct-v1:0
      aws_region_name: us-east-1
  - model_name: gpt
    litellm_params:
      model: openai/gpt-6.1-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-rec
    litellm_params:
      model: openai/gpt-6.1-terra
      api_key: os.environ/OPENAI_API_KEY
      api_base: http://<docker gateway>:28001/v1
  - model_name: claude-direct
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-direct-rec
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      api_base: http://<docker gateway>:27468

Env vars: the same five plus OPENAI_API_KEY, ANTHROPIC_API_KEY and DATABASE_URL

docker run -d --name work-e6dbfc97-abbase --network work-e6dbfc97-net -p 127.0.0.1:32597:4000 --env-file $E -v $Q/abbase/config.yaml:/app/config.yaml litellm-e6dbfc97-base:local --config /app/config.yaml --port 4000 --num_workers 2
docker run -d --name work-e6dbfc97-abhead --network work-e6dbfc97-net -p 127.0.0.1:28642:4000 --env-file $E -v $Q/abhead/config.yaml:/app/config.yaml litellm-e6dbfc97-head:local --config /app/config.yaml --port 4000 --num_workers 2

Case scripts

Case 1, run as case3.sh base-dc378de207 29385 eu-central-1 and the same for head-3dab9ffab4. Each abort gets its own 6-minute CloudWatch window

#!/bin/zsh
set -u
leg=$1 PORT=$2 REGION=$3
Q=${0:A:h}
D=$Q/$leg
KEY=$(<$Q/master_key)
A=$HOME/.cache/litellm-qa-sso/aws-439-admin-e6dbfc97.env
export PATH=$Q/bin:$PATH
run() { print -r -- "\$ $1"; eval "$1"; }
cw() {
  AWS_ACCESS_KEY_ID=$($HOME/.claude/bin/dotenv-get AWS_ACCESS_KEY_ID --file $A) \
  AWS_SECRET_ACCESS_KEY=$($HOME/.claude/bin/dotenv-get AWS_SECRET_ACCESS_KEY --file $A) \
  AWS_SESSION_TOKEN=$($HOME/.claude/bin/dotenv-get AWS_SESSION_TOKEN --file $A) \
  run "aws cloudwatch get-metric-statistics --region $REGION --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time $1 --end-time $2 --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text"
}
next_minute() { sleep $(( 61 - $(date +%s) % 60 )); }
iso() { date -u -d @$1 +%Y-%m-%dT%H:%M:%SZ; }
PROMPT='Write a long, detailed story of at least 10000 words about a lighthouse keeper on a remote island. Write the whole story in this reply, with no outline, summary, or questions.'
starts=()
print "== leg $leg, proxy http://127.0.0.1:$PORT, Bedrock region $REGION =="
for i in 1 2 3; do
  next_minute; t=$(date +%s); starts+=($t)
  print "\n== abort call $i: max_tokens 16384, client disconnects 7 s after the first content_block_delta ($(iso $t)) =="
  run "curl -sN http://127.0.0.1:$PORT/v1/messages -H \"x-api-key: \$KEY\" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{\"model\":\"claude-sonnet-4-5\",\"max_tokens\":16384,\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}' -o $D/s45_$i.sse & CURL=\$!"
  run "until grep -q '^event: content_block_delta' $D/s45_$i.sse 2>/dev/null || ! kill -0 \$CURL 2>/dev/null; do sleep 0.2; done; print first delta at \$(date -u +%H:%M:%SZ); sleep 7; kill \$CURL; print client disconnected at \$(date -u +%H:%M:%SZ)"
  wait $CURL 2>/dev/null
  run "grep -c '^event: content_block_delta' $D/s45_$i.sse; grep -c '^event: message_stop' $D/s45_$i.sse"
  sleep $(( t + 300 - $(date +%s) ))
done
sleep $(( t + 360 - $(date +%s) ))
print "\n== waiting 3 min for CloudWatch to publish =="
sleep 180
print "\n== Bedrock OutputTokenCount per minute, one 6-minute window per abort call =="
for n in {1..$#starts}; do
  s=$starts[$n]
  print "\n-- window for abort call $n, started $(iso $s)"
  cw $(iso $s) $(iso $(( s + 359 )))
done
print "\nLEG DONE"

Case 2, run as sockprobe_ss.sh base-dc378de207 32597 work-e6dbfc97-abbase and the same for abhead. The sidecar prints bytes_received per :443 socket every 2 s

#!/bin/zsh
set -u
leg=$1 PORT=$2 CONTAINER=$3
Q=${0:A:h}
KEY=$(<$Q/master_key)
mkdir -p $Q/$leg
F=$Q/$leg/probe_$(date +%s).sse
SIDE=e6-ss-$leg
docker rm -f $SIDE >/dev/null 2>&1
docker run -d --name $SIDE --network container:$CONTAINER nicolaka/netshoot:latest sh -c 'while true; do printf "%s " "$(date -u +%H:%M:%SZ)"; ss -tni state established "( dport = :443 )" | grep -o "bytes_received:[0-9]*" | tr "\n" " "; echo; sleep 2; done' >/dev/null
PROMPT='Write a long, detailed story of at least 10000 words about a lighthouse keeper on a remote island. Write the whole story in this reply, with no outline, summary, or questions.'
curl -sN http://127.0.0.1:$PORT/v1/messages -H "x-api-key: $KEY" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d "{\"model\":\"claude-sonnet-5-5\",\"max_tokens\":16384,\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}" -o $F & CURL=$!
until grep -q '^event: content_block_delta' $F 2>/dev/null || ! kill -0 $CURL 2>/dev/null; do sleep 0.2; done
print "first delta $(date -u +%H:%M:%SZ)"; sleep 7; kill $CURL; print "disconnect $(date -u +%H:%M:%SZ)"
sleep 60
print "deltas received by client: $(grep -c '^event: content_block_delta' $F)"
print "-- sidecar ss samples (UTC, bytes_received per established :443 socket)"
docker logs $SIDE 2>&1 | tail -45
docker rm -f $SIDE >/dev/null

Case 3, run as case4.sh base-dc378de207 29385 and the same for head-3dab9ffab4

#!/bin/zsh
set -u
leg=$1 PORT=$2
Q=${0:A:h}
D=$Q/$leg
mkdir -p $D
KEY=$(<$Q/master_key)
run() { print -r -- "\$ $1"; eval "$1"; }
print "== leg $leg, proxy http://127.0.0.1:$PORT =="
for i in 1 2 3; do
  print "\n== full stream $i =="
  run "curl -sN http://127.0.0.1:$PORT/v1/messages -H \"x-api-key: \$KEY\" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{\"model\":\"claude-sonnet-4-5\",\"max_tokens\":300,\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Count from 1 to 60, one number per line, then write DONE.\"}]}' -o $D/full_$i.sse -w 'HTTP %{http_code} in %{time_total}s\n'"
  run "grep -c '^event: content_block_delta' $D/full_$i.sse; grep -c '^event: message_stop' $D/full_$i.sse; grep '^data: {\"type\": \"message_delta\"' $D/full_$i.sse | grep -o -E '\"(stop_reason|output_tokens)\": (\"[a-z_]*\"|[0-9]+)'; grep -o 'DONE' $D/full_$i.sse | head -1"
done
print "\nLEG DONE"

Case 4, run as ab_cells.sh base 32597 and ab_cells.sh head 28642, then ab_analyze.py on each cells folder

#!/bin/zsh
set -u
leg=$1 PORT=$2
Q=${0:A:h}
D=$Q/ab$leg/cells${OUT:-}
mkdir -p $D
KEY=$(<$Q/master_key)
CTR=work-e6dbfc97-ab$leg
SIDE=work-e6dbfc97-ss$leg
LONG='Write a long, detailed story of at least 10000 words about a lighthouse keeper on a remote island. Write the whole story in this reply, with no outline, summary, or questions.'
SHORT='Count from 1 to 20, separated by spaces, and nothing else.'
docker rm -f $SIDE >/dev/null 2>&1
docker run -d --name $SIDE --network container:$CTR nicolaka/netshoot:latest sh -c 'while true; do t=$(date -u +%s.%N); ss -tnHi state established "( dport = :443 )" | paste - - | while read -r l; do echo "$t $(echo "$l" | awk "{print \$3, \$4}") $(echo "$l" | grep -o "bytes_received:[0-9]*")"; done; echo "$t TICK"; sleep 0.5; done' >/dev/null
now() { date -u +%s.%N; }
mark() { print -r -- "MARK $(now) $leg $*" | tee -a $D/marks.log; }
abort_cell() {
  [[ -n ${ONLY:-} && ! $1 =~ $ONLY ]] && return 0
  local name=$1 route=$2 body=$3 extra=$4 pat=$5
  local F=$D/$name.sse
  mark "$name start"
  curl -sN "http://127.0.0.1:$PORT$route" -H "Authorization: Bearer $KEY" -H 'content-type: application/json' ${=extra} -d "$body" -o $F & local C=$!
  local deadline=$(( $(date +%s) + 120 ))
  until (( $(grep -c -E "$pat" $F 2>/dev/null || print 0) >= 3 )) || ! kill -0 $C 2>/dev/null || (( $(date +%s) > deadline )); do sleep 0.1; done
  mark "$name first_chunks"
  sleep 3
  kill $C 2>/dev/null; wait $C 2>/dev/null
  mark "$name disconnect"
  print -r -- "CELL $name leg=$leg client_events=$(grep -c -E "$pat" $F) bytes=$(wc -c < $F) done_marker=$(grep -c -E '\[DONE\]|message_stop|response.completed' $F)" | tee -a $D/summary.log
  sleep 36
  mark "$name settled"
}
full_cell() {
  [[ -n ${ONLY:-} && ! $1 =~ $ONLY ]] && return 0
  local name=$1 route=$2 body=$3 extra=$4
  local F=$D/$name.sse
  mark "$name start"
  local out=$(curl -sN "http://127.0.0.1:$PORT$route" -H "Authorization: Bearer $KEY" -H 'content-type: application/json' ${=extra} -d "$body" -o $F -w '%{http_code} %{time_total}')
  mark "$name end"
  print -r -- "CELL $name leg=$leg status_time=$out bytes=$(wc -c < $F) done_marker=$(grep -c -E '\[DONE\]|message_stop|response.completed' $F)" | tee -a $D/summary.log
}
docker exec $CTR sh -c 'python3 --version; python3 -c "import litellm,sys; print(litellm.__file__)"; F=$(python3 -c "import litellm.llms.custom_httpx.aiohttp_transport as m; print(m.__file__)"); echo head_symbol_count=$(grep -c _abort_unreleased_connection $F); python3 -c "import uvloop; print(\"uvloop\", uvloop.__version__)"' 2>&1 | tee $D/tree.log
CHAT_HDR=''
MSG_HDR='-H anthropic-version:2023-06-01'
abort_cell A_openai_chat_abort /v1/chat/completions "{\"model\":\"gpt\",\"stream\":true,\"max_completion_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
full_cell B_openai_full_rec /v1/chat/completions "{\"model\":\"gpt-rec\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"
for i in {1..10}; do full_cell B_openai_full_seq_$i /v1/chat/completions "{\"model\":\"gpt\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"; done
sleep 3
abort_cell C_anthropic_chat_abort /v1/chat/completions "{\"model\":\"claude-direct\",\"stream\":true,\"max_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
abort_cell C_anthropic_messages_abort /v1/messages "{\"model\":\"claude-direct\",\"stream\":true,\"max_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$MSG_HDR" '^event: content_block_delta'
full_cell D_anthropic_full_rec /v1/chat/completions "{\"model\":\"claude-direct-rec\",\"stream\":true,\"max_tokens\":200,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"
for i in {1..10}; do full_cell D_anthropic_full_seq_$i /v1/chat/completions "{\"model\":\"claude-direct\",\"stream\":true,\"max_tokens\":200,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"; done
sleep 3
full_cell E_bedrock_invoke_chat_full /v1/chat/completions "{\"model\":\"claude-sonnet-5-5\",\"stream\":true,\"max_tokens\":200,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"
abort_cell E_bedrock_invoke_chat_abort /v1/chat/completions "{\"model\":\"claude-sonnet-5-5\",\"stream\":true,\"max_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
full_cell F_converse_deepseek_full /v1/chat/completions "{\"model\":\"deepseek-v3\",\"stream\":true,\"max_tokens\":200,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"
abort_cell F_converse_deepseek_abort /v1/chat/completions "{\"model\":\"deepseek-v3\",\"stream\":true,\"max_tokens\":8192,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
full_cell F_converse_llama_full /v1/chat/completions "{\"model\":\"llama4-maverick\",\"stream\":true,\"max_tokens\":200,\"messages\":[{\"role\":\"user\",\"content\":\"$SHORT\"}]}" "$CHAT_HDR"
abort_cell F_converse_llama_abort /v1/chat/completions "{\"model\":\"llama4-maverick\",\"stream\":true,\"max_tokens\":8192,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
abort_cell F_converse_claude_abort /v1/chat/completions "{\"model\":\"claude-sonnet-5-5-converse\",\"stream\":true,\"max_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$CHAT_HDR" '^data: \{'
abort_cell G_responses_abort /v1/responses "{\"model\":\"gpt\",\"stream\":true,\"max_output_tokens\":16384,\"input\":\"$LONG\"}" "$CHAT_HDR" '^data: \{'
abort_cell G_passthrough_anthropic_abort /anthropic/v1/messages "{\"model\":\"claude-sonnet-5-5\",\"stream\":true,\"max_tokens\":16384,\"messages\":[{\"role\":\"user\",\"content\":\"$LONG\"}]}" "$MSG_HDR" '^event: content_block_delta'
[[ -n ${ONLY:-} ]] || sleep 70
docker logs $SIDE > $D/ss.log 2>&1
docker rm -f $SIDE >/dev/null
docker exec work-e6dbfc97-pg$leg psql -U postgres -d litellm -At -F ' | ' -c 'select to_char("startTime", '"'"'HH24:MI:SS'"'"'), model_group, call_type, status, total_tokens, completion_tokens, round(spend::numeric, 6) from "LiteLLM_SpendLogs" order by "startTime"' > $D/spend.log 2>&1
print -r -- "LEG DONE $leg"

import sys
from pathlib import Path

cells_dir = Path(sys.argv[1])
marks = {}
for line in (cells_dir / "marks.log").read_text().splitlines():
    _, t, _, name, event = line.split(" ", 4)
    marks.setdefault(name, {})[event] = float(t)

samples = {}
for line in (cells_dir / "ss.log").read_text().splitlines():
    parts = line.split()
    if len(parts) < 2:
        continue
    t = float(parts[0])
    tick = samples.setdefault(t, {})
    if parts[1] == "TICK" or len(parts) < 4:
        continue
    tick[parts[1]] = int(parts[3].split(":")[1])
times = sorted(samples)


def at(t):
    before = [x for x in times if x <= t]
    return samples[before[-1]] if before else {}


def window_conns(start, end):
    return {sock for x in times if start <= x <= end for sock in samples[x]}


for name, ev in marks.items():
    if "disconnect" in ev:
        d = ev["disconnect"]
        at_disconnect = at(d)
        cells = []
        for offset in (0.5, 5, 35):
            snap = at(d + offset)
            grown = sum(max(0, snap.get(s, 0) - b) for s, b in at_disconnect.items() if s in snap)
            cells.append(f"+{offset}s open={sum(1 for s in at_disconnect if s in snap)} rx_since={grown}")
        print(f"{name}: sockets_at_disconnect={len(at_disconnect)} " + " ".join(cells))
    elif "end" in ev and name.endswith(("_seq_1", "_rec")):
        pass
seq_groups = {}
for name, ev in marks.items():
    if "_full_seq_" in name:
        prefix = name.rsplit("_", 1)[0]
        g = seq_groups.setdefault(prefix, [ev["start"], ev["end"]])
        g[0], g[1] = min(g[0], ev["start"]), max(g[1], ev["end"])
for prefix, (s, e) in seq_groups.items():
    print(f"{prefix}x10: distinct_443_connections={len(window_conns(s, e + 0.6))}")

Before (`dc378de`)

Case 1: Sonnet 4.5 CloudWatch output tokens per abort

Three abort calls, each cut 7 s after the first delta. The client got 140, 148 and 135 deltas
CloudWatch OutputTokenCount per call window reads 1569, 1564 and 1462 tokens

Case 2: Sonnet 5.5 upstream socket after the client disconnects

The client disconnects at 11:26:03Z after 211 deltas
The proxy's socket to Bedrock keeps receiving until 11:26:31Z, 28 s and about 255 KB later

Case 3: Sonnet 4.5 full stream

Three calls with max_tokens 300 each return HTTP 200, one message_stop, end_turn and 126 output tokens

Case 4: the other streaming routes on the same transport

Aborts on OpenAI chat, Anthropic chat, Bedrock Invoke chat, Converse (DeepSeek, Llama, Claude), OpenAI /v1/responses and /anthropic pass-through
On all 8, the upstream socket still receives 5 s after the disconnect and closes by 35 s. Anthropic /v1/messages already stops reading at once

Raw output, Before

== leg base-dc378de207, proxy http://127.0.0.1:29385, Bedrock region eu-central-1 ==

== abort call 1: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T11:26:01Z) ==
$ curl -sN http://127.0.0.1:29385/v1/messages -H "x-api-key: $KEY" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{"model":"claude-sonnet-4-5","max_tokens":16384,"stream":true,"messages":[{"role":"user","content":"Write a long, detailed story of at least 10000 words about a lighthouse keeper on a remote island. Write the whole story in this reply, with no outline, summary, or questions."}]}' -o $Q/base-dc378de207/s45_1.sse & CURL=$!
$ until grep -q '^event: content_block_delta' $Q/base-dc378de207/s45_1.sse 2>/dev/null || ! kill -0 $CURL 2>/dev/null; do sleep 0.2; done; print first delta at $(date -u +%H:%M:%SZ); sleep 7; kill $CURL; print client disconnected at $(date -u +%H:%M:%SZ)
first delta at 11:26:03Z
client disconnected at 11:26:10Z
$ grep -c '^event: content_block_delta' $Q/base-dc378de207/s45_1.sse; grep -c '^event: message_stop' $Q/base-dc378de207/s45_1.sse
140
0

== abort call 2: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T11:32:01Z) ==
first delta at 11:32:05Z
client disconnected at 11:32:12Z
148
0

== abort call 3: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T11:38:01Z) ==
first delta at 11:38:03Z
client disconnected at 11:38:10Z
135
0

== waiting 3 min for CloudWatch to publish ==

== Bedrock OutputTokenCount per minute, one 6-minute window per abort call ==

-- window for abort call 1, started 2026-11-14T11:26:01Z
$ aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time 2026-11-14T11:26:01Z --end-time 2026-11-14T11:32:00Z --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
1794655560.0	1569.0

-- window for abort call 2, started 2026-11-14T11:32:01Z
1794655920.0	1564.0

-- window for abort call 3, started 2026-11-14T11:38:01Z
1794656280.0	1462.0

LEG DONE

first delta 11:25:56Z
disconnect 11:26:03Z
deltas received by client: 211
$Q/sockprobe_ss.sh:print:17: bad option: - 
11:25:28Z bytes_received:5956 
11:25:30Z bytes_received:5956 
11:25:33Z bytes_received:5956 
11:25:35Z bytes_received:5956 
11:25:37Z bytes_received:5956 
11:25:39Z bytes_received:5956 
11:25:41Z bytes_received:5956 
11:25:43Z bytes_received:5956 
11:25:45Z bytes_received:5956 
11:25:47Z bytes_received:5956 
11:25:49Z bytes_received:5956 
11:25:51Z bytes_received:5956 
11:25:54Z bytes_received:5956 
11:25:56Z bytes_received:19451 
11:25:58Z bytes_received:38838 
11:26:00Z bytes_received:57375 
11:26:02Z bytes_received:75763 
11:26:04Z bytes_received:94053 
11:26:06Z bytes_received:110201 
11:26:08Z bytes_received:128804 
11:26:10Z bytes_received:147275 
11:26:13Z bytes_received:165988 
11:26:15Z bytes_received:184631 
11:26:17Z bytes_received:203161 
11:26:19Z bytes_received:221596 
11:26:21Z bytes_received:240434 
11:26:23Z bytes_received:259091 
11:26:25Z bytes_received:275571 
11:26:27Z bytes_received:293973 
11:26:29Z bytes_received:312576 
11:26:31Z bytes_received:330889 
11:26:34Z 
11:26:36Z 
11:26:38Z 
11:26:40Z 
11:26:42Z 
11:26:44Z 
11:26:46Z 
11:26:48Z 
11:26:50Z 
11:26:53Z 
11:26:55Z 
11:26:57Z 
11:26:59Z 
11:27:01Z

== leg base-dc378de207, proxy http://127.0.0.1:29385 ==

== full stream 1 ==
$ curl -sN http://127.0.0.1:29385/v1/messages -H "x-api-key: $KEY" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{"model":"claude-sonnet-4-5","max_tokens":300,"stream":true,"messages":[{"role":"user","content":"Count from 1 to 60, one number per line, then write DONE."}]}' -o $Q/base-dc378de207/full_1.sse -w 'HTTP %{http_code} in %{time_total}s\n'
HTTP 200 in 5.588187s
$ grep -c '^event: content_block_delta' $Q/base-dc378de207/full_1.sse; grep -c '^event: message_stop' $Q/base-dc378de207/full_1.sse; grep '^data: {"type": "message_delta"' $Q/base-dc378de207/full_1.sse | grep -o -E '"(stop_reason|output_tokens)": ("[a-z_]*"|[0-9]+)'; grep -o 'DONE' $Q/base-dc378de207/full_1.sse | head -1
16
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

== full stream 2 ==
HTTP 200 in 3.504057s
16
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

== full stream 3 ==
HTTP 200 in 2.367693s
16
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

LEG DONE

DeepSeek reran after a model id fix, and the Responses abort ran twice more

== cells
CELL A_openai_chat_abort leg=base client_events=153 bytes=34750 done_marker=0
CELL B_openai_full_rec leg=base status_time=200 1.461757 bytes=9507 done_marker=1
CELL B_openai_full_seq_1 leg=base status_time=200 1.180647 bytes=9343 done_marker=1
CELL B_openai_full_seq_2 leg=base status_time=200 1.325688 bytes=9343 done_marker=1
CELL B_openai_full_seq_3 leg=base status_time=200 2.907529 bytes=9343 done_marker=1
CELL B_openai_full_seq_4 leg=base status_time=200 1.083820 bytes=9343 done_marker=1
CELL B_openai_full_seq_5 leg=base status_time=200 1.311854 bytes=9343 done_marker=1
CELL B_openai_full_seq_6 leg=base status_time=200 1.097221 bytes=9343 done_marker=1
CELL B_openai_full_seq_7 leg=base status_time=200 1.253751 bytes=9343 done_marker=1
CELL B_openai_full_seq_8 leg=base status_time=200 1.210322 bytes=9343 done_marker=1
CELL B_openai_full_seq_9 leg=base status_time=200 1.335446 bytes=9343 done_marker=1
CELL B_openai_full_seq_10 leg=base status_time=200 1.095298 bytes=9343 done_marker=1
CELL C_anthropic_chat_abort leg=base client_events=96 bytes=49039 done_marker=0
CELL C_anthropic_messages_abort leg=base client_events=95 bytes=28246 done_marker=0
CELL D_anthropic_full_rec leg=base status_time=200 1.694354 bytes=1240 done_marker=1
CELL D_anthropic_full_seq_1 leg=base status_time=200 1.631587 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_2 leg=base status_time=200 1.219075 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_3 leg=base status_time=200 1.198627 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_4 leg=base status_time=200 1.156441 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_5 leg=base status_time=200 1.182581 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_6 leg=base status_time=200 1.153822 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_7 leg=base status_time=200 1.175278 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_8 leg=base status_time=200 1.209474 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_9 leg=base status_time=200 1.166234 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_10 leg=base status_time=200 1.191993 bytes=1216 done_marker=1
CELL E_bedrock_invoke_chat_full leg=base status_time=200 1.115248 bytes=1240 done_marker=1
CELL E_bedrock_invoke_chat_abort leg=base client_events=93 bytes=63125 done_marker=0
CELL F_converse_deepseek_full leg=base status_time=400 2.371449 bytes=278 done_marker=0
CELL F_converse_deepseek_abort leg=base client_events=0 bytes=278 done_marker=0
CELL F_converse_llama_full leg=base status_time=200 0.625035 bytes=2284 done_marker=1
CELL F_converse_llama_abort leg=base client_events=230 bytes=52713 done_marker=0
CELL F_converse_claude_abort leg=base client_events=105 bytes=54121 done_marker=0
CELL G_responses_abort leg=base client_events=3 bytes=4632 done_marker=0
CELL G_passthrough_anthropic_abort leg=base client_events=94 bytes=33889 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
A_openai_chat_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=18347 +5s open=1 rx_since=75429 +35s open=0 rx_since=0
C_anthropic_chat_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=7027 +35s open=0 rx_since=0
C_anthropic_messages_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
E_bedrock_invoke_chat_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=43235 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=2 +0.5s open=2 rx_since=0 +5s open=2 rx_since=0 +35s open=0 rx_since=0
F_converse_llama_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=78509 +35s open=0 rx_since=0
F_converse_claude_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=5133 +5s open=1 rx_since=31423 +35s open=0 rx_since=0
G_responses_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=0 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=964 +5s open=1 rx_since=7508 +35s open=0 rx_since=0
B_openai_full_seqx10: distinct_443_connections=7
D_anthropic_full_seqx10: distinct_443_connections=2

== cells_deepseek
CELL F_converse_deepseek_full leg=base status_time=200 3.483480 bytes=1169 done_marker=1
CELL F_converse_deepseek_abort leg=base client_events=17 bytes=5220 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
F_converse_deepseek_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=1515 +5s open=1 rx_since=7287 +35s open=0 rx_since=0

== cells_resp1
CELL G_responses_abort leg=base client_events=3 bytes=4632 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
G_responses_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=60759 +35s open=0 rx_since=0

== cells_resp2
CELL G_responses_abort leg=base client_events=3 bytes=4612 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
G_responses_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=16345 +5s open=1 rx_since=67501 +35s open=0 rx_since=0

-- LiteLLM_SpendLogs rows after the last run (cells_resp2)
11:25:23 | claude-sonnet-5-5 | anthropic_messages | success | 512 | 448 | 0.005069
11:27:59 | gpt | acompletion | success | 198 | 152 | 0.000806
11:28:42 | gpt-rec | acompletion | success | 64 | 42 | 0.000232
11:28:44 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:45 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:47 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:49 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:51 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:52 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:53 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:55 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:56 | gpt | acompletion | success | 64 | 42 | 0.000232
11:28:57 | gpt | acompletion | success | 64 | 42 | 0.000232
11:29:02 | claude-direct | acompletion | success | 274 | 210 | 0.002228
11:30:21 | claude-direct | anthropic_messages | success | 299 | 235 | 0.002478
11:31:40 | claude-direct-rec | acompletion | success | 72 | 41 | 0.000472
11:31:42 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:43 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:44 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:46 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:47 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:48 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:49 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:51 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:52 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:54 | claude-direct | acompletion | success | 72 | 41 | 0.000472
11:31:58 | claude-sonnet-5-5 | acompletion | success | 72 | 41 | 0.000519
11:31:59 | claude-sonnet-5-5 | acompletion | success | 72 | 8 | 0.000229
11:33:44 | deepseek-v3 | acompletion | failure | 23 | 0 | 0.000000
11:33:46 | deepseek-v3 | acompletion | failure | 46 | 0 | 0.000000
11:34:27 | llama4-maverick | acompletion | success | 91 | 40 | 0.000051
11:34:28 | llama4-maverick | acompletion | success | 546 | 499 | 0.000495
11:35:08 | claude-sonnet-5-5-converse | acompletion | success | 278 | 231 | 0.002644
11:37:08 | claude-sonnet-5-5 | pass_through_endpoint | success | 355 | 291 | 0.003038
12:08:02 | deepseek-v3 | acompletion | success | 60 | 40 | 0.000086
12:08:05 | deepseek-v3 | acompletion | success | 394 | 348 | 0.000672

DeepSeek, pass-through and Anthropic `/v1/messages` ran twice more, since a first run had no stream socket at the cut
== cells_r2a
CELL C_anthropic_messages_abort leg=base client_events=102 bytes=49070 done_marker=0
CELL F_converse_deepseek_abort leg=base client_events=25 bytes=7318 done_marker=0
CELL G_passthrough_anthropic_abort leg=base client_events=96 bytes=43482 done_marker=0
C_anthropic_messages_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=5438 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=1705 +5s open=1 rx_since=7484 +35s open=0 rx_since=0
== cells_r2b
CELL C_anthropic_messages_abort leg=base client_events=97 bytes=31163 done_marker=0
CELL F_converse_deepseek_abort leg=base client_events=14 bytes=3714 done_marker=0
CELL G_passthrough_anthropic_abort leg=base client_events=99 bytes=33191 done_marker=0
C_anthropic_messages_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=485 +5s open=1 rx_since=1606 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=1 rx_since=7447 +35s open=0 rx_since=0
-- their spend rows
13:43:01 | claude-direct | anthropic_messages | success | 323 | 259 | 0.002718
13:45:19 | deepseek-v3 | acompletion | success | 468 | 422 | 0.000809
13:46:04 | claude-sonnet-5-5 | pass_through_endpoint | success | 351 | 287 | 0.002998
13:48:38 | claude-direct | anthropic_messages | success | 268 | 204 | 0.002168
13:50:08 | deepseek-v3 | acompletion | success | 202 | 156 | 0.000317
13:50:51 | claude-sonnet-5-5 | pass_through_endpoint | success | 303 | 239 | 0.002518

After (`3dab9ff`)

Case 1: Sonnet 4.5 CloudWatch output tokens per abort

The same three abort calls. The client got 143, 133 and 132 deltas
CloudWatch OutputTokenCount per call window reads 314, 287 and 277 tokens

Case 2: Sonnet 5.5 upstream socket after the client disconnects

The client disconnects at 12:27:22Z after 211 deltas
The socket is gone in the next sample at 12:27:23Z

Case 3: Sonnet 4.5 full stream

The same as Before

Case 4: the other streaming routes on the same transport

The same 8 aborts plus Anthropic /v1/messages
No upstream socket receives a byte after the disconnect, and all close within 5 s
Full streams, connection reuse over 10 calls, recorded upstream requests and spend rows match Before

Raw output, After

== leg head-3dab9ffab4, proxy http://127.0.0.1:23368, Bedrock region eu-central-1 ==

== abort call 1: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T12:28:01Z) ==
$ curl -sN http://127.0.0.1:23368/v1/messages -H "x-api-key: $KEY" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{"model":"claude-sonnet-4-5","max_tokens":16384,"stream":true,"messages":[{"role":"user","content":"Write a long, detailed story of at least 10000 words about a lighthouse keeper on a remote island. Write the whole story in this reply, with no outline, summary, or questions."}]}' -o $Q/head-3dab9ffab4/s45_1.sse & CURL=$!
$ until grep -q '^event: content_block_delta' $Q/head-3dab9ffab4/s45_1.sse 2>/dev/null || ! kill -0 $CURL 2>/dev/null; do sleep 0.2; done; print first delta at $(date -u +%H:%M:%SZ); sleep 7; kill $CURL; print client disconnected at $(date -u +%H:%M:%SZ)
first delta at 12:28:05Z
client disconnected at 12:28:12Z
$ grep -c '^event: content_block_delta' $Q/head-3dab9ffab4/s45_1.sse; grep -c '^event: message_stop' $Q/head-3dab9ffab4/s45_1.sse
143
0

== abort call 2: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T12:34:01Z) ==
first delta at 12:34:03Z
client disconnected at 12:34:10Z
133
0

== abort call 3: max_tokens 16384, client disconnects 7 s after the first content_block_delta (2026-11-14T12:40:01Z) ==
first delta at 12:40:02Z
client disconnected at 12:40:09Z
132
0

== waiting 3 min for CloudWatch to publish ==

== Bedrock OutputTokenCount per minute, one 6-minute window per abort call ==

-- window for abort call 1, started 2026-11-14T12:28:01Z
$ aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time 2026-11-14T12:28:01Z --end-time 2026-11-14T12:34:00Z --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text

An error occurred (ExpiredToken) when calling the GetMetricStatistics operation: The security token included in the request is expired

-- window for abort call 2, started 2026-11-14T12:34:01Z

An error occurred (ExpiredToken) when calling the GetMetricStatistics operation: The security token included in the request is expired

-- window for abort call 3, started 2026-11-14T12:40:01Z

An error occurred (ExpiredToken) when calling the GetMetricStatistics operation: The security token included in the request is expired

LEG DONE

CloudWatch re-read after a token renewal

== CloudWatch re-read for the After abort calls at 2026-11-14T16:28:33Z, admin role creds ==
-- window starting 2026-11-14T12:28:01Z
$ aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time 2026-11-14T12:28:01Z --end-time 2026-11-14T12:34:00Z --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
1794659280.0	314.0
-- window starting 2026-11-14T12:34:01Z
$ aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time 2026-11-14T12:34:01Z --end-time 2026-11-14T12:40:00Z --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
1794659640.0	287.0
-- window starting 2026-11-14T12:40:01Z
$ aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Bedrock --metric-name OutputTokenCount --dimensions Name=ModelId,Value=eu.anthropic.claude-sonnet-4-5-20250929-v1:0 --start-time 2026-11-14T12:40:01Z --end-time 2026-11-14T12:46:00Z --period 60 --statistics Sum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
1794660000.0	277.0

first delta 12:27:15Z
disconnect 12:27:22Z
deltas received by client: 211
$Q/sockprobe_ss.sh:print:17: bad option: - 
12:26:54Z 
12:26:56Z bytes_received:4559 
12:26:59Z bytes_received:5911 
12:27:01Z bytes_received:5911 
12:27:03Z bytes_received:5911 
12:27:05Z bytes_received:5911 
12:27:07Z bytes_received:5911 
12:27:09Z bytes_received:5911 
12:27:11Z bytes_received:5911 
12:27:13Z bytes_received:5911 
12:27:15Z bytes_received:16136 
12:27:17Z bytes_received:34569 
12:27:19Z bytes_received:50946 
12:27:21Z bytes_received:69571 
12:27:23Z 
12:27:25Z 
12:27:27Z 
12:27:30Z 
12:27:32Z 
12:27:34Z 
12:27:36Z 
12:27:38Z 
12:27:40Z 
12:27:42Z 
12:27:44Z 
12:27:46Z 
12:27:48Z 
12:27:50Z 
12:27:52Z 
12:27:54Z 
12:27:56Z 
12:27:58Z 
12:28:00Z 
12:28:02Z 
12:28:05Z 
12:28:07Z 
12:28:09Z 
12:28:11Z 
12:28:13Z 
12:28:15Z 
12:28:17Z 
12:28:19Z 
12:28:21Z

== leg head-3dab9ffab4, proxy http://127.0.0.1:23368 ==

== full stream 1 ==
$ curl -sN http://127.0.0.1:23368/v1/messages -H "x-api-key: $KEY" -H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' -d '{"model":"claude-sonnet-4-5","max_tokens":300,"stream":true,"messages":[{"role":"user","content":"Count from 1 to 60, one number per line, then write DONE."}]}' -o $Q/head-3dab9ffab4/full_1.sse -w 'HTTP %{http_code} in %{time_total}s\n'
HTTP 200 in 5.424914s
$ grep -c '^event: content_block_delta' $Q/head-3dab9ffab4/full_1.sse; grep -c '^event: message_stop' $Q/head-3dab9ffab4/full_1.sse; grep '^data: {"type": "message_delta"' $Q/head-3dab9ffab4/full_1.sse | grep -o -E '"(stop_reason|output_tokens)": ("[a-z_]*"|[0-9]+)'; grep -o 'DONE' $Q/head-3dab9ffab4/full_1.sse | head -1
16
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

== full stream 2 ==
HTTP 200 in 3.473883s
17
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

== full stream 3 ==
HTTP 200 in 2.157983s
16
1
"stop_reason": "end_turn"
"output_tokens": 126
DONE

LEG DONE

== cells
CELL A_openai_chat_abort leg=head client_events=177 bytes=40200 done_marker=0
CELL B_openai_full_rec leg=head status_time=200 1.224864 bytes=9507 done_marker=1
CELL B_openai_full_seq_1 leg=head status_time=200 1.158426 bytes=9343 done_marker=1
CELL B_openai_full_seq_2 leg=head status_time=200 1.290612 bytes=9343 done_marker=1
CELL B_openai_full_seq_3 leg=head status_time=200 2.538456 bytes=9343 done_marker=1
CELL B_openai_full_seq_4 leg=head status_time=200 1.449250 bytes=9343 done_marker=1
CELL B_openai_full_seq_5 leg=head status_time=200 1.217118 bytes=9343 done_marker=1
CELL B_openai_full_seq_6 leg=head status_time=200 1.197528 bytes=9343 done_marker=1
CELL B_openai_full_seq_7 leg=head status_time=200 1.450804 bytes=9343 done_marker=1
CELL B_openai_full_seq_8 leg=head status_time=200 1.351527 bytes=9343 done_marker=1
CELL B_openai_full_seq_9 leg=head status_time=200 1.304830 bytes=9343 done_marker=1
CELL B_openai_full_seq_10 leg=head status_time=200 1.159135 bytes=9343 done_marker=1
CELL C_anthropic_chat_abort leg=head client_events=94 bytes=48411 done_marker=0
CELL C_anthropic_messages_abort leg=head client_events=96 bytes=29148 done_marker=0
CELL D_anthropic_full_rec leg=head status_time=200 1.717938 bytes=1240 done_marker=1
CELL D_anthropic_full_seq_1 leg=head status_time=200 1.262843 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_2 leg=head status_time=200 1.438367 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_3 leg=head status_time=200 1.186698 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_4 leg=head status_time=200 1.189321 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_5 leg=head status_time=200 1.175646 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_6 leg=head status_time=200 1.267900 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_7 leg=head status_time=200 1.321462 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_8 leg=head status_time=200 1.253008 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_9 leg=head status_time=200 1.174537 bytes=1216 done_marker=1
CELL D_anthropic_full_seq_10 leg=head status_time=200 1.186639 bytes=1216 done_marker=1
CELL E_bedrock_invoke_chat_full leg=head status_time=200 1.422239 bytes=1240 done_marker=1
CELL E_bedrock_invoke_chat_abort leg=head client_events=95 bytes=54517 done_marker=0
CELL F_converse_deepseek_full leg=head status_time=200 1.817832 bytes=1599 done_marker=1
CELL F_converse_deepseek_abort leg=head client_events=16 bytes=4209 done_marker=0
CELL F_converse_llama_full leg=head status_time=200 0.656294 bytes=2284 done_marker=1
CELL F_converse_llama_abort leg=head client_events=243 bytes=55845 done_marker=0
CELL F_converse_claude_abort leg=head client_events=97 bytes=62953 done_marker=0
CELL G_responses_abort leg=head client_events=47 bytes=20036 done_marker=0
CELL G_passthrough_anthropic_abort leg=head client_events=98 bytes=29877 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
A_openai_chat_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
C_anthropic_chat_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
C_anthropic_messages_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
E_bedrock_invoke_chat_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_llama_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_claude_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
G_responses_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
B_openai_full_seqx10: distinct_443_connections=7
D_anthropic_full_seqx10: distinct_443_connections=2

== cells_resp1
CELL G_responses_abort leg=head client_events=3 bytes=4632 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
G_responses_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0

== cells_resp2
CELL G_responses_abort leg=head client_events=48 bytes=20392 done_marker=0
-- upstream :443 sockets per abort cell, from ss.log and marks.log
G_responses_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0

-- LiteLLM_SpendLogs rows after the last run (cells_resp2)
12:26:55 | claude-sonnet-5-5 | anthropic_messages | success | 523 | 459 | 0.005190
12:28:50 | gpt | acompletion | success | 222 | 176 | 0.000926
12:29:34 | gpt-rec | acompletion | success | 64 | 42 | 0.000232
12:29:35 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:36 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:38 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:40 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:41 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:43 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:44 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:46 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:47 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:49 | gpt | acompletion | success | 64 | 42 | 0.000232
12:29:53 | claude-direct | acompletion | success | 300 | 236 | 0.002488
12:31:14 | claude-direct | anthropic_messages | success | 273 | 209 | 0.002218
12:32:37 | claude-direct-rec | acompletion | success | 72 | 41 | 0.000472
12:32:39 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:40 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:42 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:43 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:44 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:46 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:47 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:48 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:50 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:51 | claude-direct | acompletion | success | 72 | 41 | 0.000472
12:32:55 | claude-sonnet-5-5 | acompletion | success | 72 | 41 | 0.000519
12:32:57 | claude-sonnet-5-5 | acompletion | success | 72 | 8 | 0.000229
12:34:26 | deepseek-v3 | acompletion | success | 60 | 40 | 0.000086
12:34:28 | deepseek-v3 | acompletion | success | 235 | 189 | 0.000378
12:35:10 | llama4-maverick | acompletion | success | 91 | 40 | 0.000051
12:35:10 | llama4-maverick | acompletion | success | 620 | 573 | 0.000567
12:35:50 | claude-sonnet-5-5-converse | acompletion | success | 261 | 214 | 0.002457
12:37:22 | gpt | aresponses | success | 87 | 41 | 0.000251
12:38:03 | claude-sonnet-5-5 | pass_through_endpoint | success | 305 | 241 | 0.002538
13:00:07 | gpt | aresponses | success | 88 | 42 | 0.000256

DeepSeek, pass-through and Anthropic `/v1/messages` ran twice more, since a first run had no stream socket at the cut
== cells_r2a
CELL C_anthropic_messages_abort leg=head client_events=112 bytes=33699 done_marker=0
CELL F_converse_deepseek_abort leg=head client_events=25 bytes=7587 done_marker=0
CELL G_passthrough_anthropic_abort leg=head client_events=77 bytes=55271 done_marker=0
C_anthropic_messages_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
== cells_r2b
CELL C_anthropic_messages_abort leg=head client_events=94 bytes=40135 done_marker=0
CELL F_converse_deepseek_abort leg=head client_events=18 bytes=5477 done_marker=0
CELL G_passthrough_anthropic_abort leg=head client_events=95 bytes=47090 done_marker=0
C_anthropic_messages_abort: sockets_at_disconnect=0 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
F_converse_deepseek_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
G_passthrough_anthropic_abort: sockets_at_disconnect=1 +0.5s open=0 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0
-- their spend rows
13:43:14 | claude-direct | anthropic_messages | success | 330 | 266 | 0.002788
13:44:49 | deepseek-v3 | acompletion | success | 543 | 497 | 0.000948
13:45:32 | claude-sonnet-5-5 | pass_through_endpoint | success | 306 | 242 | 0.002548
13:48:48 | claude-direct | anthropic_messages | success | 316 | 252 | 0.002648
13:50:45 | deepseek-v3 | acompletion | success | 406 | 360 | 0.000695
13:51:26 | claude-sonnet-5-5 | pass_through_endpoint | success | 307 | 243 | 0.002558

Recorded upstream requests of the -rec cells, Before against After, from python3 rec_diff.py abbase abhead

rec_oai.jsonl: Before 1 POST, After 1 POST
  POST /v1/chat/completions -> 200 | POST /v1/chat/completions -> 200
  header keys equal: True, header values equal: True
  body equal: True, body keys: ['messages', 'model', 'stream', 'stream_options']
rec_ant.jsonl: Before 1 POST, After 1 POST
  POST /v1/messages -> 200 | POST /v1/messages -> 200
  header keys equal: True, header values equal: True
  body equal: True, body keys: ['max_tokens', 'messages', 'model', 'stream']

httpx control, the After image with DISABLE_AIOHTTP_TRANSPORT=True

CELL A_openai_chat_abort leg=noaio client_events=185 bytes=42028 done_marker=0
A_openai_chat_abort: sockets_at_disconnect=1 +0.5s open=1 rx_since=0 +5s open=0 rx_since=0 +35s open=0 rx_since=0

Things the QA run noticed. The PR causes none of them

Only uvloop keeps reading, the default asyncio loop closes promptly
A /v1/responses stream cut before its first output delta writes no spend row on either leg

Gauntlet-mini: SURVIVED, 5/5 lenses, 1 confirmed and fixed, run 5a5e210477eb, reviewed dccf503

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Low

Every aiohttp-transport provider now gets a TCP reset, not a TLS close, on an early stream close
- Chosen over a Bedrock-only fix, see Decisions
Iterating after the opening event loop closed now raises earlier
- Net-negative to fix: a guard needs aiohttp's private _loop
A shared_session user's ssl_shutdown_timeout grace is skipped mid-body
- Net-negative to fix: that grace is what keeps Bedrock billing
The handwritten summary describes code the diff does not contain
- Kept as the author wrote it. The TLDR describes the shipped fix
Five sites still import aclose_if_supported from streaming_iterator, not stream_close
- Net-negative to move: it pulls Rust bridge files into the LTS backport

Decisions

The reset lives in the shared aiohttp transport, so it covers every provider
- Rejected: a Bedrock-only fix. Before, the other 8 routes in Case 4 kept reading for up to 30 s after an abort, so it would leave the same bug on them
- With DISABLE_AIOHTTP_TRANSPORT=True, the httpx transport this PR leaves alone already closes an aborted OpenAI stream within 5 s, so After matches it

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Bugbot summary

[!NOTE]
Medium Risk
Touches shared HTTP streaming teardown (aiohttp abort + chained aclose) across many providers; behavior change is abrupt TCP resets on early close rather than slow TLS shutdown, though aimed at fixing runaway upstream reads and billing.

Overview
Aborted streaming calls were leaving Bedrock (and other aiohttp-backed routes) still reading upstream, which inflated token billing and held connections open after client disconnect.

This PR centralizes stream teardown in litellm/llms/custom_httpx/stream_close.py (aclose_if_supported, aiter_bytes_then_close) and threads it through Bedrock invoke decoders, Bedrock /v1/messages streaming, and file-content streaming so each layer acloses its source when iteration stops early—including httpx.Response.aclose() once the byte iterator finishes or is abandoned.

On the aiohttp transport, AiohttpResponseStream now calls _abort_unreleased_connection (socket abort via transport.abort()) on finally and aclose(), so mid-body client drops tear down TLS streams promptly (notably under uvloop) instead of draining the server body.

Tests assert response closure on async Bedrock streams, detached-drain caps, and aiohttp abort behavior.

^{Reviewed by Cursor Bugbot for commit 3dab9ff. Bugbot is set up for automated code reviews on this repo. Configure here.}

3dab9ff passes /live-pr-risk

veria-ai · 2026-11-09T01:27:18Z

PR overview

No security issues were found in this pull request.

Reviews

3dab9ff — No new issues
aed8398 — No new issues
2eeb586 — No new issues

ryan-crabbe-berri · 2026-11-10T03:26:48Z

@greptileai

greptile-apps · 2026-11-10T03:29:57Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with both numbered findings addressed and no new actionable issues found.

Summary

This PR closes abandoned Bedrock streams through each wrapper and immediately aborts unreleased aiohttp connections. The latest change checks that a provider error also closes the response once.

Bedrock Invoke streams close their response when a client stops reading early.
The shared aiohttp transport drops open connections when a stream closes early.

_{Reviews (6) · Last reviewed commit: "test(bedrock): assert the converse strea..." · Reviewed by Greptile}

greptile-apps · 2026-11-10T03:30:01Z

+    finally:
+        await response.aclose()


Required live proof is missing

This changes stream cleanup and billing behavior. Repository guidance requires live before/after proxy proof. Before merging, add reproducible Bedrock disconnect and normal-stream results at afbfaeed500a3d193de8039a18fb05e8b27cae59.

Context Used: AGENTS.md (source)

Added the live proxy /v1/messages before and after reproduction on abee9c9 and bbdb38b for disconnect and full stream in the PR description.

Live proof is now in the description. Sonnet 4.5 on Bedrock billed about 1,500 tokens per abort before and 3 after

Case 4 now covers a full stream too. Three Sonnet 4.5 streams per leg end with end_turn and 126 tokens on both legs

Thanks—Case 4 closes the full-stream control gap: it demonstrates three completed real Bedrock streams on each leg with matching HTTP 200, end_turn, and 126 output tokens.

One QA detail remains: the before leg is identified as ghcr.io/berriai/litellm:main-latest / v1.111.0-dev.2, but the description doesn’t give the image’s source commit SHA. Without it, I can’t verify that the before run used the PR base or a later default-branch commit. Please add that SHA. The after run is identified as 2eeb586, the latest product commit.

The before image is main-latest built from ba1257b, 70 commits before the merge base, none touching aiohttp_transport.py or the close path

Thanks—that identifies the before image and helps show the relevant code was unchanged, but it still falls short of the required baseline: ba1257bda4 is 70 commits before the merge base dc378de207f8e79860d1fd52df26fcce092777e1. The before run must use the merge base or a later default-branch commit; unchanged close-path files don’t replace that requirement. Please rerun the before leg at dc378de207f8e79860d1fd52df26fcce092777e1 (or a later default-branch commit) and include its SHA and results. The after proof remains valid at/after the latest product commit, dccf503fd01a3a9442f654098a3a8bceb440faa7; the later PR commits 2eeb586add1aedd90f1da57daa0c1a063cda5bb9 and aed8398c2f7505f1c6ead73b831668b25502ab47 are test-only.

Both legs now run images built at the merge base dc378de and the tip 3dab9ff. CloudWatch reads about 1,500 tokens before and 300 after

…bort tests

princejain756 · 2026-11-10T06:17:39Z

@ryan-crabbe-berri @greptileai Updated the tests in bbdb38b to use Final state holders and fixed datetime(2026, 1, 1), and added live proxy before and after output.

ryan-crabbe-berri · 2026-11-10T06:47:45Z

@greptileai re review

ryan-crabbe-berri · 2026-11-10T06:47:50Z

bugbot run

ryan-crabbe-berri · 2026-11-10T06:48:08Z

@ryan-crabbe-berri @greptileai Updated the tests in bbdb38b to use Final state holders and fixed datetime(2026, 1, 1), and added live proxy before and after output.

thanks! i'm taking a look

cursor

Stale Bugbot comment from a previous run.

codspeed · 2026-11-10T07:00:06Z

Merging this PR will not alter performance

1 warning

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 44 untouched benchmarks

_{Comparing princejain756:fix/bedrock-invoke-close-httpx-response-on-abort (3dab9ff) with main (4e29aea)}

codecov · 2026-11-10T07:06:51Z

Codecov Report

❌ Patch coverage is 97.40260% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...transformations/anthropic_claude_transformation.py	95.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

…ck_abort_close # Conflicts: # litellm/llms/bedrock/chat/invoke_handler.py

Move SupportsAclose, aclose_if_supported, and aiter_bytes_then_close into litellm/llms/custom_httpx/stream_close.py so the Bedrock invoke paths, the generic HTTP handler, and the Anthropic messages iterator use one copy, and drop the mock-tolerant close in production code

…e its body ends Closing an unfinished aiohttp response on uvloop starts a graceful TLS shutdown that keeps reading the provider's body for up to 30 s. Aborting the transport first ends the upstream read at once.

mateo-berri · 2026-11-14T08:31:09Z

@greptileai

mateo-berri · 2026-11-14T08:57:52Z

@greptileai

…uvloop.run

mateo-berri · 2026-11-14T09:13:00Z

@greptileai

… exception event ends it

mateo-berri · 2026-11-14T16:30:50Z

@greptileai

mateo-berri · 2026-11-14T16:34:10Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3dab9ff. Configure here.}

mateo-berri

LGTM. Thanks for the contribution!

fix(bedrock): close httpx response stream when invoke stream is aborted

afbfaee

greptile-apps Bot reviewed Nov 10, 2026

View reviewed changes

test(bedrock): use Final state holders and fixed datetime in invoke a…

bbdb38b

…bort tests

cursor Bot reviewed Nov 10, 2026

View reviewed changes

princejain756 and others added 5 commits November 11, 2026 20:01

fix(bedrock): call response.aiter_bytes eagerly and await aclose safely

3ac9f9d

Merge remote-tracking branch 'origin/main' into litellm_pr50657_bedro…

b8d6f51

…ck_abort_close # Conflicts: # litellm/llms/bedrock/chat/invoke_handler.py

test(bedrock): cover each invoke stream layer closing its source

2eeb586

mateo-berri changed the title ~~fix(bedrock): close httpx response stream when invoke stream is aborted~~ fix(bedrock): close the Bedrock connection at once when an invoke stream is aborted Nov 14, 2026

greptile-apps Bot reviewed Nov 14, 2026

View reviewed changes

Comment thread tests/unit/llms/custom_httpx/test_aiohttp_transport.py Resolved

greptile-apps Bot reviewed Nov 14, 2026

View reviewed changes

Comment thread tests/unit/llms/custom_httpx/test_aiohttp_transport.py Resolved

Comment thread tests/unit/llms/custom_httpx/test_aiohttp_transport.py Resolved

test(aiohttp): pin the TLS test certificate dates and run it through …

aed8398

…uvloop.run

test(bedrock): assert the converse stream closes its response when an…

3dab9ff

… exception event ends it

cursor Bot reviewed Nov 14, 2026

View reviewed changes

mateo-berri changed the title ~~fix(bedrock): close the Bedrock connection at once when an invoke stream is aborted~~ fix(aiohttp): reset the upstream connection when a stream closes early, so aborted Bedrock Invoke streams stop Nov 14, 2026

mateo-berri approved these changes Nov 14, 2026

View reviewed changes

mateo-berri merged commit ec48e24 into BerriAI:main Nov 14, 2026
97 checks passed

Conversation

princejain756 commented Nov 9, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Handwritten 1-3 sentence description

TLDR

User Flow

Relevant issues

Pre-Submission checklist

Screenshots / Proof of Fix

Before (dc378de)

Case 1: Sonnet 4.5 CloudWatch output tokens per abort

Case 2: Sonnet 5.5 upstream socket after the client disconnects

Case 3: Sonnet 4.5 full stream

Case 4: the other streaming routes on the same transport

After (3dab9ff)

Case 1: Sonnet 4.5 CloudWatch output tokens per abort

Case 2: Sonnet 5.5 upstream socket after the client disconnects

Case 3: Sonnet 4.5 full stream

Case 4: the other streaming routes on the same transport

Type

Caveats (if any)

Low

Final Attestation

Uh oh!

veria-ai Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Reviews

Uh oh!

ryan-crabbe-berri commented Nov 10, 2026

Uh oh!

greptile-apps Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

greptile-apps Bot Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

princejain756 Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Nov 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

princejain756 commented Nov 10, 2026

Uh oh!

ryan-crabbe-berri commented Nov 10, 2026

Uh oh!

ryan-crabbe-berri commented Nov 10, 2026

Uh oh!

ryan-crabbe-berri commented Nov 10, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

princejain756 commented Nov 9, 2026 •

edited by mateo-berri

Loading

Before (`dc378de`)

After (`3dab9ff`)

veria-ai Bot commented Nov 9, 2026 •

edited

Loading

greptile-apps Bot commented Nov 10, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codspeed Bot commented Nov 10, 2026 •

edited

Loading

codecov Bot commented Nov 10, 2026 •

edited

Loading