Skip to content

chore: implement streaming for anthropic.rs firstparty provider#3419

Merged
michaelneale merged 4 commits intomainfrom
micn/anthropic-streaming
Jul 16, 2025
Merged

chore: implement streaming for anthropic.rs firstparty provider#3419
michaelneale merged 4 commits intomainfrom
micn/anthropic-streaming

Conversation

@michaelneale
Copy link
Copy Markdown
Collaborator

@michaelneale michaelneale commented Jul 15, 2025

Similar to openai: #3413

this adds in streaming for anthropic provider

@michaelneale michaelneale added p1 Priority 1 - High (supports roadmap) performance Performance related labels Jul 15, 2025
@michaelneale michaelneale mentioned this pull request Jul 15, 2025
Merged
@michaelneale michaelneale requested a review from jamadeo July 15, 2025 04:19
@jamadeo
Copy link
Copy Markdown
Collaborator

jamadeo commented Jul 15, 2025

Nice thanks @michaelneale ! when I run it the streaming looks good but my token count stays at 0 -- maybe not pulling the usage out correctly?

@jamadeo
Copy link
Copy Markdown
Collaborator

jamadeo commented Jul 15, 2025

Turns out I had a similar issue with OpenAI: #3441

Is there a similar flag for Anthropic?

@michaelneale
Copy link
Copy Markdown
Collaborator Author

michaelneale commented Jul 16, 2025

oh interesting, yes will take a look

@michaelneale michaelneale changed the title feat: implement streaming for anthropic.rs firstparty provider chore: implement streaming for anthropic.rs firstparty provider Jul 16, 2025
@michaelneale
Copy link
Copy Markdown
Collaborator Author

michaelneale commented Jul 16, 2025

adding in some logging... trying to see where the token counts are coming in:

🔍 Anthropic streaming line: data: {"type":"message_start","message":{"id":"msg_01VSXceb8PEy1o1DgnrvAuSv","type":"message","role":"assistant","model":"claude-sonnet-4-20250514","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":3972,"cache_read_input_tokens":0,"output_tokens":1,"service_tier":"standard"}}            }
🔍 Anthropic streaming event: type=message_start, data_keys=["message"]
🔍 Anthropic streaming line:
◐  Yielding yaml yearnings...                                                                                                                                                                     🔍 Anthropic streaming line: event: content_block_start
🔍 Anthropic streaming line: data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}          }
🔍 Anthropic streaming event: type=content_block_start, data_keys=["content_block", "index"]
🔍 Anthropic streaming line:
🔍 Anthropic streaming line: event: ping
🔍 Anthropic streaming line: data: {"type": "ping"}
🔍 Anthropic streaming event: type=ping, data_keys=[]
🔍 Anthropic streaming line:
🔍 Anthropic streaming line: event: content_block_delta
🔍 Anthropic streaming line: data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Here"}          }
🔍 Anthropic streaming event: type=content_block_delta, data_keys=["delta", "index"]
Here🔍 Anthropic streaming line:
🔍 Anthropic streaming line: event: content_block_delta
🔍 Anthropic streaming line: data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"'s a short joke for you:"}  }
🔍 Anthropic streaming event: type=content_block_delta, data_keys=["delta", "index"]
's a short joke for you:🔍 Anthropic streaming line:
🔍 Anthropic streaming line: event: content_block_delta
🔍 Anthropic streaming line: data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"\n\nWhy don't scientists trust atoms?"}   }
🔍 Anthropic streaming event: type=content_block_delta, data_keys=["delta", "index"]

according to docs:

https://docs.anthropic.com/en/docs/build-with-claude/streaming

The token counts shown in the usage field of the message_delta event are cumulative.

but am not seeing it - hrm... will have to put this on the back burner for a bit as not clear for a while. need to read some more.

if anyone knows and picks this up - go for it but will revisit it.

@michaelneale michaelneale self-assigned this Jul 16, 2025
@michaelneale
Copy link
Copy Markdown
Collaborator Author

michaelneale commented Jul 16, 2025

ok that took more time than I had hoped - it seems you have to accumulate/calculate as you go.

jamadeo
jamadeo approved these changes Jul 16, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@jamadeo jamadeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this worked for me.

@michaelneale michaelneale merged commit eaba38a into main Jul 16, 2025
8 checks passed
@michaelneale michaelneale deleted the micn/anthropic-streaming branch July 16, 2025 21:32
lifeizhou-ap added a commit that referenced this pull request Jul 17, 2025
@lifeizhou-ap
Merge branch 'main' into wtang/execute_dynamic_tasks
7c492e3 
* main:
  feat(gcpvertexai): do HTTP 429 like retries for Anthropic API HTTP 529 overloaded status code (#3026)
  Fix a few ui edge cases - refresh occasionally crashing, chat loader over text and chat input height returning to auto (#3469)
  Don't default to main for build-cli (#3467)
  docs: add MongoDB MCP server tutorial (#2660)
  feat: run sub recipe multiple times in parallel (Experimental feature) (#3274)
  chore(release): release version 1.1.0 (#3465)
  chore: implement streaming for anthropic.rs firstparty provider (#3419)
  Fix regression: add back detail to tool-call banners (#3231)
  Document release process and update some just recipes (#3460)
  feat: add download_cli.ps1 file for windows (#3354)
  fix: session_file is optional (#3462)
  Bump more space for goose is working on it so it doesnt overlap incoming agent chat messages (#3453)
  Align chat input action buttons to bottom when large amount of text (#3455)
  docs: add Cloudflare MCP Server tutorial (#3278)
  feat(cli): Clear persisted session file with /clear command (#3145)
s-soroosh pushed a commit to s-soroosh/goose that referenced this pull request Jul 18, 2025
@michaelneale @s-soroosh
chore: implement streaming for anthropic.rs firstparty provider (bloc…
3f52866 
…k#3419)

Signed-off-by: Soroosh <[email protected]>
cbruyndoncx pushed a commit to cbruyndoncx/goose that referenced this pull request Jul 20, 2025
@michaelneale
chore: implement streaming for anthropic.rs firstparty provider (bloc…
144a7f3 
…k#3419)
atarantino pushed a commit to atarantino/goose that referenced this pull request Aug 5, 2025
@michaelneale
chore: implement streaming for anthropic.rs firstparty provider (bloc…
d80c394 
…k#3419)

Signed-off-by: Adam Tarantino <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jamadeo jamadeo jamadeo approved these changes

Assignees

@michaelneale michaelneale

Labels

p1 Priority 1 - High (supports roadmap) performance Performance related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@michaelneale @jamadeo