Skip to content

fix: Gemini streaming and function calling#32

Merged
yuanhao merged 2 commits into
yologdev:mainfrom
cablehead:fix/gemini-streaming
Apr 8, 2026
Merged

fix: Gemini streaming and function calling#32
yuanhao merged 2 commits into
yologdev:mainfrom
cablehead:fix/gemini-streaming

Conversation

@cablehead

@cablehead cablehead commented Apr 3, 2026

Copy link
Copy Markdown
Contributor

Fixes several issues with the Google Gemini provider that break streaming and function calling.

SSE parsing

  • Handle \r\n line endings (Google's API uses CRLF, parser only handled LF)
  • Skip empty text parts sent during the thinking phase
  • Upgrade parse error logging from debug! to warn!

Function calling

  • Don't let finishReason: "STOP" override StopReason::ToolUse -- Gemini returns "STOP" even when it emits function calls, so the agent loop never executed tools
  • Capture and round-trip thought signatures via provider_metadata field on Content::ToolCall (keeps provider-specific data out of tool arguments)
  • Use real function call id from response, with synthetic google-fc-{n} fallback for models that omit it
  • Include id in functionResponse when sending tool results back

Content::ToolCall change

  • Add optional provider_metadata: Option<serde_json::Value> field for provider-specific data that shouldn't reach tool execution
  • All pattern match sites use .. to ignore the field; only construction sites set it explicitly

Tests

  • 7 unit tests: SSE parsing, thought signature round-tripping, function call ID handling
  • 1 MockProvider test: verifies agent loop executes tools when provider_metadata is Some(...)
  • 2 integration tests: simple text, tool use with bash (gemini-3-flash-preview)

Generated with LLM assistance (Claude Code). Changes have been manually exercised against live APIs via yoke, a headless agent harness built on yoagent.

@yuanhao

yuanhao commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR — fixes real Gemini compatibility issues and the test coverage is solid. Two design concerns worth discussing before merging:

1. __thought_signature leaks into tool execution

The signature stashing in arguments round-trips correctly back to Gemini, but between parse and re-serialization the tool call flows through the agent loop like any other, so tools actually receive __thought_signature in their arguments when execute() is called. Most built-in tools ignore unknown keys, but OpenAPI-adapted tools (see src/openapi/) and MCP tools that do strict JSON schema validation may reject it.

A cleaner fix would be to add an optional field to Content::ToolCall:

ToolCall {
    id: String,
    name: String,
    arguments: serde_json::Value,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    provider_metadata: Option<serde_json::Value>,
},

This follows the precedent of Content::Thinking { signature } and keeps provider-specific data out of the tool-facing payload. Happy to do this as a follow-up if you'd prefer to land this PR as-is.

2. "google-fc-" prefix as a sentinel

The id filter uses !tool_call_id.starts_with("google-fc-") to distinguish synthetic from real IDs. If Google ever returns an ID that happens to start with "google-fc-", the ID would be silently dropped and multi-turn function calling would break — the exact thing this PR is fixing. An Option<String> on Content::ToolCall (or storing synthetic-ness out-of-band) would be more robust, but this is a minor nit given the low likelihood.


Neither is a blocker. Nice work on the unit + integration test coverage.

One small thing for the integration tests: gemini-3-flash-preview isn't a real model name — likely gemini-2.5-flash or similar. Since they're #[ignore]d it won't break CI, but worth updating so someone can actually run them.

🤖 Generated with Claude Code

@yuanhao

yuanhao commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator

@cablehead thanks for trusting yoagent!

@cablehead
cablehead force-pushed the fix/gemini-streaming branch 2 times, most recently from 65a463d to f3d880d Compare April 7, 2026 16:34
@cablehead

Copy link
Copy Markdown
Contributor Author

Thanks for the thoughtful review -- and for building yoagent, it's a great foundation to work with.

Pushed updates addressing both points:

1. provider_metadata instead of __thought_signature in arguments

Added an optional provider_metadata: Option<serde_json::Value> field to Content::ToolCall. The Gemini provider stores {"thought_signature": "..."} there, and reads it back when building the next request. Tools never see it.

2. Dropped the "google-fc-" prefix

Gemini returns real function call IDs, so the synthetic fallback was unnecessary. Now it's just fc.id.clone().unwrap_or_default() and the check is !id.is_empty(). No sentinel strings.

Re: gemini-3-flash-preview -- it does resolve and work (tested against the live API), but happy to change it to gemini-2.5-flash if you'd prefer a GA model name in the tests.

Repository owner deleted a comment from yuanhao Apr 8, 2026
@yuanhao

yuanhao commented Apr 8, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the iteration — provider_metadata and dropping the google-fc- sentinel are exactly the right shape. But the refactor (looks like cargo fix autoexpansion) introduced a load-bearing pattern bug that reintroduces the very behavior this PR is fixing.

Blocking: literal provider_metadata: None used as a match guard

The new field is destructured with a literal None in five places. That makes it a refinement filter, not a default — any ToolCall carrying Some(metadata) falls through and is silently dropped.

src/agent_loop.rs:382 — the headline regression:

.filter_map(|c| match c {
    Content::ToolCall {
        provider_metadata: None,   // ← excludes Some(...)
        id, name, arguments,
    } => Some(...)

Gemini sets Some({"thought_signature": ...}) on every function call when thinking is on. The agent loop extracts zero tool calls, dispatches nothing, and the loop terminates. End-to-end, the PR's fix doesn't run tools on the exact path it's targeting.

Same pattern in four more sites — latent today (Azure/Responses never set metadata, Vertex sets None), but each one is a landmine that silently corrupts the next provider that populates the field:

  • src/provider/azure_openai.rs:164StopReason::ToolUse detection
  • src/provider/openai_responses.rs:184 — same
  • src/provider/azure_openai.rs:248, openai_responses.rs:271, openai_compat.rs:275, google_vertex.rs:302 — request-body builders silently strip ToolCalls with metadata from outgoing requests

Fix: replace provider_metadata: None, with .. in all non-construction sites. Construction sites (where you build a fresh Content::ToolCall) are correct as written.

Important

Empty function call IDs collidesrc/provider/google.rs:139 uses fc.id.clone().unwrap_or_default(). Gemini commonly omits id outside parallel function calling, so every call gets id = "". The agent loop correlates tool results to calls by id; two empty IDs in the same turn collide silently. Suggest preserving the synthetic fallback for the None branch:

let id = fc.id.clone().unwrap_or_else(|| format!("google-fc-{}", content.len()));

Breaking API change — adding provider_metadata to Content::ToolCall (a public enum variant) breaks any downstream crate that constructs the variant by name. Marking Content (or just the variant) #[non_exhaustive] would prevent this from biting again.

SSE parse errors swallowed at debug!src/provider/google.rs:107. A schema change from Google would silently truncate turns with no signal in default logging. Should be warn! at minimum.

Test coverage gaps

  • No agent-loop test for ToolCall with Some(provider_metadata) — a MockProvider-based test in tests/agent_loop_test.rs would have caught the blocking bug above. MockProvider (src/provider/mock.rs:99) hardcodes provider_metadata: None and needs an injection point.
  • test_parse_chunk_with_crlf_sse is vacuous — it re-implements the separator detection inside the test body and asserts on its own copy. Breaking the real loop wouldn't break the test.
  • test_parse_chunk_with_empty_text only verifies that serde parses {"text": ""}, not the continue guard at google.rs:118.
  • Integration tests in tests/integration_gemini.rs are #[ignore] + require GEMINI_API_KEY → won't run in CI → zero regression protection. The thought-signature round-trip can be expressed against MockProvider.

Minor

  • tests/integration_gemini.rs still uses gemini-3-flash-preview — confirmed working live, but a GA name would be safer.
  • Option<serde_json::Value> for provider_metadata is fully untyped; future-proofing this as enum ProviderMetadata { Gemini { thought_signature: String }, ... } would give compile-time exhaustiveness and prevent key collisions across providers. Not blocking — just worth considering.

Suggested action

A single sweep replacing literal provider_metadata: None, with .. in the five non-construction sites unblocks the merge. The other items can land in follow-ups.

- Fix SSE parsing to handle \r\n line endings from Google's API
- Skip empty text parts sent during thinking phase
- Fix stop_reason: don't let "STOP" override ToolUse (Gemini returns
  "STOP" even when emitting function calls)
- Capture and round-trip thought signatures via provider_metadata field
  on Content::ToolCall (keeps provider-specific data out of tool args)
- Use real function call IDs from Gemini response
- Include function call ID in functionResponse
- Add unit tests for parsing, thought signature round-tripping,
  CRLF SSE handling, and function response ID behavior
- Add Gemini integration tests for simple text and tool use
@cablehead
cablehead force-pushed the fix/gemini-streaming branch from f3d880d to c7eeb37 Compare April 8, 2026 15:37
@cablehead

Copy link
Copy Markdown
Contributor Author

Good catch on the provider_metadata: None pattern bug -- that was a mechanical sed that went wrong. Fixed in all 7 non-construction sites (replaced with ..).

Also addressed:

  • Restored synthetic google-fc-{n} fallback for models that omit id
  • Upgraded SSE parse error logging from debug! to warn!

The test coverage gaps are fair points. Happy to add a MockProvider-based test for ToolCall with Some(provider_metadata) if you'd like that in this PR, or as a follow-up.

- Replace literal provider_metadata: None with .. in all pattern match
  sites (agent_loop, azure_openai, openai_compat, openai_responses,
  google_vertex) -- None acted as a match guard, silently dropping
  ToolCalls with Some(metadata)
- Add provider_metadata field to MockToolCall for test injection
- Add test_tool_call_with_provider_metadata_executes: verifies agent
  loop executes tools even when provider_metadata is Some
- Restore synthetic google-fc-{n} ID fallback for models that omit id
- Upgrade SSE parse error log from debug to warn
@cablehead

Copy link
Copy Markdown
Contributor Author

Follow-up commit adds the MockProvider test: test_tool_call_with_provider_metadata_executes verifies the agent loop executes tools when provider_metadata is Some(...). Added provider_metadata field to MockToolCall for injection.

@yuanhao

yuanhao commented Apr 8, 2026

Copy link
Copy Markdown
Collaborator

Re-reviewed after 368c0ee. All blocking issues are resolved:

  • agent_loop.rs:382 — literal provider_metadata: None replaced with ..; Gemini tool calls with thought signatures now execute end-to-end
  • Same anti-pattern cleaned up in azure_openai.rs, openai_responses.rs, openai_compat.rs stop-reason checks and request builders
  • Empty function call ID collision fixed — google.rs:139 restores the synthetic google-fc-{n} fallback
  • SSE parse errors bumped from debug! to warn!
  • test_tool_call_with_provider_metadata_executes is a proper regression guard — would have caught the original bug, and passes now
  • MockToolCall now exposes provider_metadata so future tests can inject metadata cleanly

Verified locally: cargo check clean, 89 unit tests + the new agent-loop test all green on the branch.

LGTM — approving. Thanks for the careful fixup.

A few non-blocking follow-ups I'll track in a separate issue (breaking-change #[non_exhaustive], vacuous CRLF/empty-text tests, MockProvider-based coverage for the Gemini integration tests so they run in CI, typed ProviderMetadata enum). None of these block merge.

Note for the release: adding a required field to the public Content::ToolCall variant is a breaking change, so the next release should be 0.8.0, not 0.7.6. Confirmed this breaks at least one downstream construction site in our own yoyo-evolve — we'll fix that when we bump.

@yuanhao
yuanhao merged commit 4cb5230 into yologdev:main Apr 8, 2026
yuanhao added a commit that referenced this pull request Jul 4, 2026
…google.rs error surfacing

Type policy (completes the non_exhaustive work coherently):
- ModelConfig::custom(api, provider, base_url, id, name): the construction
  path for Bedrock/Vertex/Azure and future protocols — non_exhaustive had
  removed downstream struct literals without a replacement for those three
- variant-level non_exhaustive on Message::Assistant and Content::Thinking
  (same PR-#32 exposure class as ToolCall); Message::assistant() +
  with_error_message()/with_timestamp() and Content::thinking()/
  thinking_signed() constructors; external tests migrated
- doc fixes: enum-level vs variant-level levers explained separately,
  "0.9.0" phrased as slated rather than shipped, FRU/serde(default) notes,
  stale enum listings in messages-events.md and CLAUDE.md updated

google.rs error surfacing (pre-existing gaps in the rewired loop):
- mid-stream {"error": ...} payloads now classify and return Err instead
  of deserializing into an empty chunk and vanishing
- transport errors return Err(Network) like every other provider instead
  of warn + break -> fake successful turn
- SAFETY/PROHIBITED_CONTENT/BLOCKLIST/SPII finish reasons map to
  StopReason::Refusal with an explanatory error_message

Tests (closing the coverage-review gaps):
- empty-text part with functionCall no longer swallowed (loop-level)
- next_sse_data consumes keepalives/data-less events (Some(""), not None)
- multi-event text-delta accumulation; usage mapping; in-stream error;
  SAFETY -> Refusal; structural thoughtSignature placement assertion;
  constructor field-pinning; metadata ToolCall persistence round-trip

Co-Authored-By: Claude Fable 5 <[email protected]>
yuanhao added a commit that referenced this pull request Jul 5, 2026
…ening (#55)

Exhaustiveness policy: #[non_exhaustive] on Content (+ ToolCall/Thinking variants), Message::Assistant variant, and ModelConfig, with constructors (Content::tool_call*/thinking*, Message::assistant, ModelConfig::custom); StopReason deliberately exhaustive. google.rs: real tests for extracted SSE helpers (earliest-separator fix, empty-text no longer swallows functionCall), in-stream errors and transport failures now surface as Err, SAFETY finish reasons map to StopReason::Refusal. PR #32 scenarios covered in CI via wiremock.

Refs #33 (items 1-3; 4-5 deferred)

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants