Repository navigation
fix(agent): surface model refusals as content_filter (salvage #43108 + edge-case fix) - #46013
Merged
kshitijk4poor merged 4 commits intoJun 14, 2026
Conversation
A Claude refusal (HTTP 200, stop_reason="refusal", empty content) was laundered into a generic retry loop and surfaced as a misleading "rate limited / invalid response" or "no content after retries" error, burning paid attempts reproducing a deterministic refusal. This hit two distinct paths: - Direct Anthropic (anthropic_messages): validate_response rejected the empty-content refusal *before* normalize_response mapped refusal -> content_filter, so it fell into the invalid-response retry loop. - Nous Portal / OpenAI-compatible (chat_completions): the portal surfaces a Claude refusal via message.refusal with empty content, which sailed past validation and died in the empty-response retry loop. Fix (one unified content_filter dispatch for all backends): - AnthropicTransport.validate_response: accept empty content when stop_reason == "refusal" so it flows to normalize_response. - ChatCompletionsTransport.normalize_response: promote message.refusal to content + a content_filter finish reason. - conversation_loop: handle finish_reason == "content_filter" - fire the api_request_error hook (content_policy_blocked), try a configured fallback once, else return a clear terminal refusal message. Never retry a deterministic refusal. Supersedes NousResearch#43084, which fixed only the direct-Anthropic path and could not reach the chat_completions/portal path. Tests: transport-level (validate_response refusal, message.refusal promotion) + end-to-end loop (refusal surfaced, exactly one API call). (cherry picked from commit 01f546f)
OpenRouter (and every other OpenAI-compatible provider) uses the default chat_completions transport, so it is already covered by the refusal fix: an upstream Claude / moderation refusal arrives as finish_reason="content_filter" (often empty content, no message.refusal). Add a regression test asserting the transport passes that finish reason straight through to the loop's content_filter handler. (cherry picked from commit 60168a5)
… payload A chat-completions response that carries real text or tool calls *alongside* a `message.refusal` note is a normal, usable turn — the model did work. The prior logic flipped finish_reason to `content_filter` whenever a refusal string was present, so the conversation loop reframed a content-bearing turn as a *failed* safety refusal (failed=True) and buried the model's actual output inside the "model declined" template, or dropped tool calls entirely. Only promote to a terminal `content_filter` when the refusal is the sole payload (no visible text AND no tool calls). The refusal explanation is still recorded in provider_data in every case for observability. Refusal-only responses (the bug this feature targets) are unaffected and still surface terminally; the empty+refusal, bare content_filter passthrough, and no-refusal common cases are byte-identical to before. Updates the partial-content test to the corrected contract and adds a tool_calls-alongside-refusal regression guard.
…covery hint
The HTTP-200 refusal handler (finish_reason=content_filter) and the
exception-path handler (a provider moderation error classified as
content_policy_blocked) independently built the same terminal turn result —
the same {final_response, messages, api_calls, completed:False, failed:True,
error:'content_policy_blocked: ...'} dict — and ended their user-facing
message with the same 'Try rephrasing... hermes fallback add' trailer, copied
verbatim. The two copies could drift.
Funnel both through a shared _content_policy_blocked_result() builder and a
shared _CONTENT_POLICY_RECOVERY_HINT constant. Also collapse the HTTP-200
path's two near-identical with/without-explanation templates into one (compute
the detail fragment once) and pass reason=FailoverReason.content_policy_blocked
.value to the error hook instead of a hand-written string literal, matching the
sibling hook call.
Behavior-preserving: the provider/refusal lead-in wording stays distinct (a
provider safety filter vs the model declining are genuinely different signals),
the with-text and exception messages are byte-identical to before, and the
no-explanation case only gains a paragraph break for consistency. Surfaced by
the simplify-code reuse/quality reviewers.
The efficiency reviewer's 'redundant normalize_response' flag was deliberately
NOT applied: that branch is cold (refusal-only) and pure-CPU, and reusing the
sibling-branch normalized locals would risk a NameError on the codex_responses
path (which sets finish_reason without normalizing) — re-normalizing is the
robust choice.
kshitijk4poor
enabled auto-merge
June 14, 2026 06:54
AIalliAI
pushed a commit
to AIalliAI/Hermes
that referenced
this pull request
Jun 14, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
T02200059
pushed a commit
to T02200059/hermes-agent
that referenced
this pull request
Jun 18, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
1 task
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
Methodician
added a commit
to Methodician/hermes-agent
that referenced
this pull request
Jul 4, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
habarmc1223-sudo
pushed a commit
to habarmc1223-sudo/hermes-agent-fluxmem
that referenced
this pull request
Jul 8, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
teknium1
added a commit
that referenced
this pull request
Sep 15, 2026
…ncode#43343) A model that declines mid-stream delivers the explanation on the structured refusal channel (chat_completions delta.refusal; Responses response.refusal.delta / refusal content parts) and leaves content empty. The streaming accumulators dropped that channel entirely, so a streamed refusal assembled into an empty message and fell into the empty/invalid-response retry loops - burning paid retries reproducing a deterministic refusal - while the non-streaming path had already fixed this class in #46013. - chat_completions streaming: accumulate delta.refusal (incl. model_extra), expose message.refusal on the assembled mock response so ChatCompletionsTransport.normalize_response applies the existing sole-payload -> content_filter promotion; count refusal deltas in the zero-chunk guard; carry refusal in the Relay final-response dict. - Codex Responses stream consumer: collect response.refusal.delta as answer text so a refusal-only stream no longer raises 'did not emit a terminal response' with zero usable content. - Responses normalizer: read type=refusal content parts in _extract_responses_message_text (attr and dict shapes). Sabotage-verified: each new test fails with its wiring line disabled. E2E: refusal-only stream -> terminal content_filter with explanation; refusal-alongside-content stays a normal usable turn; plain-text streams unchanged.
teknium1
added a commit
that referenced
this pull request
Sep 15, 2026
…ncode#43343) A model that declines mid-stream delivers the explanation on the structured refusal channel (chat_completions delta.refusal; Responses response.refusal.delta / refusal content parts) and leaves content empty. The streaming accumulators dropped that channel entirely, so a streamed refusal assembled into an empty message and fell into the empty/invalid-response retry loops - burning paid retries reproducing a deterministic refusal - while the non-streaming path had already fixed this class in #46013. - chat_completions streaming: accumulate delta.refusal (incl. model_extra), expose message.refusal on the assembled mock response so ChatCompletionsTransport.normalize_response applies the existing sole-payload -> content_filter promotion; count refusal deltas in the zero-chunk guard; carry refusal in the Relay final-response dict. - Codex Responses stream consumer: collect response.refusal.delta as answer text so a refusal-only stream no longer raises 'did not emit a terminal response' with zero usable content. - Responses normalizer: read type=refusal content parts in _extract_responses_message_text (attr and dict shapes). Sabotage-verified: each new test fails with its wiring line disabled. E2E: refusal-only stream -> terminal content_filter with explanation; refusal-alongside-content stays a normal usable turn; plain-text streams unchanged.
fsca8
pushed a commit
to fsca8/hermes-agent
that referenced
this pull request
Oct 6, 2026
…sal-content-filter fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Salvages @SHL0MS's refusal-handling fix from #43108 onto current
main, with two follow-up commits that fix an edge case and apply a reuse cleanup. Authorship on the original work is preserved in git history.A Claude model can return HTTP 200 with a refusal — a deterministic, server-side safety halt. Hermes mislabeled it as a transient infra error and retried it 3× (burning paid attempts) with no signal that the prompt was actually refused. The fix routes every backend through one normalized
content_filterdispatch:AnthropicTransport.validate_response— accept emptycontentwhenstop_reason == "refusal"(a complete response, not a truncation), so it flows throughnormalize_response→content_filter.ChatCompletionsTransport.normalize_response— promote OpenAI'smessage.refusal(and themodel_extraproxy variant) to content + acontent_filterfinish reason; barecontent_filter(OpenRouter) passes straight through.conversation_loop.py— handlefinish_reason == "content_filter"once: fire theapi_request_errorhook withreason=content_policy_blocked, try one configured fallback, otherwise return an honest terminal result. No retry burn — the block is deterministic.One fix covers Anthropic-direct, Nous Portal, OpenRouter, other OpenAI-compatible providers, and Bedrock (whose
guardrail_intervenedalready maps tocontent_filter), because the handler keys on the normalized finish reason rather than a path-specific flag.Commits
fix(agent): surface model refusals…— @SHL0MS's original fix (authorship preserved).test(transports): lock in content_filter passthrough for OpenRouter— @SHL0MS's transport tests (authorship preserved).fix(transports): only treat a refusal as terminal when it is the sole payload— edge-case fix. A chat-completions response that carries real text or tool calls alongside amessage.refusalnote is a normal, usable turn. The prior logic flippedfinish_reason → content_filterwhenever a refusal string was present, so a content-bearing turn was reframed as a failed safety refusal (failed=True) and the model's actual output was buried in the "model declined" template, or its tool calls dropped. Now promotion to terminalcontent_filterhappens only when the refusal is the sole payload (no visible text AND no tool calls); the explanation is still recorded inprovider_datain every case. Refusal-only responses (the bug this PR targets) are unaffected and still surface terminally. Adds a partial-content regression guard and a tool-calls-alongside-refusal guard (both proven to fail on the prior behavior).refactor(agent): share the content_policy_blocked result builder + recovery hint— the HTTP-200 refusal handler and the existing exception-path handler independently built the same terminal result dict and ended their message with the samehermes fallback addtrailer copied verbatim. Funnel both through a shared_content_policy_blocked_result()builder +_CONTENT_POLICY_RECOVERY_HINTconstant, collapse the HTTP-200 path's two near-identical templates into one, and passreason=FailoverReason.content_policy_blocked.valueinstead of a string literal (matching the sibling hook call). Behavior-preserving: the provider-filter vs model-declined lead-in wording stays distinct, and the with-text and exception messages are byte-identical to before.Tests
tests/agent/transports/test_transport.py— empty-contentrefusalvalidates.tests/agent/transports/test_chat_completions.py—message.refusalpromotion; barecontent_filterpassthrough (OpenRouter); no-op when absent; partial-content not terminated and tool-calls-alongside-refusal not terminated (commit 3 guards).tests/run_agent/test_run_agent.py::TestRetryExhaustion::test_content_filter_refusal_surfaced_not_retried— end-to-end: a portal-stylemessage.refusalis surfaced ascontent_policy_blockedwith the refusal text, model called exactly once (no retry loop).tests/run_agent/test_18028_content_policy_blocked.py(exception path) stays green through the shared-helper refactor.Local:
ruffclean; full transports + run_agent + content-policy + fallback suites pass (740 passed).Closes #43108