Skip to content

fix(agent): surface model refusals as content_filter (salvage #43108 + edge-case fix) - #46013

Merged
kshitijk4poor merged 4 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/refusal-content-filter
Jun 14, 2026
Merged

kshitijk4poor merged 4 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/refusal-content-filter

Conversation

@kshitijk4poor

Copy link
Copy Markdown

Summary

Salvages @SHL0MS's refusal-handling fix from #43108 onto current main, with two follow-up commits that fix an edge case and apply a reuse cleanup. Authorship on the original work is preserved in git history.

A Claude model can return HTTP 200 with a refusal — a deterministic, server-side safety halt. Hermes mislabeled it as a transient infra error and retried it 3× (burning paid attempts) with no signal that the prompt was actually refused. The fix routes every backend through one normalized content_filter dispatch:

  • AnthropicTransport.validate_response — accept empty content when stop_reason == "refusal" (a complete response, not a truncation), so it flows through normalize_response → content_filter.
  • ChatCompletionsTransport.normalize_response — promote OpenAI's message.refusal (and the model_extra proxy variant) to content + a content_filter finish reason; bare content_filter (OpenRouter) passes straight through.
  • conversation_loop.py — handle finish_reason == "content_filter" once: fire the api_request_error hook with reason=content_policy_blocked, try one configured fallback, otherwise return an honest terminal result. No retry burn — the block is deterministic.

One fix covers Anthropic-direct, Nous Portal, OpenRouter, other OpenAI-compatible providers, and Bedrock (whose guardrail_intervened already maps to content_filter), because the handler keys on the normalized finish reason rather than a path-specific flag.

Commits

  1. fix(agent): surface model refusals… — @SHL0MS's original fix (authorship preserved).
  2. test(transports): lock in content_filter passthrough for OpenRouter — @SHL0MS's transport tests (authorship preserved).
  3. fix(transports): only treat a refusal as terminal when it is the sole payload — edge-case fix. A chat-completions response that carries real text or tool calls alongside a message.refusal note is a normal, usable turn. The prior logic flipped finish_reason → content_filter whenever a refusal string was present, so a content-bearing turn was reframed as a failed safety refusal (failed=True) and the model's actual output was buried in the "model declined" template, or its tool calls dropped. Now promotion to terminal content_filter happens only when the refusal is the sole payload (no visible text AND no tool calls); the explanation is still recorded in provider_data in every case. Refusal-only responses (the bug this PR targets) are unaffected and still surface terminally. Adds a partial-content regression guard and a tool-calls-alongside-refusal guard (both proven to fail on the prior behavior).
  4. refactor(agent): share the content_policy_blocked result builder + recovery hint — the HTTP-200 refusal handler and the existing exception-path handler independently built the same terminal result dict and ended their message with the same hermes fallback add trailer copied verbatim. Funnel both through a shared _content_policy_blocked_result() builder + _CONTENT_POLICY_RECOVERY_HINT constant, collapse the HTTP-200 path's two near-identical templates into one, and pass reason=FailoverReason.content_policy_blocked.value instead of a string literal (matching the sibling hook call). Behavior-preserving: the provider-filter vs model-declined lead-in wording stays distinct, and the with-text and exception messages are byte-identical to before.

Tests

  • tests/agent/transports/test_transport.py — empty-content refusal validates.
  • tests/agent/transports/test_chat_completions.py — message.refusal promotion; bare content_filter passthrough (OpenRouter); no-op when absent; partial-content not terminated and tool-calls-alongside-refusal not terminated (commit 3 guards).
  • tests/run_agent/test_run_agent.py::TestRetryExhaustion::test_content_filter_refusal_surfaced_not_retried — end-to-end: a portal-style message.refusal is surfaced as content_policy_blocked with the refusal text, model called exactly once (no retry loop).
  • Existing tests/run_agent/test_18028_content_policy_blocked.py (exception path) stays green through the shared-helper refactor.

Local: ruff clean; full transports + run_agent + content-policy + fallback suites pass (740 passed).

Closes #43108

SHL0MS and others added 4 commits June 14, 2026 12:10
A Claude refusal (HTTP 200, stop_reason="refusal", empty content) was
laundered into a generic retry loop and surfaced as a misleading
"rate limited / invalid response" or "no content after retries" error,
burning paid attempts reproducing a deterministic refusal.

This hit two distinct paths:

- Direct Anthropic (anthropic_messages): validate_response rejected the
  empty-content refusal *before* normalize_response mapped refusal ->
  content_filter, so it fell into the invalid-response retry loop.
- Nous Portal / OpenAI-compatible (chat_completions): the portal surfaces
  a Claude refusal via message.refusal with empty content, which sailed
  past validation and died in the empty-response retry loop.

Fix (one unified content_filter dispatch for all backends):
- AnthropicTransport.validate_response: accept empty content when
  stop_reason == "refusal" so it flows to normalize_response.
- ChatCompletionsTransport.normalize_response: promote message.refusal to
  content + a content_filter finish reason.
- conversation_loop: handle finish_reason == "content_filter" - fire the
  api_request_error hook (content_policy_blocked), try a configured
  fallback once, else return a clear terminal refusal message. Never retry
  a deterministic refusal.

Supersedes NousResearch#43084, which fixed only the direct-Anthropic path and could
not reach the chat_completions/portal path.

Tests: transport-level (validate_response refusal, message.refusal
promotion) + end-to-end loop (refusal surfaced, exactly one API call).

(cherry picked from commit 01f546f)
OpenRouter (and every other OpenAI-compatible provider) uses the default
chat_completions transport, so it is already covered by the refusal fix:
an upstream Claude / moderation refusal arrives as
finish_reason="content_filter" (often empty content, no message.refusal).
Add a regression test asserting the transport passes that finish reason
straight through to the loop's content_filter handler.

(cherry picked from commit 60168a5)
… payload

A chat-completions response that carries real text or tool calls *alongside*
a `message.refusal` note is a normal, usable turn — the model did work. The
prior logic flipped finish_reason to `content_filter` whenever a refusal
string was present, so the conversation loop reframed a content-bearing turn
as a *failed* safety refusal (failed=True) and buried the model's actual
output inside the "model declined" template, or dropped tool calls entirely.

Only promote to a terminal `content_filter` when the refusal is the sole
payload (no visible text AND no tool calls). The refusal explanation is still
recorded in provider_data in every case for observability. Refusal-only
responses (the bug this feature targets) are unaffected and still surface
terminally; the empty+refusal, bare content_filter passthrough, and no-refusal
common cases are byte-identical to before.

Updates the partial-content test to the corrected contract and adds a
tool_calls-alongside-refusal regression guard.
…covery hint

The HTTP-200 refusal handler (finish_reason=content_filter) and the
exception-path handler (a provider moderation error classified as
content_policy_blocked) independently built the same terminal turn result —
the same {final_response, messages, api_calls, completed:False, failed:True,
error:'content_policy_blocked: ...'} dict — and ended their user-facing
message with the same 'Try rephrasing... hermes fallback add' trailer, copied
verbatim. The two copies could drift.

Funnel both through a shared _content_policy_blocked_result() builder and a
shared _CONTENT_POLICY_RECOVERY_HINT constant. Also collapse the HTTP-200
path's two near-identical with/without-explanation templates into one (compute
the detail fragment once) and pass reason=FailoverReason.content_policy_blocked
.value to the error hook instead of a hand-written string literal, matching the
sibling hook call.

Behavior-preserving: the provider/refusal lead-in wording stays distinct (a
provider safety filter vs the model declining are genuinely different signals),
the with-text and exception messages are byte-identical to before, and the
no-explanation case only gains a paragraph break for consistency. Surfaced by
the simplify-code reuse/quality reviewers.

The efficiency reviewer's 'redundant normalize_response' flag was deliberately
NOT applied: that branch is cold (refusal-only) and pure-CPU, and reusing the
sibling-branch normalized locals would risk a NameError on the codex_responses
path (which sets finish_reason without normalizing) — re-normalizing is the
robust choice.
@kshitijk4poor
kshitijk4poor enabled auto-merge June 14, 2026 06:54
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 14, 2026
@kshitijk4poor
kshitijk4poor merged commit c8ad2ca into NousResearch:main Jun 14, 2026
20 checks passed
AIalliAI pushed a commit to AIalliAI/Hermes that referenced this pull request Jun 14, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
T02200059 pushed a commit to T02200059/hermes-agent that referenced this pull request Jun 18, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
Methodician added a commit to Methodician/hermes-agent that referenced this pull request Jul 4, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
@kshitijk4poor
kshitijk4poor deleted the salvage/refusal-content-filter branch August 5, 2026 07:09
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
teknium1 added a commit that referenced this pull request Sep 15, 2026
…ncode#43343)

A model that declines mid-stream delivers the explanation on the
structured refusal channel (chat_completions delta.refusal; Responses
response.refusal.delta / refusal content parts) and leaves content
empty. The streaming accumulators dropped that channel entirely, so a
streamed refusal assembled into an empty message and fell into the
empty/invalid-response retry loops - burning paid retries reproducing a
deterministic refusal - while the non-streaming path had already fixed
this class in #46013.

- chat_completions streaming: accumulate delta.refusal (incl.
  model_extra), expose message.refusal on the assembled mock response so
  ChatCompletionsTransport.normalize_response applies the existing
  sole-payload -> content_filter promotion; count refusal deltas in the
  zero-chunk guard; carry refusal in the Relay final-response dict.
- Codex Responses stream consumer: collect response.refusal.delta as
  answer text so a refusal-only stream no longer raises 'did not emit a
  terminal response' with zero usable content.
- Responses normalizer: read type=refusal content parts in
  _extract_responses_message_text (attr and dict shapes).

Sabotage-verified: each new test fails with its wiring line disabled.
E2E: refusal-only stream -> terminal content_filter with explanation;
refusal-alongside-content stays a normal usable turn; plain-text
streams unchanged.
teknium1 added a commit that referenced this pull request Sep 15, 2026
…ncode#43343)

A model that declines mid-stream delivers the explanation on the
structured refusal channel (chat_completions delta.refusal; Responses
response.refusal.delta / refusal content parts) and leaves content
empty. The streaming accumulators dropped that channel entirely, so a
streamed refusal assembled into an empty message and fell into the
empty/invalid-response retry loops - burning paid retries reproducing a
deterministic refusal - while the non-streaming path had already fixed
this class in #46013.

- chat_completions streaming: accumulate delta.refusal (incl.
  model_extra), expose message.refusal on the assembled mock response so
  ChatCompletionsTransport.normalize_response applies the existing
  sole-payload -> content_filter promotion; count refusal deltas in the
  zero-chunk guard; carry refusal in the Relay final-response dict.
- Codex Responses stream consumer: collect response.refusal.delta as
  answer text so a refusal-only stream no longer raises 'did not emit a
  terminal response' with zero usable content.
- Responses normalizer: read type=refusal content parts in
  _extract_responses_message_text (attr and dict shapes).

Sabotage-verified: each new test fails with its wiring line disabled.
E2E: refusal-only stream -> terminal content_filter with explanation;
refusal-alongside-content stays a normal usable turn; plain-text
streams unchanged.
fsca8 pushed a commit to fsca8/hermes-agent that referenced this pull request Oct 6, 2026
…sal-content-filter

fix(agent): surface model refusals as content_filter (salvage NousResearch#43108 + edge-case fix)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants