Skip to content

fix(gateway): prevent per-turn agent eviction from model normalization mismatch and signature instability#21299

Closed
fayenix wants to merge 4 commits into
NousResearch:mainfrom
chiefmojo:fix/cache-eviction
Closed

fix(gateway): prevent per-turn agent eviction from model normalization mismatch and signature instability#21299
fayenix wants to merge 4 commits into
NousResearch:mainfrom
chiefmojo:fix/cache-eviction

Conversation

@fayenix

@fayenix fayenix commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the root cause of per-turn agent eviction that causes memory provider bridges to spawn on every message (issue #20939).

Two independent bugs in the gateway agent cache were causing the cached AIAgent (and its MemoryProvider) to be destroyed and recreated on every turn instead of persisting across messages:

  1. Model normalization mismatch_agent.model (normalized by AIAgent.__init__) never matched _cfg_model (raw gateway config with vendor prefix), so the eviction check at ~L13131 fired every turn.
  2. Session signature instabilitywas_auto_reset notice and message_id were feeding into the cache signature hash, causing it to flip between turns.

Root Cause Details

1. Model Normalization Mismatch

gateway/run.py ~L13131 compares the cached agent's model against the gateway config model to decide whether to evict:

  • _agent.model = "deepseek-v4-pro" (normalized by AIAgent.__init__ via normalize_model_for_provider)
  • _cfg_model = "deepseek/deepseek-v4-pro" (raw gateway config, vendor-prefixed)

"deepseek-v4-pro" != "deepseek/deepseek-v4-pro"_evict_cached_agent()AIAgent destroyed → MemoryProvider destroyed → next turn creates fresh MemoryProviderinitialize() → new bridge pair. Every. Single. Turn.

Fix: Normalize _cfg_model using the same normalize_model_for_provider() function that AIAgent.__init__ uses, before the equality check.

2. Session Signature Instability

Even after the model fix, the cache signature was flip-flopping (ec05b8ea → 67f721c3 → f3d18768 → 67f721c3). Two unstable inputs were feeding the sig hash:

  • was_auto_reset notice was prepended to context_prompt, which feeds into the sig hash. Being a one-turn transient flag, it flipped the sig on the first post-restart turn, then back.
  • message_id was in the Discord IDs block in build_session_context_prompt(). Per-message value would change the sig every turn.

Fix: Decouple the reset notice from context_prompt (inject into combined_ephemeral after sig is computed). Remove message_id from the context prompt.

Results (live gateway, ~16 hour burn-in)

# Bridge processes: 8 (4 sessions × 2 processes)
# Before fix: 60-100 bridges/day

# Cache log — 20+ consecutive HITs overnight:
04:56  CACHE HIT (sig=155bc36b)  Sanctuary
05:21  CACHE HIT (sig=155bc36b)
05:42  CACHE HIT (sig=155bc36b)
...
06:33  CACHE HIT (sig=155bc36b)
06:35  CACHE HIT (sig=155bc36b)

# Zero "CACHE MISS: sig changed" lines since the fix.
# Signature stable across all sessions.

Commits

Commit Description
d480b910b Fix silent post-turn agent eviction (model normalization mismatch)
81ca7089f Fix sig instability: decouple was_auto_reset + remove message_id from hash
83276fb4c Fix NameError from closure scoping bug in d480b91
d7e2683c3 chore: bump Discord VOICE_TIMEOUT 300s→3600s

Related

fayenix added 3 commits May 7, 2026 06:40
Root cause:  returns vendor-prefixed model name
(e.g. 'deepseek/deepseek-v4-pro') while AIAgent.__init__ normalizes it by
stripping the prefix ('deepseek-v4-pro'). The comparison at run.py:13131
was always True, causing _evict_cached_agent to fire after EVERY turn.

Fix: normalize _cfg_model the same way AIAgent.__init__ does (importing
normalize_model_for_provider from hermes_cli.model_normalize) before
the equality check. Also add debug logging to _evict_cached_agent so
evictions are traceable in gateway logs.

- gateway/run.py: add model normalization block at line 13131
- gateway/run.py: add logger.debug to _evict_cached_agent

Thanks to Claude for the root-cause analysis and fix.
…g agent cache signature

Two independent fixes for session signature instability causing
spurious agent evictions mid-conversation:

1. gateway/run.py: Decouple was_auto_reset notice from context_prompt.
   Previously, the '[System note: session suspended/reset...]' notice
   was prepended to context_prompt, which feeds into the cache signature
   hash. Since was_auto_reset is a one-turn transient flag, the sig
   would flip on the first post-restart/reset turn, evict the cached
   agent, then flip back on the next turn — causing back-to-back
   CACHE MISSes (the ec05b8ea→67f721c3→f3d18768→67f721c3 flip-flop).

   Fix: store as _reset_context_note without modifying context_prompt.
   Inject into combined_ephemeral AFTER _sig is computed, so the agent
   still sees the notice but the sig stays stable.

2. gateway/session.py: Remove 'Triggering message: <message_id>' from
   the Discord IDs block in build_session_context_prompt(). message_id
   is per-message and would cause _sig to change every single turn if
   the discord/discord_admin toolset is ever enabled (latent bomb).

Root cause analysis: Claude (via fayenix)
Branch: fix/discord-voice-fixes
…ping bug)

The _reset_context_note variable was defined in _handle_message_with_agent
but referenced in run_sync, a nested function inside _run_agent. Python
closures can see variables from enclosing scopes, but only up to the
nearest enclosing function — run_sync cannot see _reset_context_note
because it's in a different function (_handle_message_with_agent), not
in _run_agent.

Fix: add reset_context_note as a parameter to _run_agent (line 11616),
pass it from the call site (line 6070), and reference the parameter
name in run_sync instead of the outer function's local.

Fixes: NameError "name '_reset_context_note' is not defined"
Bug introduced in: d480b91
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery labels May 7, 2026

@liuhao1024 liuhao1024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall fix — deferring the reset-context-note injection past _sig computation and normalizing _cfg_model — looks correct and well-motivated.

One bug on the diagnostic logging path:

_cached_for_log = cached if '_cache_lock' in dir() and cached is not None else None

dir() (with no arguments) returns local scope names. There is no local variable named _cache_lock in this scope — the lock is _agent_cache_lock — so '_cache_lock' in dir() is always False. This means _cached_for_log is always None and the "CACHE MISS: sig changed (old=%s new=%s)" log will never fire, even when a signature mismatch is the actual cause.

Fix — simplify to just check cached directly:

_cached_for_log = cached
if _cached_for_log:
    logger.info("CACHE MISS: sig changed (old=%s new=%s)",
               _cached_for_log[1][:8], _sig[:8])
else:
    logger.info("CACHE MISS: no cached agent for session %s (sig=%s)", session_key, _sig[:8])

@rxdxxxx

rxdxxxx commented May 8, 2026

Copy link
Copy Markdown

Confirming this bug from a different angle — config model change not taking effect in existing session

We hit this in production on our Weixin (WeChat) gateway today. Symptoms:

Setup:

  • Changed config.yaml model from minimax-m2.5-freemimo-v2.5-pro
  • Gateway was restarted after the config change

Observed:

  • session_meta in the JSONL correctly recorded mimo-v2.5-pro
  • But the assistant's reasoning revealed the system prompt still contained:
    Model: minimax-m2.5-free
    Provider: minimax
    
  • The agent was clearly self-identifying as the old model

Resolution:

  • Sending /new to reset the session immediately fixed the issue
  • New session correctly showed Model: xiaomi/mimo-v2-pro in the system prompt

Root cause analysis:

The cached AIAgent (constructed with the old model name baked into its system prompt) was being reused across the config change. The eviction check at ~L13131:

if _agent.model != _cfg_model:
    _evict_cached_agent()

…failed to trigger because of the normalization mismatch described in this PR. The _cfg_model (raw from config) and _agent.model (normalized by AIAgent.__init__) were being compared without consistent normalization, causing the eviction logic to either:

  1. Not fire when it should (our case — stale agent persists), or
  2. Fire every turn when it shouldn't (the MemOS bridge spawn issue you describe)

This PR's fix to normalize _cfg_model before comparison would have caught our case. Looking forward to this being merged — it's a real user-facing bug that causes confusion ("why is the model still the old one after I changed config?").

Environment: macOS, gateway with telegram + feishu + weixin platforms, hermes-agent from fork.

liuhao1024 caught that '_cache_lock' in dir() always returns False
because dir() with no arguments checks local scope, and the lock is
_agent_cache_lock. Simplify to check cached directly so the
'CACHE MISS: sig changed' log actually fires when it should.

Co-authored-by: liuhao1024 <[email protected]>

@fayenix fayenix left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you! Fixed in 6d2c34e — simplified to just check cached directly. You were absolutely right that dir() with no args only sees local scope, and the lock is _agent_cache_lock, not _cache_lock — so the diagnostic log literally never fired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: MemOS memory provider spawns new bridge.cts process on every turn instead of once per session

5 participants