Summary

A live cron job inherited a temporary global provider change to provider=nous / model=anthropic/claude-fable-5, continued making paid inference calls after the operator attempted to stop it, and only stopped once the Nous credential/funds path failed.

This created real spend on a protected root-machine runtime where the operator expected a strict no-spend default.

Related existing issue: #19615 (Cron jobs should distinguish global vs pinned model configuration), but this report adds a concrete production incident showing the failure mode is more severe than config ambiguity alone.

Impact

• Unintended paid inference calls against Nous (anthropic/claude-fable-5)
• At least one real affected operator incident with reported credit loss of USD $7.73
• Operator pause/stop actions did not immediately contain spend
• Gateway stop was followed by service restart behavior, reopening cron execution

Environment

• Host: macOS 26.5.1
• Hermes home: profile default under ~/.hermes
• Runtime surface: gateway + cron enabled
• Repo: NousResearch/hermes-agent
• Observed on: 2026-06-11 around 20:10–20:20 local time

Expected behavior

1. A cron job should not silently inherit a temporary global provider change for paid inference unless explicitly pinned/approved.
2. Pausing a cron job should make it obvious whether an in-flight run will continue, and ideally provide a hard-stop path for the running execution.
3. Stopping the gateway as an emergency containment action should not allow the same spend-risk surface to revive unexpectedly without the operator explicitly choosing restart behavior.
4. For shared live runtimes, provider/auth mutation should fail closed or loudly warn when enabled cron agent jobs could inherit the change.

Actual behavior

• Cron job 3590b82a5f25 (Desk Drift daily conversion sales push) ran with:
  • provider=nous
  • base_url=https://inference-api.nousresearch.com/v1
  • model=anthropic/claude-fable-5
• It continued making repeated paid calls after pause attempts.
• Gateway stop was followed by service restart behavior, and cron resumed.
• First termination attempt did not fully kill the in-flight job tree.
• Nous calls continued until the credential/funds path failed with 401, after which fallback engaged.

Evidence from logs

From ~/.hermes/logs/agent.log:

Job starts on Nous/Fable 5

• 2026-06-11 20:10:13,258 ... provider=nous base_url=https://inference-api.nousresearch.com/v1 model=anthropic/claude-fable-5
• 2026-06-11 20:10:15,905 [cron_3590b82a5f25_20260611_201005] ... model=anthropic/claude-fable-5 provider=nous

Repeated paid API calls

• 20:10:23 ... API call #1 ... provider=nous ... model=anthropic/claude-fable-5
• 20:10:36 ... API call #2 ... provider=nous ... model=anthropic/claude-fable-5
• 20:10:44 ... API call #3 ... provider=nous ... model=anthropic/claude-fable-5
• 20:11:01 ... API call #4 ... provider=nous ... model=anthropic/claude-fable-5
• calls continued through at least #28

Gateway stop did not equal permanent containment

• 20:12:23 ... gateway.run: Received SIGTERM — initiating shutdown
• 20:12:25 ... gateway.run: Gateway stopped
• 20:12:25 ... Exiting with code 1 ... so systemd Restart=on-failure can revive the gateway.
• 20:12:28 ... gateway.run: Starting Hermes Gateway...
• 20:12:33 ... gateway.run: Cron ticker started

Spend continued much later in the same in-flight cron run

• 20:19:18 ... API call #27 ... provider=nous ... model=anthropic/claude-fable-5
• 20:19:25 ... API call #28 ... provider=nous ... model=anthropic/claude-fable-5

Only then did Nous fail and fallback occur

• 20:19:51 ... HTTP 401: Your API key is invalid, blocked or out of funds
• 20:19:52 ... Fallback activated: anthropic/claude-fable-5 → gpt-5.4-mini (openai-codex)

Why this is dangerous

This is not just “cron inherited the wrong provider.” It is a compound failure mode:

1. Temporary global provider mutation is visible to live cron agents.
2. Pause does not stop an in-flight run.
3. Gateway stop may restart under service policy.
4. The operator does not get a single reliable “panic stop paid inference now” path.
5. The system can continue burning credits during containment attempts.

For operators using Hermes on a root machine with autonomous jobs, this is a serious trust/safety issue.

Suspected root causes

• Cron jobs can inherit live global provider/model state instead of being insulated from it.
• Pause semantics do not strongly separate “future schedules disabled” from “current in-flight run is still alive.”
• Gateway stop semantics do not provide an obvious emergency no-restart containment mode.
• There may be no enforced interlock that warns/blocks provider mutation while enabled agent cron jobs exist.

Suggested fixes

1. Add a hard interlock before provider/auth/model mutation when enabled agent cron jobs exist.

   • At minimum: loud confirmation + list of affected jobs
   • Preferably: fail closed unless --force-live-runtime-mutation or equivalent

2. Add explicit per-job pinning semantics and surface unpinned jobs as risky.

   • Jobs without explicit provider/model should be clearly marked as inheriting global state.

3. Add a true emergency stop for cron agent executions.

   • Stop future schedules
   • Kill in-flight job process trees
   • Confirm zero active agent runs

4. Add a gateway stop mode that disables automatic service revival for emergency containment.

5. Add log / UI language that distinguishes:

   • job paused for future ticks
   • job currently running
   • running process tree killed / not killed

6. Consider a zero-spend lock mode for root/shared runtimes.

   • If enabled, paid-capable provider changes and paid fallbacks are blocked unless explicitly approved per action.

Reproduction sketch

I cannot provide a clean minimal repro without intentionally recreating paid spend, but the high-level sequence was:

1. Have gateway + cron enabled with at least one unpinned agent cron job.
2. Temporarily switch global provider/model to a paid provider path.
3. Allow cron to tick / run.
4. Attempt to pause/stop after launch.
5. Observe that the in-flight job can continue on the inherited provider.
6. Observe that gateway/service restart behavior can reopen cron execution unless explicitly neutralized.

Request

Please treat this as a real bug report, not just operator error. The operator made mistakes during containment, but Hermes should make this class of spend-risk incident much harder to trigger and much easier to stop.