Skip to content

CHRONICLE_ENABLED=0 silently disables budget enforcement: spend is never recorded, so no cap trips #151

Open

Description

@susheem-k

Context

TokenOps caps spend for a whole agent workflow and enforces the cap before every model call. The pieces this issue touches:

  • run: one whole workflow, not one request. It shares a run_id and one ledger. tokenops_run() opens or joins a run.
  • wrap_complete(): wraps the model call. It is the enforcement point.
  • Ledger: records spend and holds the halt flag. Amounts are Micros, integer micro-USD ($1.00 == 1_000_000).
  • Governor: runs the policies (for example cost_budget) built for the run.
  • Halt: raised to abort a run. It extends BaseException on purpose (control/core.py:246) so a framework's except Exception cannot swallow it.

TokenOps does not record spend itself. It learns what a call cost from Chronicle, the record-and-replay library it depends on. The order inside one wrap_complete call (control/integration.py):

  1. governor.pre_call(request) (:267): checks the budget. Before the provider is contacted.
  2. governor.ledger.admit(seg) (:275): marks a call in flight.
  3. traced(...) (:283): traced is wrap_llm(boundary_id, dispatch) (:251). Chronicle calls the provider, records the crossing, then calls the session hook on_crossing.
  4. crossing.on_crossing (control/crossing.py:134) calls emit_observation (:171), which calls Governor.observe (control/boundary.py:132). observe calls ledger.record(obs) (control/engine.py:311): this is the only place spend is written.
  5. if controls.retry (:284): retry decision, using what step 4 recorded.

This is deliberate. The module docstring says the observation comes only through the hook, "do not also emit_observation here (double-billing)" (integration.py:14).

Problem

Setting CHRONICLE_ENABLED=0 silently turns off budget enforcement.

Chronicle documents that variable as a way to disable LIVE recording for A/B timing runs (Chronicle README.md:633). When it is off, Chronicle's wrappers return before step 3's hooks fire (chronicle/boundary.py:147, :171), so step 4 never happens and the ledger never sees any spend.

Repro (setup for reg, attr, run_scope and the run's ledger entry is as in tests/test_llm_boundary_precall.py; budget cap 100 micros; the fake provider costs 4000 micros per call under the test price function toy_price in tests/conftest.py:24, which charges 10 micros per input token and 30 per output token; 5 calls through wrap_complete):

cfg = {"governance": {"budgets": [{"id": "b", "limit_micros": 100, "dimension": "run"}],
                      "policies": {"cost_budget": {"budget": "b"}}}}
controls = ApplyControls()
gov = build_governor(cfg, toy_price, controls)
dispatch = lambda p, m, messages, max_output_tokens=None, **kw: ModelResponse(
    content="x", input_tokens=100, output_tokens=100)
governed = wrap_complete(gov, controls, attr, provider="openai", model="gpt-4o-mini",
                         dispatch=dispatch, service="agent")
with run_scope(reg, SpanContext(span_id="s", service="agent")), \
     _governance_scope(gov, attr, provider="openai", model="gpt-4o-mini"):
    for _ in range(5):
        governed("openai", "gpt-4o-mini", [{"role": "user", "content": "hi"}])

Observed:

CHRONICLE_ENABLED unset: model calls made=1, halted=Halt, ledger cost_micros=4000
CHRONICLE_ENABLED=0:     model calls made=5, halted=None, ledger cost_micros=0

What the user sees versus what is happening. They see nothing: no exception, no log line, no Halt, five successful model calls. What is happening is that pre_call still runs, but it compares a cap against a ledger that stays at zero, so it always passes. The spend was real (20,000 micros against a 100 micro cap) and is not recorded anywhere.

Blast radius. Every provider, and every policy that reads spend from the ledger (cost_budget is the one the repro exercises). Anyone who sets the variable in a shell or .env file for a Chronicle benchmark, then runs a governed agent, gets no enforcement. I verified this by running wrap_complete. I did not run wrap_stream (it has the same wrap_llm shape, integration.py:392) or the hook-only path where an @boundary(kind="llm") is governed by crossing.on_enter (crossing.py:57); by reading chronicle/boundary.py the second one never reaches on_enter at all when the switch is off.

Nothing in the code marks this as deliberate: the docstring at integration.py:14 says spend arrives through the hook, and does not say what happens when the hook cannot fire.

Proposed approach

Refuse to start a governed run when the hooks TokenOps depends on cannot fire. Fail at the start of the run, not on the first call.

# control/run.py, top of tokenops_run(), next to install_crossing_hook() (run.py:175)
import chronicle

if not chronicle.is_enabled():
    raise GovernanceUnavailable(
        "CHRONICLE_ENABLED is off. TokenOps records spend through Chronicle's on_crossing "
        "hook, so no spend would be recorded and budgets would not be enforced. "
        "Unset CHRONICLE_ENABLED to run governed."
    )
  • GovernanceUnavailable is a new RuntimeError subclass exported from tokenops.control. It is not a BaseException like Halt: it is raised on entering with tokenops_run(): before any agent code runs, so nothing can swallow it.
  • Add the repro above as a regression test in tests/ using monkeypatch.setenv("CHRONICLE_ENABLED", "0"), asserting the raise. Add a second test asserting no raise when the variable is unset.
  • Document the requirement in docs/architecture.md next to the description of the crossing hook.

What stays the same: with CHRONICLE_ENABLED unset or on, nothing changes. No new config, no migration.

Open question (recommendation attached). Should there be an opt-out for someone who deliberately wants to run without accounting? I recommend no opt-out. That case is already covered by not calling tokenops_run, and an opt-out flag would be the way an accidental .env value goes unnoticed again.

This is the TokenOps half of a two-part fix. The root cause is in Chronicle, where one switch turns off both recording and interceptor hooks: theagentplane/chronicle#60. When that lands, this guard should be relaxed to a check that hooks are registered rather than that Chronicle is enabled.

Tradeoffs / non-scope

  • Accepted: anyone who had set CHRONICLE_ENABLED=0 alongside tokenops_run will now get an error instead of an unenforced run. That is the desired outcome, since those runs had no cap.
  • Accepted: the guard runs once at the start of a run. If the variable is flipped mid-run (is_enabled() reads the environment on every call), it will not catch that. I judge that not worth a per-call check.
  • Rejected: have wrap_complete record its own spend so the ledger no longer needs the hook. It would fix the wrap_complete path, but it reverses the single-observation rule at integration.py:14, needs a second skip mechanism in the hook to avoid double-billing, and leaves the hook-only path unenforced. Whether TokenOps should own its accounting is a design discussion that belongs in its own issue.
  • Out of scope: the Chronicle-side change (CHRONICLE_ENABLED=0 also disables on_enter/on_crossing/on_leave, so switching off recording silently switches off consumers built on the hooks chronicle#60). Calls that never enter the governed wrapper at all (Fail-closed when a run has no governed dispatch, instead of silently running ungoverned #109, which is about raw complete bypassing governance, a different cause with the same symptom of silent non-enforcement). The two governance paths doing different things (hook versus wrap_complete): needs a new issue.

Activity

  1. Yashika-1705 commented on Sep 26, 2026

    @Yashika-1705
    Contributor

    Hi @susheem-k, I'd like to work on this!

    I can implement the proposed fail-closed check:

    • Add and export GovernanceUnavailable from tokenops.control
    • Add the chronicle.is_enabled() check at the start of tokenops_run()
    • Add tests covering CHRONICLE_ENABLED=0 and the default unset case
    • Update docs/architecture.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions