Skip to content

Commit 1a44050

Browse files
authored
fix(sleep): enforce max_tokens_per_night in the nightly cycle (#317)
* fix(sleep): enforce max_tokens_per_night in the nightly cycle The config default and the Budget controller existed, but nothing in the engine read the key or constructed a Budget, so a night ran every step no matter how many tokens it had already used. Build a Budget from max_tokens_per_night at the start of each cycle and check it, against the backend's token counter, before each step that would start new model work: the consolidation, its skill and memory edit steps, the held-out test replay and every per-skill fan-out group. Steps that never started are listed in a report note, in budget skip evidence events and in the cycle end event. A started step always finishes, so a gate decision never rests on partial evidence and nothing is cut off mid-write. A handoff round counts unanswered prompts and placeholder replies, so when the budget is reached while prompts are still pending the round pauses for answers instead of stopping. Otherwise it would stage and save a night built from placeholders. If the budget is already spent before consolidation, the night records no held-out scores instead of zeros. 0 or null keeps the night unbudgeted, matching Budget's existing semantics. Negative, boolean, fractional, infinite or non-numeric values fail before any model call. Only tonight's spend counts, so a backend reused across nights is not stopped early. Fixes #313 * fix(sleep): skip the final validation replay once the budget is spent The budget check assumed the final validation replay was free because it re-scores documents an earlier replay already scored. That holds for cached text replies, but tool replay is never cached and a failed or timed out call is not kept in the cache, so on those routes the final replay started new provider calls after both edit steps had been skipped. Check the budget before the final validation replay as well. When it is spent, decide from the validation replay that already scored the final skill and memory (the baseline replay, or the trial replay of the last kept edit) instead of starting a new one, and list the skipped replay in the report note and the budget skip events. Nights that never reach the budget still run the fresh final replay as before. The new tests drive ClaudeCliBackend through its real tool and text routes with a synthetic process standing in for the claude CLI and count every process it starts, for answered and timed out calls.
1 parent 618d5d2 commit 1a44050

9 files changed

Lines changed: 918 additions & 27 deletions

File tree

‎docs/sleep/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,60 @@ documents the separate HTTPS-only boundary for Azure managed-identity credential
358358
Deterministic proof (no API key):
359359
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves`.
360360

361+
### Nightly token budget
362+
363+
`max_tokens_per_night` in `~/.skillopt-sleep/config.json` (default `400000`)
364+
stops a `run` or `dry-run` from starting more model work once tonight's spend
365+
reaches it. The cycle checks the budget before the consolidation, before its
366+
skill and memory edit steps, before its final validation replay, before the
367+
held-out test replay, and before each per-skill fan-out group. Steps that did
368+
not start are listed in a note in `report.md`, for example
369+
`max_tokens_per_night reached (431870 of 400000 tokens); skipped: memory edits, final validation replay`.
370+
371+
When the final validation replay is skipped, the gate decides from the
372+
validation replay that already scored the same skill and memory: the baseline
373+
replay when no edit was kept, or the trial replay of the last edit that was
374+
kept. That replay is complete, so no decision rests on partial evidence, and
375+
it is not repeated. This matters for replays the response cache does not
376+
serve, such as tool replay or a call that failed or timed out, which would
377+
otherwise start the model again.
378+
379+
```json
380+
{"max_tokens_per_night": 200000}
381+
```
382+
383+
Set the value to `0` or `null` for no budget. Anything other than a whole,
384+
non-negative number stops the run with an error before any model call. There
385+
is no elapsed-time budget.
386+
387+
This is not a hard cap. A step that has started always finishes, so a night
388+
can end above the budget by the cost of one step: the first replay of the
389+
validation and training tasks, one edit step (a training replay, a reflect
390+
call and a validation replay of the candidate; with `dream_rollouts` set to K,
391+
K attempts per training task), the final validation replay, or the held-out
392+
test replay. Each fan-out group that starts runs its own first replay.
393+
394+
Spend comes from the backend's token counter, the one the report header prints
395+
as `tokens used`. The header shows the backend's running total and the note
396+
shows tonight's spend; they are equal unless a caller reuses one backend object
397+
across nights. What the counter covers depends on the backend:
398+
399+
- CLI backends estimate tokens from prompt and reply length. LLM task mining and
400+
contrastive reflection (`dream_rollouts` above 1) call the model directly and
401+
are not counted.
402+
- `azure_openai` adds the usage reported by the API to that estimate, so it
403+
counts mining, and replay and reflect calls count roughly twice.
404+
- `handoff` counts the prompt and answer text of the current round. A budget
405+
check that finds unanswered prompts pauses the round for answers instead of
406+
stopping it, so a round only stops on the budget once every counted call has
407+
a real answer.
408+
- `mock` reports no usage and never reaches the budget.
409+
410+
A night that stops on the budget still advances the harvest checkpoint, so its
411+
sessions are not mined again, and skipped work is not retried on a later night.
412+
Tonight's tasks stay in the task archive, which later nights only use through
413+
`recall_k`.
414+
361415
### Opt-in: per-skill fan-out
362416

363417
Set `"multi_skill_fanout": true` in `~/.skillopt-sleep/config.json` to add an

‎plugins/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,9 @@ the shipping CLI defaults.
298298
- `mock` and `handoff` make no network calls.
299299
- `run` stages proposals; `adopt` is the normal live-change boundary.
300300
- Adoption backs up existing target files.
301-
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not yet
302-
enforce a hard token or elapsed-time budget.
301+
- `--max-sessions` and `--max-tasks` bound work. `max_tokens_per_night` in the
302+
config file (default `400000`) stops a night from starting new steps once its
303+
token count reaches the budget, but a started step finishes, so it is not a
304+
hard cap. There is no elapsed-time budget.
303305
- Treat real-backend transcript excerpts as data shared with the selected
304306
provider.

‎plugins/claude-code/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -175,8 +175,10 @@ The scheduled run stages proposals unless `--auto-adopt` is explicitly selected.
175175
- **Read-only** harvest of `~/.claude`. `mock` replay has no side effects.
176176
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
177177
- Every adopt writes a backup under the staging dir's `backup/`.
178-
- `--max-sessions` and `--max-tasks` bound work, but the main CLI does not enforce
179-
a hard token or wall-clock budget.
178+
- `--max-sessions` and `--max-tasks` bound work. `max_tokens_per_night` in
179+
`~/.skillopt-sleep/config.json` stops a night from starting new steps once its
180+
token count reaches the budget; a started step finishes, so it is not a hard
181+
cap, and there is no wall-clock budget.
180182
- Real backends share truncated session/task content with the selected provider;
181183
do not assume outbound prompts have been fully redacted.
182184

‎plugins/openclaw/README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,12 @@ staged files before changing a live skill.
5656
paths all point to isolated test locations.
5757

5858
3. Review `config.json`. In particular, do not assume that values such as
59-
`max_tokens_per_night` or `replay_mode` are enforced by this custom wrapper
60-
merely because they appear in the example config.
59+
`replay_mode` are enforced by this custom wrapper merely because they appear
60+
in the example config. `max_tokens_per_night` is applied by the shared cycle,
61+
but it is measured with the wrapper backend's rough `tokens_used()` estimate
62+
(a quarter of the attempt prompt length plus 200 per attempt, 600 per rubric
63+
judge call, and a third of the reflect prompt plus 1500 per reflect call), so
64+
it is not a billing limit.
6165

6266
4. Supply credentials through your normal secret-management mechanism. Do not
6367
commit a DeepSeek key or place it in a world-readable file.

‎skillopt_sleep/consolidate.py‎

Lines changed: 53 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010

1111
import math
1212
from dataclasses import dataclass, field
13-
from typing import List, Tuple
13+
from typing import Callable, List, Optional, Tuple
1414

1515
from skillopt_sleep.backend import Backend
1616

@@ -23,6 +23,10 @@
2323

2424
_HAVE_REPO_GATE = True
2525

26+
# gate_action of an epoch that never ran because the token budget was spent.
27+
# Its scores were not measured, so callers must not report them as results.
28+
BUDGET_EXHAUSTED = "budget_exhausted"
29+
2630

2731
def _finite_score(value: float) -> float | None:
2832
"""Return a JSON-safe score, preserving finite values only."""
@@ -59,6 +63,8 @@ class ConsolidationResult:
5963
# fresh final replay, so aggregate changes never hide an individual task.
6064
# Keep new optional fields last to preserve positional construction.
6165
gate_trials: List[dict] = field(default_factory=list)
66+
# Steps that never started because the caller's token budget was spent.
67+
budget_skipped: List[str] = field(default_factory=list)
6268

6369

6470
def _split(tasks: List[TaskRecord]) -> Tuple[List[TaskRecord], List[TaskRecord], bool]:
@@ -192,6 +198,7 @@ def consolidate(
192198
evolve_skill: bool = True,
193199
evolve_memory: bool = True,
194200
night: int = 1,
201+
budget_exhausted: Optional[Callable[[], bool]] = None,
195202
) -> ConsolidationResult:
196203
"""Run one consolidation epoch: reflect -> bounded edit -> gate.
197204
@@ -201,12 +208,44 @@ def consolidate(
201208
recorded so the report shows whether quality moved.
202209
203210
Skill and memory are evolved in sequence (skill first if both enabled).
211+
212+
``budget_exhausted`` is checked before the epoch starts, before each
213+
evolve target and before the final validation replay. Once it returns
214+
True, the steps that have not started are skipped and listed in
215+
``budget_skipped``. A started step always finishes, so every gate decision
216+
still rests on complete evidence. When the final replay is skipped, the
217+
decision uses the validation replay that already scored the final skill
218+
and memory: the baseline replay, or the trial replay of the last kept edit.
204219
"""
205220
from skillopt_sleep import evidence as evlog
206221
ev = evlog.get(backend)
207222
train_tasks, val_tasks, holdout_leaked = _split(tasks)
208223
gate_off = str(gate_mode).strip().lower() in {"off", "none", "false", "greedy"}
209224
holdout_detail: List[dict] = []
225+
budget_skipped: List[str] = []
226+
227+
def _budget_spent(step: str) -> bool:
228+
if budget_exhausted is None or not budget_exhausted():
229+
return False
230+
budget_skipped.append(step)
231+
if ev is not None:
232+
ev.log("budget", "skip", step=step, night=night)
233+
return True
234+
235+
if _budget_spent("consolidation"):
236+
return ConsolidationResult(
237+
accepted=False,
238+
gate_action=BUDGET_EXHAUSTED,
239+
baseline_score=0.0,
240+
candidate_score=0.0,
241+
new_skill=skill,
242+
new_memory=memory,
243+
applied_edits=[],
244+
rejected_edits=[],
245+
holdout_baseline=0.0,
246+
holdout_candidate=0.0,
247+
budget_skipped=budget_skipped,
248+
)
210249

211250
# ── baseline on the VAL slice (the gate reference) ────────────────────
212251
# When the gate is OFF the user has opted out of holding out a validation set
@@ -307,7 +346,7 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
307346
all_rejected.extend(applied)
308347
return doc
309348

310-
if evolve_skill:
349+
if evolve_skill and not _budget_spent("skill edits"):
311350
if rollouts_k > 1:
312351
# multi-rollout contrastive reflection: run each train task K times
313352
# and distill a rule from the good-vs-bad contrast (the imagination signal).
@@ -354,7 +393,7 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
354393
)
355394
cand_skill = _gate_apply(cand_skill, edits, "skill")
356395

357-
if evolve_memory:
396+
if evolve_memory and not _budget_spent("memory edits"):
358397
# re-evaluate failures under the (possibly improved) skill
359398
evlog.set_phase(backend, "train_post_skill")
360399
train_pairs2 = replay_batch(backend, train_tasks, cand_skill, cand_memory)
@@ -378,8 +417,16 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
378417
base_gate_score = 0.0
379418
else:
380419
# scored on the VAL slice (the gate reference)
381-
evlog.set_phase(backend, "final_val")
382-
final_pairs = replay_batch(backend, val_tasks, cand_skill, cand_memory)
420+
if val_tasks and _budget_spent("final validation replay"):
421+
# Replaying is not free on every route: tool replay and failed calls
422+
# are not served from the response cache. current_pairs already
423+
# scores exactly (cand_skill, cand_memory) on val_tasks (the
424+
# baseline replay, or the trial of the last edit kept), so decide
425+
# from that complete replay instead of starting a new one.
426+
final_pairs = current_pairs
427+
else:
428+
evlog.set_phase(backend, "final_val")
429+
final_pairs = replay_batch(backend, val_tasks, cand_skill, cand_memory)
383430
final_hard, final_soft = aggregate_scores(final_pairs)
384431
final_score = select_gate_score(final_hard, final_soft, gate_metric, gate_mixed_weight)
385432
base_gate_score = select_gate_score(base_hard, base_soft, gate_metric, gate_mixed_weight)
@@ -493,4 +540,5 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
493540
reflect_raw=getattr(backend, "last_reflect_raw", "") or "",
494541
call_error=getattr(backend, "last_call_error", "") or "",
495542
holdout_leaked=holdout_leaked,
543+
budget_skipped=budget_skipped,
496544
)

0 commit comments

Comments
 (0)