Skip to content

Commit 0322fbe

Browse files
committed
fix(sleep): skip the final validation replay once the budget is spent
The budget check assumed the final validation replay was free because it re-scores documents an earlier replay already scored. That holds for cached text replies, but tool replay is never cached and a failed or timed out call is not kept in the cache, so on those routes the final replay started new provider calls after both edit steps had been skipped. Check the budget before the final validation replay as well. When it is spent, decide from the validation replay that already scored the final skill and memory (the baseline replay, or the trial replay of the last kept edit) instead of starting a new one, and list the skipped replay in the report note and the budget skip events. Nights that never reach the budget still run the fresh final replay as before. The new tests drive ClaudeCliBackend through its real tool and text routes with a synthetic process standing in for the claude CLI and count every process it starts, for answered and timed out calls.
1 parent 356f8c8 commit 0322fbe

3 files changed

Lines changed: 226 additions & 15 deletions

File tree

‎docs/sleep/README.md‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -349,10 +349,18 @@ Deterministic proof (no API key):
349349
`max_tokens_per_night` in `~/.skillopt-sleep/config.json` (default `400000`)
350350
stops a `run` or `dry-run` from starting more model work once tonight's spend
351351
reaches it. The cycle checks the budget before the consolidation, before its
352-
skill and memory edit steps, before the held-out test replay, and before each
353-
per-skill fan-out group. Steps that did not start are listed in a note in
354-
`report.md`, for example
355-
`max_tokens_per_night reached (431870 of 400000 tokens); skipped: memory edits`.
352+
skill and memory edit steps, before its final validation replay, before the
353+
held-out test replay, and before each per-skill fan-out group. Steps that did
354+
not start are listed in a note in `report.md`, for example
355+
`max_tokens_per_night reached (431870 of 400000 tokens); skipped: memory edits, final validation replay`.
356+
357+
When the final validation replay is skipped, the gate decides from the
358+
validation replay that already scored the same skill and memory: the baseline
359+
replay when no edit was kept, or the trial replay of the last edit that was
360+
kept. That replay is complete, so no decision rests on partial evidence, and
361+
it is not repeated. This matters for replays the response cache does not
362+
serve, such as tool replay or a call that failed or timed out, which would
363+
otherwise start the model again.
356364

357365
```json
358366
{"max_tokens_per_night": 200000}
@@ -366,8 +374,8 @@ This is not a hard cap. A step that has started always finishes, so a night
366374
can end above the budget by the cost of one step: the first replay of the
367375
validation and training tasks, one edit step (a training replay, a reflect
368376
call and a validation replay of the candidate; with `dream_rollouts` set to K,
369-
K attempts per training task), or the held-out test replay. Each fan-out group
370-
that starts runs its own first replay.
377+
K attempts per training task), the final validation replay, or the held-out
378+
test replay. Each fan-out group that starts runs its own first replay.
371379

372380
Spend comes from the backend's token counter, the one the report header prints
373381
as `tokens used`. The header shows the backend's running total and the note

‎skillopt_sleep/consolidate.py‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -209,11 +209,13 @@ def consolidate(
209209
210210
Skill and memory are evolved in sequence (skill first if both enabled).
211211
212-
``budget_exhausted`` is checked before the epoch starts and before each
213-
evolve target. Once it returns True, the steps that have not started are
214-
skipped and listed in ``budget_skipped``. A started step always finishes,
215-
so every gate decision still rests on complete evidence; the final replay
216-
only re-scores documents an earlier replay already scored.
212+
``budget_exhausted`` is checked before the epoch starts, before each
213+
evolve target and before the final validation replay. Once it returns
214+
True, the steps that have not started are skipped and listed in
215+
``budget_skipped``. A started step always finishes, so every gate decision
216+
still rests on complete evidence. When the final replay is skipped, the
217+
decision uses the validation replay that already scored the final skill
218+
and memory: the baseline replay, or the trial replay of the last kept edit.
217219
"""
218220
from skillopt_sleep import evidence as evlog
219221
ev = evlog.get(backend)
@@ -415,8 +417,16 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
415417
base_gate_score = 0.0
416418
else:
417419
# scored on the VAL slice (the gate reference)
418-
evlog.set_phase(backend, "final_val")
419-
final_pairs = replay_batch(backend, val_tasks, cand_skill, cand_memory)
420+
if val_tasks and _budget_spent("final validation replay"):
421+
# Replaying is not free on every route: tool replay and failed calls
422+
# are not served from the response cache. current_pairs already
423+
# scores exactly (cand_skill, cand_memory) on val_tasks (the
424+
# baseline replay, or the trial of the last edit kept), so decide
425+
# from that complete replay instead of starting a new one.
426+
final_pairs = current_pairs
427+
else:
428+
evlog.set_phase(backend, "final_val")
429+
final_pairs = replay_batch(backend, val_tasks, cand_skill, cand_memory)
420430
final_hard, final_soft = aggregate_scores(final_pairs)
421431
final_score = select_gate_score(final_hard, final_soft, gate_metric, gate_mixed_weight)
422432
base_gate_score = select_gate_score(base_hard, base_soft, gate_metric, gate_mixed_weight)

‎tests/test_sleep_token_budget.py‎

Lines changed: 195 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,13 @@
88
import glob
99
import json
1010
import os
11+
import subprocess
1112
import tempfile
1213
import unittest
1314
from dataclasses import replace
1415
from unittest import mock
1516

16-
from skillopt_sleep.backend import MockBackend
17+
from skillopt_sleep.backend import ClaudeCliBackend, MockBackend
1718
from skillopt_sleep.config import load_config
1819
from skillopt_sleep.consolidate import consolidate
1920
from skillopt_sleep.cycle import run_sleep_cycle
@@ -112,6 +113,68 @@ def _arithmetic_tasks():
112113
return assign_splits(tasks, holdout_fraction=0.34, seed=42)
113114

114115

116+
def _tool_tasks():
117+
"""Tasks whose rule judge needs a real tool call (the tool replay route)."""
118+
tasks = [
119+
TaskRecord(
120+
id=f"look{i}", project="/p",
121+
intent=f"Find note {i} and summarize it.",
122+
reference_kind="rule",
123+
judge={"kind": "rule", "checks": [{"op": "tool_called", "arg": "search"}]},
124+
)
125+
for i in range(1, 7)
126+
]
127+
return assign_splits(tasks, val_fraction=0.34, seed=42)
128+
129+
130+
_REAL_RUN = subprocess.run
131+
_SEARCH_RULE = "Always run ./search before answering."
132+
133+
134+
class SyntheticClaudeProcess:
135+
"""Stands in for the external ``claude`` process; no model is called.
136+
137+
Records every process the backend spawns as ``(phase, route)``, where the
138+
route is ``tools`` for ``attempt_with_tools`` and ``text`` for ``_call``.
139+
Replies with a fixed answer, or raises ``TimeoutExpired`` like a process
140+
that ran past the backend's timeout. A reflect call (text route during the
141+
train phase) proposes ``_SEARCH_RULE``; a tool replay whose prompt carries
142+
that rule logs a call to the ``search`` shim, so its rule judge passes.
143+
"""
144+
145+
def __init__(self, backend: ClaudeCliBackend, *, timeout: bool = False) -> None:
146+
self.backend = backend
147+
self.timeout = timeout
148+
self.calls: list[tuple[str, str]] = []
149+
150+
def __call__(self, cmd, **kwargs):
151+
if not cmd or cmd[0] != self.backend.claude_path:
152+
return _REAL_RUN(cmd, **kwargs)
153+
route = "tools" if "--allowedTools" in cmd else "text"
154+
phase = getattr(self.backend, "evidence_phase", "")
155+
self.calls.append((phase, route))
156+
if self.timeout:
157+
raise subprocess.TimeoutExpired(cmd, kwargs.get("timeout"))
158+
prompt = kwargs.get("input") or ""
159+
if route == "text" and phase == "train":
160+
reply = json.dumps([{"op": "add", "content": _SEARCH_RULE}])
161+
else:
162+
reply = "Here is the summary."
163+
if route == "tools" and _SEARCH_RULE in prompt:
164+
with open(os.path.join(kwargs["cwd"], "_tool_calls.log"), "a") as log:
165+
log.write("search\n")
166+
return subprocess.CompletedProcess(cmd, 0, stdout=reply, stderr="")
167+
168+
def phases(self) -> list[str]:
169+
return [phase for phase, _route in self.calls]
170+
171+
172+
def _synthetic_claude(timeout: bool = False):
173+
backend = ClaudeCliBackend(claude_path="synthetic-claude-for-tests")
174+
process = SyntheticClaudeProcess(backend, timeout=timeout)
175+
return backend, process
176+
177+
115178
class TokenBudgetCycleTest(unittest.TestCase):
116179
def setUp(self) -> None:
117180
self._proj = tempfile.TemporaryDirectory()
@@ -176,7 +239,10 @@ def test_night_stops_scheduling_work_once_the_budget_is_spent(self):
176239
events = read_events(os.path.join(outcome.staging_dir, "evidence.jsonl"))
177240
end = [e for e in events if (e["stage"], e["event"]) == ("cycle", "end")]
178241
self.assertEqual(end[-1]["max_tokens_per_night"], limit)
179-
self.assertEqual(end[-1]["budget_skipped"], ["skill edits", "memory edits"])
242+
self.assertEqual(
243+
end[-1]["budget_skipped"],
244+
["skill edits", "memory edits", "final validation replay"],
245+
)
180246

181247
def test_one_token_of_headroom_lets_the_next_step_run(self):
182248
tasks = _tasks()
@@ -463,5 +529,132 @@ def test_groups_not_started_are_skipped_once_the_budget_is_spent(self):
463529
self.assertIsNone(second.result)
464530

465531

532+
class TokenBudgetProviderCallTest(unittest.TestCase):
533+
"""Count the provider processes a budget-stopped night really starts.
534+
535+
These drive ``ClaudeCliBackend`` through its own replay routes, with
536+
``subprocess.run`` answered by ``SyntheticClaudeProcess``. Replies the
537+
response cache does not serve (tool replay, a timed out call) are visible
538+
as new processes, which a caching mock cannot show.
539+
"""
540+
541+
def setUp(self) -> None:
542+
self._proj = tempfile.TemporaryDirectory()
543+
self._home = tempfile.TemporaryDirectory()
544+
self.addCleanup(self._proj.cleanup)
545+
self.addCleanup(self._home.cleanup)
546+
patcher = mock.patch.dict(os.environ, {"SKILLOPT_SLEEP_WORKERS": "1"})
547+
patcher.start()
548+
self.addCleanup(patcher.stop)
549+
550+
def _night(self, tasks, *, timeout: bool):
551+
cfg = load_config(
552+
invoked_project=self._proj.name,
553+
projects="invoked",
554+
backend="claude",
555+
claude_home=os.path.join(self._home.name, ".claude"),
556+
state_dir=os.path.join(self._home.name, "state"),
557+
auto_adopt=False,
558+
)
559+
# Any first call spends this, so only the initial replay may run.
560+
cfg.data["max_tokens_per_night"] = 1
561+
backend, process = _synthetic_claude(timeout)
562+
with mock.patch("skillopt_sleep.backend.subprocess.run", side_effect=process):
563+
outcome = run_sleep_cycle(cfg, seed_tasks=tasks, backend=backend)
564+
return process, outcome
565+
566+
def _assert_only_the_initial_replay_ran(self, tasks, process, outcome) -> None:
567+
n_val = sum(1 for t in tasks if t.split == "val")
568+
n_train = sum(1 for t in tasks if t.split == "train")
569+
self.assertGreater(n_val, 0)
570+
phases = process.phases()
571+
self.assertEqual(phases.count("final_val"), 0)
572+
self.assertEqual(
573+
phases, ["baseline_val"] * n_val + ["train"] * n_train
574+
)
575+
report = outcome.report
576+
self.assertFalse(report.accepted)
577+
notes = _budget_notes(report)
578+
self.assertEqual(len(notes), 1)
579+
self.assertIn(
580+
"skipped: skill edits, memory edits, final validation replay", notes[0]
581+
)
582+
events = read_events(os.path.join(outcome.staging_dir, "evidence.jsonl"))
583+
self.assertFalse([
584+
e for e in events
585+
if (e["stage"], e["event"]) == ("replay", "result")
586+
and e.get("phase") == "final_val"
587+
])
588+
skips = [e["step"] for e in events if (e["stage"], e["event"]) == ("budget", "skip")]
589+
self.assertIn("final validation replay", skips)
590+
decision = [e for e in events if (e["stage"], e["event"]) == ("gate", "decision")]
591+
self.assertEqual(len(decision), 1)
592+
self.assertEqual(
593+
decision[0]["candidate_score"], decision[0]["baseline_score"]
594+
)
595+
596+
def test_tool_route_answers_make_no_final_validation_call(self):
597+
tasks = _tool_tasks()
598+
process, outcome = self._night(tasks, timeout=False)
599+
self.assertEqual({route for _phase, route in process.calls}, {"tools"})
600+
self._assert_only_the_initial_replay_ran(tasks, process, outcome)
601+
602+
def test_tool_route_timeouts_make_no_final_validation_call(self):
603+
tasks = _tool_tasks()
604+
process, outcome = self._night(tasks, timeout=True)
605+
self.assertEqual({route for _phase, route in process.calls}, {"tools"})
606+
self._assert_only_the_initial_replay_ran(tasks, process, outcome)
607+
608+
def test_text_route_timeouts_make_no_final_validation_call(self):
609+
# An empty reply from a failed call is not kept in the response cache,
610+
# so re-scoring the same documents would start the process again.
611+
tasks = _arithmetic_tasks()
612+
process, outcome = self._night(tasks, timeout=True)
613+
self.assertEqual({route for _phase, route in process.calls}, {"text"})
614+
self._assert_only_the_initial_replay_ran(tasks, process, outcome)
615+
616+
def test_text_route_answers_stay_free_cache_hits(self):
617+
# Control: here the final replay would have been served from cache.
618+
tasks = _arithmetic_tasks()
619+
process, outcome = self._night(tasks, timeout=False)
620+
self._assert_only_the_initial_replay_ran(tasks, process, outcome)
621+
622+
623+
class TokenBudgetFinalScoringTest(unittest.TestCase):
624+
"""With the budget spent, final scoring reuses the replay of the kept edit."""
625+
626+
def test_kept_skill_edit_is_scored_from_its_own_validation_replay(self):
627+
tasks = _tool_tasks()
628+
n_val = sum(1 for t in tasks if t.split == "val")
629+
n_train = sum(1 for t in tasks if t.split == "train")
630+
backend, process = _synthetic_claude()
631+
632+
def spent() -> bool:
633+
# The budget runs out once the skill step's reflect call started.
634+
return ("train", "text") in process.calls
635+
636+
with mock.patch.dict(os.environ, {"SKILLOPT_SLEEP_WORKERS": "1"}), \
637+
mock.patch("skillopt_sleep.backend.subprocess.run", side_effect=process):
638+
result = consolidate(
639+
backend, tasks, "# skill\n", "", budget_exhausted=spent
640+
)
641+
642+
self.assertEqual(
643+
process.calls,
644+
[("baseline_val", "tools")] * n_val
645+
+ [("train", "tools")] * n_train
646+
+ [("train", "text")]
647+
+ [("gate_trial:skill", "tools")] * n_val,
648+
)
649+
self.assertEqual(result.budget_skipped, ["memory edits", "final validation replay"])
650+
self.assertTrue(result.accepted)
651+
self.assertIn(_SEARCH_RULE, result.new_skill)
652+
self.assertEqual(result.candidate_score, 1.0)
653+
trial, final = result.gate_trials
654+
self.assertEqual(trial["target"], "skill")
655+
self.assertEqual(final["target"], "final")
656+
self.assertEqual(final["candidate_score"], trial["candidate_score"])
657+
658+
466659
if __name__ == "__main__":
467660
unittest.main()

0 commit comments

Comments
 (0)