Closes lotti3-anb.5 (live entry point), lotti3-anb.6 (one eval path), lotti3-0qq, and carries everything lotti3-anb.7 — actually running it — exposed.

The entry point

eval/day_planning_eval_live_test.dart supplies only the inference layer and the output paths; everything below the model is the production pipeline.

set -a; source .env; set +a   # MELIOUS_API_KEY / MELIOUS_BASE_URL
LOTTI_DAY_PLANNING_EVAL_LIVE=1 DAY_PLANNING_EVAL_MODELS=glm-5.2 \
  fvm flutter test test/features/daily_os_next/eval/day_planning_eval_live_test.dart

It always passes when it runs. Violations are reported, never asserted: a live run is non-deterministic and costs money, and a red build people learn to ignore is worse than no signal. Missing credentials are the one hard failure — a setup error, not something a model did.

day_agent_draft_live_eval_test.dart is deleted, so there is one live path. Its subject — a same-day draft where the model's proposed times meet the past-start guard — is the lateStart scenario, which additionally seeds real work, so the run says something about planning rather than only about the guard.

Four runs against glm-5.2, and what each one fixed

Every finding in the first three was about the eval, not the model.

That climb is not the model improving. It is instrument repair.

A drafting wake offers the whole capture/reconcile tool set, not just draft_day_plan, and glm-5.2 used it — parse_capture_to_items, surface_pending_decisions, summarize_recent_patterns, create_task_from_phrase, apply_triage — before drafting. Each landed on a mocked collaborator. An unstubbed mocktail method returns a bare null where a Future is expected, so the call came back as type 'Null' is not a subtype of type 'Future<Task?>': a harness defect delivered as a rejection, which is exactly the signal compliedWithoutRejection scores. Later rounds were subtler, same shape — "task not found" for a task the scenario itself seeded, "failed to create task", "failed to update task", all because stubs returned null/false where the app succeeds.

The directive scorer refereed free text with substring matching, and was wrong twice. Requiring the reason enum to be directiveUnsatisfiable scored an escalating model SILENTLY DROPPED. Requiring the note to contain each commitment's title did the same to:

"Interviews and 1:1s cannot fit — user must defer one or both, or extend capacity."

which names both casualties in the words a person would use. Escalation is directive-level in the prompt — "escalate via raise_day_status", not "name each commitment" — so it now answers for everything unplaced, gated on a reason that could honestly describe this day failing, drawn from the fixture allowlist surfacedConflict already uses. processingBlocked still answers for nothing.

A false "silently dropped" is worse than a coarse pass: it accuses the model of the one thing it visibly did not do. This is lotti3-anb.11's thesis confirmed with evidence rather than speculation.

A task the model created and then scheduled read as fabricated (lotti3-0qq, fixed here rather than deferred). The id is recoverable from real persisted state — create_task_from_phrase stamps it onto the parsed item as matchedTaskId — so no production change, and it works identically live. This closes a latent gap the artefact only happened to expose: the same flow exists in the app, so any real plan where the model created a task and scheduled it would have been reported as invention. It cannot launder an invented id, because the capture service nulls matchedTaskId unless it resolves to a real, open, category-allowed Task.

What run 4 says about the model

All six remaining failures are genuine, and two matter:

bindingDirective — a block titled "Already scheduled" at 09:00, type cal, state committed, materialising the directive's alreadyScheduledMinutes: 60. Reproduced in runs 3 and 4, so it is habitual, not a fluke. That single block walks through both filed production holes at once: lotti3-3r1 (the prompt offers a cal exemption while the model is shown no calendar) and lotti3-abp (a model-writable state bypasses the same-day guard, which fires only on drafted). noFabricatedCalendarBlocks and noHistoryFabrication both caught it. Neither hole is hypothetical any more.

blockedWithoutCorpus — 50% on blockerBeforeBlocked, and task-a-root ignored, against a clean pass for its twin blockedChain. That is the pair doing exactly the job it was built for: same ground truth, one variable — whether the corpus was rendered — and a measurable gap.

Also stable: respectsEstimates 83% (a 180-minute task allocated 60 minutes), requiredWorkPlaced 75%, and ~95% of input tokens served from cache, consistent with the 100%-stable system prompt.

Verification

• fvm dart analyze clean; 195 eval-framework tests; the full daily_os_next suite passes with the harness changes.
• Four live runs, each read end to end through the judge bundle — which is how every one of the instrument defects above was found.

Summary by CodeRabbit

• New Features
  • Added opt-in live “day-planning” evaluation matrix with configurable models, samples, and scenarios, generating reports and artifacts.
  • Evaluation outputs now include task IDs created during a run.
• Bug Fixes
  • Improved fabricated-task detection using created task tracking.
  • Refined directive escalation scoring and more accurate journal-based task discovery.
• Documentation
  • Expanded evaluation docs with a “Running it against real models” section and updated fake-time guidance.
• Tests
  • Added/updated eval fixtures and harness helpers to better simulate journal and task lifecycles, including new live test coverage wiring.