Skip to content

fix(owner-goals): drop unverifiable quotes from a fresh save instead of refusing it whole - #5992

Merged
thedotmack merged 3 commits into
thedotmack:mainfrom
0xABCD01:fix/5969-goals-step
Nov 11, 2026
Merged

thedotmack merged 3 commits into
thedotmack:mainfrom
0xABCD01:fix/5969-goals-step

Conversation

@0xABCD01

Copy link
Copy Markdown
Contributor

Fixes #5969.

On a first My Work run with CLAUDE_MEM_WORK_VIEW_AGENT=api, the goals step maps the whole window in one answer, and save_goals.py then refused the whole answer when any one of its 60-90 evidence quotes was not a verbatim substring of the prompt at that minute. The reporter measured a 2-18% per-quote miss rate that grows with window size, so a large first run never saved a snapshot, and every later refresh took the same full-window path and failed again.

Change: on a fresh save — the refresh passes no --carried-before when there is no saved snapshot to carry quotes from — save_goals.py now drops a quote that fails verification, and drops an item left with no quote, instead of refusing every verified quote beside it. The dropped quotes are reported in the save result as dropped_unverified_quotes (item and minute, the same shape as dropped_carried_quotes). Every saved quote still verifies verbatim, and every saved item still holds at least one verified quote. If every item is dropped, the save is still refused with the failing quotes listed, so a wholly unverified answer cannot become an empty snapshot. Updates saved with --carried-before are unchanged: a carried quote that no longer matches is dropped, and a newly written quote that fails still refuses the save. The skill's step-3 text is updated to match.

Notes:

  • This deliberately does not cap or batch the first run's request. The reporter's model accepted the ~440k-token request and answered; what failed was the validator's zero tolerance. A cap that maps only the most recent slice would also lose the older prompts for good: the incremental path starts from the snapshot's generated_at, so those weeks would never be mapped. Chronological batching is a larger change and is left out rather than guessed at here.
  • The issue thread suggested dropping bad quotes with a refusal threshold of about 10% failures. This change has no percentage threshold: the reporter's own full-window run failed 16 of 90 quotes (about 18%), so a 10% line would still refuse the run the issue reports. The boundary used instead is evidence: an item survives only on quotes that verify, and an answer with no verified item left is refused.
  • The saved snapshot can therefore be partial: an item whose quotes all fail is omitted, and the dropped list is the record of what was left out.
  • Two of the reporter's failed refreshes were answers that were not valid JSON (an unescaped quote, a raw tab). Those still fail; this change covers only quote verification.

Verified without an API key at the validator level: a synthetic first-run answer with the reporter's full-window numbers — 90 quotes, 74 verbatim and 16 unverifiable, one item supported only by unverifiable quotes — was refused whole by the base script (exit 1, all 16 listed) and is saved by the fixed script with the 74 verified quotes on 29 items, the unsupported item dropped, and 16 drops reported. Tests: the new regression case in tests/work-view/skill-scripts-python.test.ts covers all three item kinds plus the all-unverified refusal; tests/work-view/ passes (280 tests), the agent-runner extraction and shared work-view writer suites pass (12 tests), and the root TypeScript check is clean.

…of refusing it whole

A first goals map's answer carries 60-90 verbatim quotes, and verbatim quoting gets less accurate as the window grows, so one inexact quote refused every verified quote beside it and no snapshot was ever saved (thedotmack#5969). Without --carried-before, save_goals.py now drops a quote that fails verification and any item left with none, reports the drops as dropped_unverified_quotes, and still refuses when no item verifies. Updates saved with --carried-before keep the strict refusal for new quotes.
@greptile-apps

greptile-apps Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues remain.

Summary

Fresh saves now drop quotes that fail verification and omit items left without evidence.

  • Fresh goal saves keep supported items when some quotes fail.

Reviews (3) · Last reviewed commit: "docs(owner-goals): tell a manual update ..." · Reviewed by Greptile

Comment thread plugin/skills/owner-goals/SKILL.md Resolved
thedotmack and others added 2 commits November 11, 2026 09:30
Without it, an update run by hand or by an agent takes the fresh-save
path and drops a bad new quote instead of refusing it. A test pins that
an update with --carried-before still refuses one bad new quote beside
a good one.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8c2BSHbdhJhnWYMGaY
@thedotmack

Copy link
Copy Markdown
Owner

Pushed a small follow-up on your branch, for the Greptile note on SKILL.md:

  • Merged main (clean).
  • "Or update the saved goals" now says to save the update with --carried-before, the same minute passed to --since. Without it, an update run by hand or by an agent would take your new fresh-save path and drop a bad new quote instead of refusing it. The automatic refresh already passes the flag.
  • Added one case to the carried-quote test: an update with one good and one made-up new quote still exits non-zero, so the strict path for updates is pinned.

@thedotmack
thedotmack merged commit 956a224 into thedotmack:main Nov 11, 2026
10 checks passed
@thedotmack

Copy link
Copy Markdown
Owner

Merged, thank you @0xABCD01! On a first goals map, save_goals.py now drops a quote that isn't the owner's words at that minute, and any item left with no quote, instead of refusing the whole answer. It lists what it dropped and still refuses when nothing checks out. So a large first run can save its snapshot, and later refreshes go back to the small incremental path instead of failing on the whole window again. Your reply in #5969 on why a 10% ceiling would still have refused the reporter's run was a good call. I added one line to the skill so manual updates keep passing --carried-before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

My Work: goals step with the api agent never succeeds on a first run with ~6k prompts (whole window in one request, all-or-nothing quote check)

2 participants