Repository navigation
fix(observer): recognise the CLI's signed-out wording as an auth failure - #3786
Conversation
Every pattern in isAuthFailureObserverOutput keys on the word 'authentication'. The CLI does not use it when the profile is signed out - it prints 'Not logged in - Please run /login' - so that response missed the auth branch, fell through to the prose fallback, and was confirmed. Confirming a claimed batch discards it: a reported fleet lost 857 batches over three hours with no observation written and nothing to retry. Two anchored patterns. The bare state must open the response, because an observer narrating a past outage uses the same words, and the remediation half must END it, so 'Please run /login in the observed project instructions' - an existing must-not-match case - keeps running on into its sentence and stays prose. Refs thedotmack#3606. Four matcher cases and one end-to-end case go red against main.
Greptile SummaryThis change recognizes the CLI signed-out response and preserves queued observer work when authentication must be restored. The previously reported leading Confidence Score: 4/5Not safe to merge until completed observer prose containing a trailing separator and One verified reliability failure remains in the authentication-output matcher: it can reset a batch that completed successfully. Files Needing Attention: src/sdk/output-classifier.ts
What T-Rex did
|
…sentence An anchor is not a report test. A completed observation can open with the same words - 'Not logged in during the reboot window; the observer recorded no new findings' - and classifying it as an auth failure resets a batch that already succeeded and pauses the generator. The state now has to end the response, or introduce the rest with a separator, which is what the CLI's own 'Not logged in - Please run /login' does. A trailing full stop counts only when it ends there, so 'Not logged in. The observer recorded...' stays prose. Three narrative cases go red against the previous commit.
|
Correct, and this is the fourth time I have made this exact mistake — including in the PR next door, where I wrote the rule that would have prevented it. Fixed in
/^not logged in\b\s*(?:[·|:\-–—]|[.!]?\s*$)/The state must be the response — it ends there, or it introduces the rest with a separator, which is exactly what the CLI's own The trailing full stop is allowed only when it ends the response, so Why I did not lean toward over-detectionIn the transport classifier I argued that a miss is the safe direction. That argument does not transfer here, and it is worth saying so rather than reusing it: on this path a miss confirms the claim and destroys the batch, while a false positive resets completed work and pauses the generator. Both directions cost real work, so the rule has to be precise rather than conservative — which is why the fix is a shape test and not a longer keyword list. VerificationYour case and two siblings, red against the previous commit (source file stashed, tests untouched): After: 32 pass, 0 fail. The must-match side gained |
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
A separator after the status is not a report test either: 'Not logged in - the observer recorded nothing new' opens with the status, follows it with a separator, and is a completed observation. A leading anchor answers prose that runs on; it does not answer prose that runs on after punctuation. Anchoring both ends closes it instead of narrowing it again. The CLI emits two shapes and no third - the status alone, or the status plus its own /login remediation - so both are matched exactly and nothing can trail them. Three narrative cases go red against the previous commit.
|
Right again, and I was still narrowing the rule instead of closing it. Fixed in Last round I required the status to end the response or introduce the rest with a separator. That answers prose which runs straight on — and not prose which runs on after punctuation, which is your case: Closed rather than narrowedThe CLI emits two shapes and no third: the status alone, or the status plus its own /^not logged in\b\s*[.!]?\s*$/
/^not logged in\b\s*[·|:\-–—]\s*(?:please\s+)?run\s+\/login\b\s*[.!]?\s*$/Nothing can trail either one, so there is no remaining shape for a sentence to occupy. That is the difference from the previous two attempts: I was extending a prefix test each time you found a way past it, which is a losing game against natural language. Matching the complete response ends it.
VerificationYour three cases, red against the previous commit with only the source file stashed: After: 36 pass, 0 fail. The must-match side gained Three rounds on one predicate is worth naming: each time I tested whether the text started like a status, when the question is whether the text is the status. Same error I made on #3759. Anchoring both ends is the form that does not have to be revisited. |
| // do rather than an observation quoting the instruction — the existing | ||
| // false cases ("Please run /login in the observed project instructions") | ||
| // run on into a sentence and stay unmatched. | ||
| /(?:^|[·|]\s*)please run \/login\b\s*[.!]?\s*$/.test(text) || |
There was a problem hiding this comment.
Trailing
/login remediation misclassifies completed prose
The separator branch is not anchored to the start of the response. Consequently, completed observer text such as The completed observation says · Please run /login or its pipe equivalent is treated as an authentication failure solely because it ends in the remediation phrase. ResponseProcessor resets the claimed batch and aborts processing rather than confirming the ordinary prose response, causing completed work to be retried and pausing generation. Restrict this branch to the complete CLI status/remediation format.
Artifacts
Focused Bun harness source for candidate and prior-root observer prose cases
- Captured the exact temporary test source that calls the real matcher and ResponseProcessor branch; it defines all four exercised inputs and expected side effects.
Initial focused Bun harness load attempt
- Captured the first command execution, which exposed a missing mocked `workerHttpRequest` export required before ResponseProcessor could load; no repository source was modified.
Focused Bun execution showing candidate reset and prior-root prose confirmation
- Ran the completed harness successfully: both candidate endings reset the batch while both prior-root narratives confirm it, proving the candidate and disproving both earlier claims.
|
All-night ship gate: PASS
Rebased onto |
|
Update: GitHub accepted the squash merge of this head onto current |
Refs #3606, from @Priscu's fleet report there: a reboot signed the default
~/.claudeprofile out, and over ~3 hours 857 batches were logged asreturned non-XML prose response — ignoring queued batch, confirmed, and lost. Zero observations written.The matcher looks for a word the CLI does not use
Every pattern in
isAuthFailureObserverOutputkeys on authentication:The CLI, when the profile is signed out, says
Not logged in · Please run /login. That phrase contains none of those. The last pattern nearly catches it, but it requires a qualifier after/loginand the CLI's line ends there.Measured against
mainbefore touching anything:Why a miss here loses data rather than degrading
ResponseProcessorhandles the auth case by preserving the work:Fall past it and the prose fallback runs
confirmClaimedMessages(session.sessionDbId)— the claim is acknowledged, the batch is gone, and there is nothing left to retry once the user logs back in. Which is exactly the reported outcome, and it matches the log line in the report verbatim.The fix
Two anchored patterns:
The anchoring is the load-bearing part, and it is aimed at two existing must-not-match cases in the suite:
Please run /login in the observed project instructions.— an observation quoting the instruction. The remediation pattern requires the phrase to end the response, so this runs on into its sentence and stays prose.not logged into open the response, because an observer narrating a past outage ("the session was not logged in during the reboot window") uses the same words, and preserving a batch that already succeeded pauses the generator and retries completed work — the mirror of the bug being fixed.Invalid API key · Please run /loginis caught as a side effect of the separator form rather than by enumerating prefixes, which is why I wrote it that way instead of listing every signed-out state the CLI might print.Verification
Five cases red against
main— matcher cases with the source file stashed:and one end-to-end case over
processAgentResponse, which is the assertion that actually describes the harm:It asserts
confirmClaimedMessageswas not called andabortReason === 'auth:observer_text', so a future change that classifies the prose correctly but still confirms the claim would not pass.After:
output-classifier27 pass, 0 fail;response-processor24 pass, 0 fail. Four narrative cases guard the other direction.Scope
Only the matcher. #3606's broader proposal — never confirm a batch on any unparsed prose — is the right end state and this does not pre-empt it; it closes the specific wording that is losing data today, in a form that is easy to verify against a real fleet. @Priscu offered to test a build, and this branch is the smallest thing worth testing.
I did not touch the existing patterns. They are unanchored, which is loose in the same way, but widening or tightening them is a separate change with its own blast radius.