Skip to content

fix(observer): recognise the CLI's signed-out wording as an auth failure - #3786

Merged
thedotmack merged 3 commits into
thedotmack:mainfrom
ntdatt812:fix/3606-auth-not-logged-in
Sep 11, 2026
Merged

thedotmack merged 3 commits into
thedotmack:mainfrom
ntdatt812:fix/3606-auth-not-logged-in

Conversation

@ntdatt812

Copy link
Copy Markdown
Contributor

Refs #3606, from @Priscu's fleet report there: a reboot signed the default ~/.claude profile out, and over ~3 hours 857 batches were logged as returned non-XML prose response — ignoring queued batch, confirmed, and lost. Zero observations written.

The matcher looks for a word the CLI does not use

Every pattern in isAuthFailureObserverOutput keys on authentication:

/\bfailed to authenticate\b/
/\bauthentication (?:failed|failure|error)\b/
/\b(?:authentication|auth)\b.{0,20}\b(?:required|expired|invalid|again)\b.{0,20}\/login\b/
…401/403 shapes…
/\/login\b.{0,40}\b(?:to\s+authenticate|again|to\s+continue|…)\b/

The CLI, when the profile is signed out, says Not logged in · Please run /login. That phrase contains none of those. The last pattern nearly catches it, but it requires a qualifier after /login and the CLI's line ends there.

Measured against main before touching anything:

false  "Not logged in · Please run /login"
false  "Not logged in"
false  "Invalid API key · Please run /login"
false  "Please run /login"
true   "Authentication failed. Please run /login to authenticate."

Why a miss here loses data rather than degrading

ResponseProcessor handles the auth case by preserving the work:

if (isAuthFailureObserverOutput(text)) {
  await sessionManager.resetProcessingToPending(session.sessionDbId);
  session.abortReason = 'auth:observer_text';
  …

Fall past it and the prose fallback runs confirmClaimedMessages(session.sessionDbId) — the claim is acknowledged, the batch is gone, and there is nothing left to retry once the user logs back in. Which is exactly the reported outcome, and it matches the log line in the report verbatim.

The fix

Two anchored patterns:

/^not logged in\b/
/(?:^|[·|]\s*)please run \/login\b\s*[.!]?\s*$/

The anchoring is the load-bearing part, and it is aimed at two existing must-not-match cases in the suite:

  • Please run /login in the observed project instructions. — an observation quoting the instruction. The remediation pattern requires the phrase to end the response, so this runs on into its sentence and stays prose.
  • The bare state pattern requires not logged in to open the response, because an observer narrating a past outage ("the session was not logged in during the reboot window") uses the same words, and preserving a batch that already succeeded pauses the generator and retries completed work — the mirror of the bug being fixed.

Invalid API key · Please run /login is caught as a side effect of the separator form rather than by enumerating prefixes, which is why I wrote it that way instead of listing every signed-out state the CLI might print.

Verification

Five cases red against main — matcher cases with the source file stashed:

(fail) preserves the batch for: Not logged in · Please run /login
(fail) preserves the batch for: Not logged in
(fail) preserves the batch for: Invalid API key · Please run /login
(fail) preserves the batch for: Please run /login
 23 pass, 4 fail

and one end-to-end case over processAgentResponse, which is the assertion that actually describes the harm:

(fail) ResponseProcessor > signed-out CLI prose preserves the batch (#3606)
       > resets the batch to pending instead of confirming it
       expect(resetProcessingToPending).toHaveBeenCalledWith(1)
 23 pass, 1 fail

It asserts confirmClaimedMessages was not called and abortReason === 'auth:observer_text', so a future change that classifies the prose correctly but still confirms the claim would not pass.

After: output-classifier 27 pass, 0 fail; response-processor 24 pass, 0 fail. Four narrative cases guard the other direction.

Scope

Only the matcher. #3606's broader proposal — never confirm a batch on any unparsed prose — is the right end state and this does not pre-empt it; it closes the specific wording that is losing data today, in a form that is easy to verify against a real fleet. @Priscu offered to test a build, and this branch is the smallest thing worth testing.

I did not touch the existing patterns. They are unanchored, which is loose in the same way, but widening or tightening them is a separate change with its own blast radius.

Every pattern in isAuthFailureObserverOutput keys on the word 'authentication'.
The CLI does not use it when the profile is signed out - it prints 'Not logged
in - Please run /login' - so that response missed the auth branch, fell through
to the prose fallback, and was confirmed. Confirming a claimed batch discards
it: a reported fleet lost 857 batches over three hours with no observation
written and nothing to retry.

Two anchored patterns. The bare state must open the response, because an
observer narrating a past outage uses the same words, and the remediation half
must END it, so 'Please run /login in the observed project instructions' - an
existing must-not-match case - keeps running on into its sentence and stays
prose.

Refs thedotmack#3606. Four matcher cases and one end-to-end case go red against main.
@greptile-apps

greptile-apps Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This change recognizes the CLI signed-out response and preserves queued observer work when authentication must be restored. The previously reported leading Not logged in and separator narrative examples now follow the ordinary prose-confirmation path. However, completed prose ending in · Please run /login or | Please run /login is still mistaken for an authentication failure, which resets completed work for retry and pauses generation.

Confidence Score: 4/5

Not safe to merge until completed observer prose containing a trailing separator and /login remediation is no longer treated as an authentication failure.

One verified reliability failure remains in the authentication-output matcher: it can reset a batch that completed successfully.

Files Needing Attention: src/sdk/output-classifier.ts

T-Rex T-Rex Logs

What T-Rex did

  • Validated the auth-observer-prose harness, including the harness source and two run captures, with the final run reporting four exact runtime outcomes and a successful exit.
  • Posted a P1 finding with a focused Bun harness, including an initial load attempt and an execution that confirmed candidate reset and prior-root prose.
  • Posted a second P1 finding proof via a finding-comment-proof, directing reviewers to the corresponding review comment.

View all artifacts

T-Rex Ran code and verified through T-Rex

Comments Outside Diff (1)

  1. General comment

    P1 Completed observer prose ending in separator + /login is treated as authentication failure

    • Bug
      • The completed observation says · Please run /login and The completed observation says | Please run /login both matched authentication failure and made ResponseProcessor call resetProcessingToPending without confirmClaimedMessages. This reproduces the supplied candidate exactly.
    • Cause
      • The matcher at src/sdk/output-classifier.ts:111 uses (?:^|[·|]\s*) before please run /login without constraining the separator to the beginning/status-line form, allowing arbitrary prose before · or |.
    • Fix
      • Restrict the remediation matcher to a complete CLI status/remediation response, or remove the separator-prefix alternative and rely on the already anchored signed-out status patterns at lines 104-105.

    T-Rex Ran code and verified through T-Rex

Reviews (3): Last reviewed commit: "fix(observer): anchor the signed-out sta..." | Re-trigger Greptile

Comment thread src/sdk/output-classifier.ts Resolved
…sentence

An anchor is not a report test. A completed observation can open with the same
words - 'Not logged in during the reboot window; the observer recorded no new
findings' - and classifying it as an auth failure resets a batch that already
succeeded and pauses the generator.

The state now has to end the response, or introduce the rest with a separator,
which is what the CLI's own 'Not logged in - Please run /login' does. A trailing
full stop counts only when it ends there, so 'Not logged in. The observer
recorded...' stays prose.

Three narrative cases go red against the previous commit.
@ntdatt812

Copy link
Copy Markdown
Contributor Author

Correct, and this is the fourth time I have made this exact mistake — including in the PR next door, where I wrote the rule that would have prevented it. Fixed in c0f48a9.

^ is an anchor, not a report test. I keep reaching for it as though starting with the token means the response is the failure, and it does not: prose starts with the token too.

/^not logged in\b\s*(?:[·|:\-–—]|[.!]?\s*$)/

The state must be the response — it ends there, or it introduces the rest with a separator, which is exactly what the CLI's own Not logged in · Please run /login does. A sentence runs straight on and is left alone.

The trailing full stop is allowed only when it ends the response, so Not logged in. matches while Not logged in. The observer recorded no new findings for this batch. does not. That case is in the suite, because a period is the obvious way to defeat a naive fix here.

Why I did not lean toward over-detection

In the transport classifier I argued that a miss is the safe direction. That argument does not transfer here, and it is worth saying so rather than reusing it: on this path a miss confirms the claim and destroys the batch, while a false positive resets completed work and pauses the generator. Both directions cost real work, so the rule has to be precise rather than conservative — which is why the fix is a shape test and not a longer keyword list.

Verification

Your case and two siblings, red against the previous commit (source file stashed, tests untouched):

(fail) leaves prose alone: Not logged in during the reboot window; the o…
(fail) leaves prose alone: Not logged in. The observer recorded no new f…
(fail) leaves prose alone: Not logged in sessions were reviewed and the …
 29 pass, 3 fail

After: 32 pass, 0 fail. The must-match side gained Not logged in. and Not logged in: run /login so the separator form is pinned in both directions, and the end-to-end processAgentResponse case still passes — response-processor 24 pass, 0 fail.

Comment thread src/sdk/output-classifier.ts Resolved
@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

A separator after the status is not a report test either: 'Not logged in - the
observer recorded nothing new' opens with the status, follows it with a
separator, and is a completed observation. A leading anchor answers prose that
runs on; it does not answer prose that runs on after punctuation.

Anchoring both ends closes it instead of narrowing it again. The CLI emits two
shapes and no third - the status alone, or the status plus its own /login
remediation - so both are matched exactly and nothing can trail them.

Three narrative cases go red against the previous commit.
@ntdatt812

Copy link
Copy Markdown
Contributor Author

Right again, and I was still narrowing the rule instead of closing it. Fixed in aef8019.

Last round I required the status to end the response or introduce the rest with a separator. That answers prose which runs straight on — and not prose which runs on after punctuation, which is your case:

Not logged in — the observer recorded nothing new.
Not logged in: sessions were reviewed and the retry path documented.
Not logged in · the reboot window is documented in the runbook.

Closed rather than narrowed

The CLI emits two shapes and no third: the status alone, or the status plus its own /login remediation. Both are now matched whole, anchored at both ends:

/^not logged in\b\s*[.!]?\s*$/
/^not logged in\b\s*[·|:\-–—]\s*(?:please\s+)?run\s+\/login\b\s*[.!]?\s*$/

Nothing can trail either one, so there is no remaining shape for a sentence to occupy. That is the difference from the previous two attempts: I was extending a prefix test each time you found a way past it, which is a losing game against natural language. Matching the complete response ends it.

Invalid API key · Please run /login is unaffected — it is caught by the separate remediation pattern, which already required the phrase to end the response.

Verification

Your three cases, red against the previous commit with only the source file stashed:

(fail) leaves prose alone: Not logged in — the observer recorded nothing…
(fail) leaves prose alone: Not logged in: sessions were reviewed and the…
(fail) leaves prose alone: Not logged in · the reboot window is document…
 33 pass, 3 fail

After: 36 pass, 0 fail. The must-match side gained Not logged in — Please run /login so the em-dash separator is pinned, and the end-to-end processAgentResponse case still passes — response-processor 24 pass, 0 fail.

Three rounds on one predicate is worth naming: each time I tested whether the text started like a status, when the question is whether the text is the status. Same error I made on #3759. Anchoring both ends is the form that does not have to be revisited.

// do rather than an observation quoting the instruction — the existing
// false cases ("Please run /login in the observed project instructions")
// run on into a sentence and stay unmatched.
/(?:^|[·|]\s*)please run \/login\b\s*[.!]?\s*$/.test(text) ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Trailing /login remediation misclassifies completed prose

The separator branch is not anchored to the start of the response. Consequently, completed observer text such as The completed observation says · Please run /login or its pipe equivalent is treated as an authentication failure solely because it ends in the remediation phrase. ResponseProcessor resets the claimed batch and aborts processing rather than confirming the ordinary prose response, causing completed work to be retried and pausing generation. Restrict this branch to the complete CLI status/remediation format.

Artifacts

Focused Bun harness source for candidate and prior-root observer prose cases

  • Captured the exact temporary test source that calls the real matcher and ResponseProcessor branch; it defines all four exercised inputs and expected side effects.

Initial focused Bun harness load attempt

  • Captured the first command execution, which exposed a missing mocked `workerHttpRequest` export required before ResponseProcessor could load; no repository source was modified.

Focused Bun execution showing candidate reset and prior-root prose confirmation

  • Ran the completed harness successfully: both candidate endings reset the batch while both prior-root narratives confirm it, proving the candidate and disproving both earlier claims.

View artifacts

T-Rex Ran code and verified through T-Rex

@thedotmack

Copy link
Copy Markdown
Owner

All-night ship gate: PASS

  1. RCA logged — CLI signed-out wording Not logged in · Please run /login matched none of the authentication-keyed patterns, so 857 batches were confirmed and dropped ([plan-18] Observer Response Pipeline — classify every output, never confirm a batch on failure, bound every history #3606).
  2. No second-system — two whole-response matcher patterns only; existing patterns untouched.
  3. Narrow / not opinionated — matcher + classifier/processor tests. False-positive prose cases are pinned.

Rebased onto 7335467f locally (fork branch update blocked). Focused tests: bun test tests/sdk/output-classifier.test.ts tests/worker/agents/response-processor.test.ts — 73 pass. This head has no CI runs; opening a rebased ship branch to verify CI before merge. No version bump.

@thedotmack
thedotmack merged commit e81dd2a into thedotmack:main Sep 11, 2026
@thedotmack

Copy link
Copy Markdown
Owner

Update: GitHub accepted the squash merge of this head onto current main after the local rebase onto 7335467f stayed clean and the 73 focused tests passed. No separate ship branch needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants