fix(server): limited threads auto-resume when allowance returns early - #24511

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This PR changes the server’s existing auto-resume workflow so provider usage readings can launch continuations before the scheduled reset, with new state to control subsequent retries. That is a meaningful production behavior and work-scheduling change despite the focused scope and backward-compatible contract fields.

^{You can add or adjust custom eligibility rules. Learn more.}

No actionable comments were generated in the recent review.

Merge Risk: Minimal · up to 78d8f

Early usage-limit recovery now waits for the reset only when the same run has already hit the limit again. A run on a different provider no longer inherits the old wait, so fresh usage readings can resume it early. No concrete merge-blocking risk remains in the reviewed changes.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title "fix(server): limited threads auto-resume when allowance returns early" matches the main change. The PR lets armed usage-limit recovery continue before the saved reset time when a fresh usag…
Linked Issues check		[#22307] Requirement: an armed usage-limit continuation becomes eligible before its saved reset once fresh provider usage shows the blocking limits cleared. Existing guards must stay in place. The re…
Out of Scope Changes check		Each changed area connects to early usage-limit recovery or to its guards: - Contracts (orchestrationV2.ts): optional waitForReset on the limit recovery, and optional usageLimitClearedAt on message.di…
Description check		The description follows the template's sections (Problem, Change, Scope, Verification) and explains the problem, the change, and the linked issue #22307. It gives detailed verification: targeted test …

Full details: Title check

Explanation

The title "fix(server): limited threads auto-resume when allowance returns early" matches the main change. The PR lets armed usage-limit recovery continue before the saved reset time when a fresh usage snapshot shows the limits cleared. The title is specific, uses a conventional commit prefix, and names the primary behavior. It does not mention the supporting candidate-query, contract, or UI changes, which is expected.

Full details: Linked Issues check

Explanation

[#22307] Requirement: an armed usage-limit continuation becomes eligible before its saved reset once fresh provider usage shows the blocking limits cleared. Existing guards must stay in place.

The reviewed head (78d8f97) implements this:

• Candidate query (ProjectionStore.ts): returns armed threads before their reset, only when the latest run has a usage_limit failure. Adds providerInstanceId so the sweep can match the provider snapshot.
• Sweep (UsageLimitRecoveryWorker.ts): loads providers only when candidates exist. Passes the matching provider's usageLimits to limitRecoveryCommand.
• limitRecoveryCommand: builds an early message.dispatch only when the reading is available, has at least one window, every window is below 100%, the reading is after the failed run ended, and recovery.waitForReset is not set. The command carries usageLimitClearedAt and uses a distinct ':early' delivery identity. A missing reading, a reading at 100%, or no windows keeps the reset-time path.
• Orchestrator.ts: accepts the early path only when usageLimitClearedAt is after run.completedAt ?? run.requestedAt. Rejects it when waitForReset is set. Cancel, stop, archive, settle, pending-request, and recovery request id checks are unchanged.
• A run that fails again before the prior reset gets waitForReset on its new recovery, so each limit episode allows one early resume.
• ProjectionStore shellErrorSummary: keeps the saved reset as usageLimitResetAt when a waitForReset recovery matches the failed run. This addresses the reported shell case where a retry returns no reset.

No unmet coding requirement found in the code.

Residual items, not treated as failures:

• PR comments report that a different-provider run may inherit a waitForReset state. In the reviewed source, waitForReset applies only when recovery.runId equals the failed run id (limitRecoveryWaitsForReset; candidate query). I did not find a path where a run from another provider shares that id. The report was not reproduced.
• Real provider usage readings were not verified. A dev-environment comment reports windows=[] for Codex. Early resume depends on readings that contain windows, which matches the issue condition 'once T3 can confirm'.
• The sweep itself has no direct test. The runtimeLayer test calls limitRecoveryCommand. The issue does not require sweep tests.

Full details: Out of Scope Changes check

Explanation

Each changed area connects to early usage-limit recovery or to its guards:

• Contracts (orchestrationV2.ts): optional waitForReset on the limit recovery, and optional usageLimitClearedAt on message.dispatch. Both are optional, so older clients remain compatible.
• ProjectionStore.ts and its test: candidate selection for armed recoveries before reset, limited to usage_limit failures. providerInstanceId supports the snapshot lookup. shellErrorSummary keeps the reset time visible for a waitForReset recovery, so the cancel control stays usable.
• UsageLimitRecoveryWorker.ts: early eligibility, distinct early delivery identity, provider lookup, and the re-arm of autoResume for a waitForReset recovery.
• Orchestrator.ts: acceptance checks for usageLimitClearedAt, waitForReset handling on repeat failures and stop-time updates, and request-id rotation when a pending runtime request or snooze blocks an accepted recovery dispatch. These serve the early-resume and repeat-limit guards.
• runtimeLayer.test.ts and ProjectionStore.test.ts: tests for the new behavior.
• docs/user/thread-sidebar.md: a three-line note on the early continuation and the repeat-limit wait.

No settings, UI component, or unrelated feature change appears in the reviewed diffs. No concrete unrelated change was found.

Full details: Description check

Explanation

The description follows the template's sections (Problem, Change, Scope, Verification) and explains the problem, the change, and the linked issue #22307. It gives detailed verification: targeted test failures and passes, merge notes, and signed-in runs with proxy timelines. It also lists what was not checked. The description is long and mixes in merge-history notes and a takeover section. The template's title line is missing from the body, and the Verification section is heavy on test output. The 'Problem' section does not state the expected behavior in one place, and the author does not state the agent model in the required trailing line, though the footer names models. The description is mostly complete, so it passes. Minor cleanup would help: move the takeover notes out of the body, and keep the Scope section focused on the bug fix.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

• Source mode: Auto
• Repository: pingdotgg/t3code
• Branch: main
• File: .coderabbit.config.ts
• Commit: 476590d
• Layers: Repository TS config
• Review profile: CHILL
• Plan: Advanced
• Run ID: dd17a828-6f5c-4e35-8fe8-74b79e927d19

Commits

Reviewing files that changed from the base of the PR and between 6738b5a and 78d8f97.

Files selected for processing: 6

• apps/server/src/orchestration-v2/Orchestrator.ts
• apps/server/src/orchestration-v2/ProjectionStore.test.ts
• apps/server/src/orchestration-v2/ProjectionStore.ts
• apps/server/src/orchestration-v2/runtimeLayer.test.ts
• docs/user/thread-sidebar.md
• packages/contracts/src/orchestrationV2.ts

Files skipped from review as they are similar to previous changes: 1

• docs/user/thread-sidebar.md

Limit details: You’ve used all 10 included reviews currently available.

thanks @only21mil. the fresh provider reading, bounded early attempt, and separate reset delivery preserve the original intent. i kept your commits and merged current main into this writable head.

added coverage through the actual recovery sweep and fixed explicit "resume at reset" recovery surviving a failed early continuation, including a changed reset with global auto-resume off. cancellation, stop, provider/run/request identity, pending-request guards, and snooze behavior remain enforced.

verification at a1469144b9d4d1bb952979ce100ef959a075fd55: all 27 ci checks green; blacksmith passed 186 focused tests and vp run check:changed 3f9448747bb8848050dc2cacf83affa6f94a66a1 (format, lint, knip, affected typechecks, and 163 tests). regressions failed before their fixes. two independent follow-up reviewers approved this head without findings.

browser evidence uses the real client/server path with a scripted codex peer and matching thread data: main remained limited after allowance restoration; this head delivered and completed the continuation before reset. real exhausted-subscription recovery, native desktop/mobile, and relay/tunnel remain unverified. provider readings can omit a limiting window, so one speculative attempt remains intentional.

known low-priority limit from coderabbit: a pending request appearing after candidate selection can consume the stable early command receipt, so that recovery waits until reset rather than attempting early again. the independent reset delivery remains available. deferred under the requested one-round-plus-follow-up review budget rather than changing shared receipt semantics. coderabbit's latest-head review was rate-limited; its green check is not a fresh review. maria's approval of the scheduling behavior is still required.

addressed the pending-request race reported in the out-of-diff review. a matching recovery now rotates its request identity when a pending runtime request blocks dispatch, as it already did for snooze. resolving the request lets the next sweep deliver the continuation, while successful delivery remains deduplicated. shared receipt semantics are unchanged.

merged current github/main with a merge commit. the guide preserves main's settings location and this pr's early-resume guidance.

verification for 4b8f39860a2fd4d3045e8c44c4d6cecce672f7db: the added pending-request race in the existing runtime test failed before the fix (one run instead of two after resolution). blacksmith passed all 138 runtime/projection tests and vp run check:changed becf853defb6127dbe4447860383777d69346b6f: format, lint, both knip passes, graphql, all 31 affected typechecks, and 163 tests. no new test files or UI changes. live exhausted-provider recovery and native/remote client paths were not re-exercised in this pass; prior evidence remains in the pr.

second fix round pushed as 6738b5a69b1d8df839fbceb61a9b82223f8601cd: a matching early retry that reports usage_limit without a new reset keeps the saved reset fallback. explicit replacement resets take precedence. blacksmith passed 139 focused tests and check:changed (format, lint, knip, graphql, 31 affected typechecks, 164 tests).

one final reviewer approved this head; the other found a real remaining issue: the saved reset is used by worker candidates but is not exposed in the client shell. after a retry with resetAt:null, recovery remains armed while web/desktop and mobile hide the cancel-auto-resume control and show manual-retry guidance. the shell must expose the matching saved reset before this is ready to merge. confirmed against ProjectionStore.ts shell builders, ChatView.tsx, and mobile UsageLimitRecoveryCard.tsx.

stopping code changes at the requested two-fix-round limit. this head is not ready to merge despite passing scoped checks. real exhausted-provider recovery and native/relay paths remain unverified this pass. no labels changed, no force push, rebase, or merge performed.

final CI: all 27 checks green at 6738b5a69b1d8df839fbceb61a9b82223f8601cd, no conflicts.

coderabbit's latest different-provider finding is valid: limitRecoveryWaitsForReset uses the old reset timestamp for a different run, while provider changes preserve the old recovery. this can incorrectly suppress early recovery on the new provider. deferred at the two-fix-round cap, alongside the saved-reset shell/cancel-control issue documented above. the suggested date-parsing cleanup is a nit and is deferred to keep scope focused.

this is still not ready to merge. one independent final reviewer approved; the other requires the shell/cancel-control fix. no further code changes made after the second fix round.

Verified in a real dev environment

partial verification at 6738b5a69b1d8df839fbceb61a9b82223f8601cd. early allowance recovery remains unverified.

ran vp i and the real background dev server on github/main (d2429f733c30772ab5ef8818685615e656ec432f) in ../main, then on the exact pr head. stopped main by its captured process group before starting the pr server. drove /usr/bin/google-chrome with the repo's playwright dependency. created the usage-recovery project through the app against a real git repository and sent real codex gpt-6.1-sol turns through the local proxy. no scripted provider, injected failure, seeded thread, or synthetic usage reading.

normal command execution completed on both versions. on the pr, stop generation interrupted a turn; a subsequent message in the same thread completed. read-only inspection of the dev databases confirmed:

main: codex / gpt-6.1-sol, run 1 completed
pr:   codex / gpt-6.1-sol, run 1 completed
pr:   codex / gpt-6.1-sol, runs 2 and 3 interrupted
pr:   codex / gpt-6.1-sol, run 4 completed
provider stream: windows=[]; unavailable.reason=probeFailed
message: Codex could not read usage (JSON-RPC -32600).

blacksmith ran vp test run apps/server/src/orchestration-v2/runtimeLayer.test.ts -t "usage-limit recovery": 19 passed, 72 excluded by the filter. current head has 27 successful ci checks. no source changes or pushes in this pass.

main's real usage probe:

pr's real usage probe with matching scenario:

neighboring stop and follow-up behavior, not early-recovery proof. inspected the clip and checked uploaded playback and seeking in t3's browser:

Not verified

the original bug on main and the early-resume fix: neither running server can read usage windows through this proxy, and no genuine usage-limit failure/reset credit was available. without a real future reset followed by restored allowance, the worker's changed path cannot be exercised. the focused tests do not replace that proof.

also unverified: recovery cancellation/snooze/reset fallback through an exhausted real account; other providers; native desktop/mobile; relay/tunnel. the previously documented saved-reset shell/cancel-control issue and cross-provider wait inheritance remain present in source, but were not reproduced with a real provider here. this pass does not clear those findings or establish merge readiness.

addressed the remaining cross-provider finding: reset waits now stay with the early retry's run id. early dispatch already records that id, so bounded retries remain intact. the pending-request receipt finding was already fixed by request-id rotation and remains covered.

fixed the saved-reset cancel-control issue in the shared error summary. both server shells and loaded web/desktop/mobile runtimes now use the matching recovery reset when an early retry omits one. explicit provider resets still take precedence.

merged github/main without rebasing and pushed two scoped fix rounds to the contributor's existing branch. final head: 9f7544a658676a72f48e648853490f1b5c9340dc. github reports CLEAN/MERGEABLE; all checks passed or were skipped, with none pending or failing. two independent source reviews found no remaining blockers. no unresolved review threads.

blacksmith check:changed passed format, lint, both knip checks, graphql, 35 package typechecks, and 196 tests. regressions failed before their fixes. no new test files.

playwright with system chrome verified the loaded client can cancel, rearm, snooze, and wake through the real server. the matching before/after captures use seeded early-retry state with no provider reset. this proves the control and mutation path, not real exhausted-provider allowance restoration. real allowance restoration, native clients, and relay/tunnel remain unverified this pass. the date-parsing nit is deferred to preserve scope.

Re your comment: applied in 016489f, with follow-ups a65b7c6 and 30c1df5. This ports the idea from your e43d5f4 onto this branch.

• The recovery's own snooze, one that ends at the recovery's reset, no longer blocks the early continuation. That covers the candidate query, the recovery sweep and the orchestrator's guard. When the continuation goes out, the thread leaves Snoozed.
• A snooze set from the sidebar still holds, even one that ends at the reset, because it clears the recovery's snooze flag.
• The existing early-recovery case in runtimeLayer.test.ts now also runs with the snooze on. It fails on the previous head and passes here.

The PR description has captures from the app.

One case is left as is: if the early continuation hits the same limit again, the thread waits for the reset without being snoozed again. A sweep can't tell that state from someone waking the thread.

Merged main to clear the conflicts. The arming path keeps both the 24h overdue guard from main and this PR's waitForReset re-arm. The limitRecoveryCommand arming fixture now sets providerInstanceId, which this PR made required. check:changed passes: format, lint, knip, typecheck of 36 packages, 9 test files.

Confirming this fixes my case: I redeemed a Codex reset manually while a thread had auto-resume armed, and it stayed paused until I sent a message by hand. The polling approach here also covers resets done outside T3 and plan upgrades. Thanks for working on it.

Thanks for the report from a real Codex account: after a manual reset with auto-resume armed, the thread stayed paused until you sent a message by hand. That is the case this PR targets.

A run with a real Claude Code showed the early continuation firing on a reading Claude Code served from its cache before the failure.
Branch only21mil:fix/i22307-usage-limit-auto-resume-ignores-allowanc-offer at faf234f406 fast-forwards this head: one merge of main, then one commit.
That commit dates Claude's usage reading by the Claude Code cache entry it came from, so a reading cached before the failure no longer counts as taken after it.

@maria-rcks your head was 6b09b4a57a; the branch is now faf234f406, the commit my offer comment named.
It adds one merge of main 01b815ffc2 with no conflicts, then one commit in ClaudeProvider.ts and ClaudeCapabilitiesProbe.test.ts: a Claude usage reading is dated by when Claude Code fetched it, so a reading Claude Code cached before the failure no longer sends the early continuation.
Your commits and your description section are unchanged; the Verification section now covers main, your head, the local merge and this head, including a signed-in Claude run.

I merged main into this branch because commits on main changed files this PR changes.
The merge had no conflicts, and this PR's own change is unchanged.

The automated pipeline stopped on this pull request and will not merge it:

the fix is written and reviewed, but proving it needs something no worker has:

• Permanent blocker: First-party Claude OAuth access supporting subscription usage readings, usage-limit failures, and reset-credit redemption is unavailable on these VMs. The scenario replay provider cannot create the required usage-limit-stopped state or reproduce this provider-specific flow.
• Early automatic continuation and the cancellation, explicit-stop, manual-snooze, and other recovery guards were not observed in the real app on either commit.

It stays open for a person to finish or close.

I merged main to clear the conflict with #33275.
The one part I resolved by hand is in ClaudeProvider.ts, where main's configurable probe timeout and this branch's usage cache file are both kept; nothing else changed.

I ran this on 2026-11-13 on 7d517188bd (the main ff55baa008 merges) and ff55baa008 with Claude Code 2.1.323 signed in and Claude Haiku 5.5; a proxy turned one turn per thread into a usage-limit 429 and passed the rest to the real API.
Early continuation: on 7d517188bd the thread stayed limited through two fresh usage readings; on ff55baa008 a thread sent "Continue where you left off." on a fresh reading and the real model answered (a second thread's one early continuation met another injected 429, then it waited for the reset).
Guards on ff55baa008: Cancel auto-resume, a sidebar snooze and archive each held through that same reading; a new message ran as a normal turn with no continuation after it; a limited thread has no stop control, so explicit stop rests on its tests.
The red Test job on ff55baa008 failed in Checkout (git fetch ended in early EOF), so no test ran. Not shown: a really exhausted plan, and redeeming a real reset credit.
Captures and the proxy timeline: https://gist.github.com/only21mil/9fb29ab9970efbc4fcbdd1c01c9c992a

I merged main into ff55baa008 as 3be580b6b8 so the required Test and Check run again, and nothing conflicted. Test had failed in its Checkout step before any test ran. The fix's files are unchanged.

I merged main bc7fe11002 into 3be580b6b8 as 33f3ef5fd4, and nothing conflicted. Main's #36327 now imports HostProcess in ClaudeProvider.ts as well, so the merge drops this branch's copy of that import. No other line of the fix changed.