Skip to content

Commit f4bcebf

Browse files
committed
fix(t3x): stop a user message destroying a pending auto-resume
Closes #39. Two defects compounded into a permanently stranded thread: a usage limit armed a resume, the user typed "keep going" at the banner, that message was itself rejected so it started nothing, and the wake tick then threw the arm away as `user-took-over`. Net pending afterwards: zero. Measured on the reporting install at 4 of 17 armed resumes (~24%) since #6. 1. `guards.ts` — drop the `user-took-over` branch. It is the same negative-evidence mistake #6 fixed on the line below it: "a message exists that wasn't there when we armed" is not evidence the human took the wheel, and in practice it is the opposite signal. Everything the branch reached for is still covered — `progressing` (actively driving), `awaiting-input` (blocked on a prompt), `thread-advanced` (a different live turn), and the per-thread switch for "stop entirely". `baseline.newestUserMessageId` stays: it is part of the persisted record, it is re-captured on every (re)schedule, and it is what makes a stranded arm diagnosable from the state file. 2. `decide.ts` — narrow `already-pending`. A rejection naming a CONCRETE reset time later than the pending one now supersedes it instead of being dropped, so a `seven_day` limit landing on an armed `five_hour` no longer leaves the arm firing into a window that is still shut. Restricted to `windowOpensInFuture` on purpose: a ladder-derived time is `nowMs + delay`, so it is later on every telemetry re-emit, and letting those supersede would push the arm out forever and flood the timeline. An earlier reset never supersedes either — the existing arm is already the conservative choice. A supersede re-writes the record with a fresh `captureBaseline`, so it is also the re-arm path, and posts a distinct `t3x.auto-resume.rescheduled` note rather than a second "scheduled". The issue's third item — detecting the org monthly spend limit — is a genuinely separate feature (no `account.rate-limits.updated` path reaches it at all) and is deliberately not in scope here, per the triage. Tests: 4 new behavioural cases, each verified to fail against the old code — guards (a newer user message does not cancel; it still cancels when that message is actually being worked on), decide (supersede, earlier-window no-op, ladder-churn guard), and two reactor integration tests covering the reported shape end to end. Reactor suite run 12x for flake. `docs/t3x/loop/DESIGN.md` §6 gets a dated note: guard #9 still stands, but the hazard it was defending against is gone.
1 parent 98e00e3 commit f4bcebf

7 files changed

Lines changed: 178 additions & 23 deletions

File tree

‎apps/server/src/t3x/autoResume/Reactor.test.ts‎

Lines changed: 53 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,10 @@ describe("AutoResumeReactor (integration)", () => {
309309
}).pipe(Effect.scoped, Effect.provide(Layer.mergeAll(NodeServices.layer, TestClock.layer()))),
310310
);
311311

312-
it.effect("does NOT resume when the user takes over before the window reopens", () =>
312+
// Regression for radroid/t3code#39 — the reported shape: the user types "keep going"
313+
// while the resume is pending, that message is itself rejected by a limit so it starts
314+
// nothing, and the arm used to be destroyed as `user-took-over`. It must survive.
315+
it.effect("still resumes when the user posted a message while the resume was pending", () =>
313316
Effect.gen(function* () {
314317
const { dispatched, modelRef, deps, store } = yield* harness(readModel({}), [
315318
rejectedEvent(100),
@@ -318,7 +321,7 @@ describe("AutoResumeReactor (integration)", () => {
318321
yield* Effect.gen(function* () {
319322
yield* settleUntil(scheduledOne(store), "detection to schedule from the pre-loaded event");
320323

321-
// User sends a new message before the resume is due -> guard must cancel.
324+
// A new user message lands, and goes nowhere: the thread is still idle at wake time.
322325
yield* Ref.set(
323326
modelRef,
324327
readModel({
@@ -329,10 +332,56 @@ describe("AutoResumeReactor (integration)", () => {
329332
}),
330333
);
331334

332-
yield* advancePastResume;
335+
yield* advanceUntil(dispatchedIncludes(dispatched, "thread.turn.start"), "the resume turn");
333336

334337
const commands = yield* Ref.get(dispatched);
335-
assert.notInclude(types(commands), "thread.turn.start");
338+
assert.strictEqual(
339+
commands.filter((c) => c.type === "thread.turn.start").length,
340+
1,
341+
"the resume must fire despite the newer user message",
342+
);
343+
const summaries = commands
344+
.filter((c) => c.type === "thread.activity.append")
345+
.map((c) => (c as unknown as { activity: { summary: string } }).activity.summary);
346+
assert.isFalse(
347+
summaries.some((s) => s.includes("cancelled")),
348+
"no cancellation may be posted",
349+
);
350+
}).pipe(Effect.provide(AutoResumeReactorLive.pipe(Layer.provideMerge(deps))));
351+
}).pipe(Effect.scoped, Effect.provide(Layer.mergeAll(NodeServices.layer, TestClock.layer()))),
352+
);
353+
354+
// The other half of #39: a second, longer limit arriving while a resume is armed used
355+
// to be dropped as `already-pending`, so the arm fired into a window still shut.
356+
it.effect("moves a pending resume out when a longer limit supersedes it", () =>
357+
Effect.gen(function* () {
358+
const { dispatched, deps, store } = yield* harness(readModel({}), [
359+
rejectedEvent(100), // due at 100_000 + 60_000 margin
360+
rejectedEvent(1000), // due at 1_000_000 + 60_000 margin
361+
]);
362+
363+
yield* Effect.gen(function* () {
364+
yield* settleUntil(
365+
store.listPending.pipe(Effect.map((p) => p[0]?.resumeAtMs === 1_060_000)),
366+
"the second rejection to supersede the first arm",
367+
);
368+
369+
assert.strictEqual(
370+
(yield* store.listPending).length,
371+
1,
372+
"superseding replaces the arm, it does not add a second one",
373+
);
374+
375+
const kinds = (yield* Ref.get(dispatched))
376+
.filter((c) => c.type === "thread.activity.append")
377+
.map((c) => (c as unknown as { activity: { kind: string } }).activity.kind);
378+
assert.deepStrictEqual(kinds, ["t3x.auto-resume.scheduled", "t3x.auto-resume.rescheduled"]);
379+
380+
// The original 160_000 due time passes without firing: that window is still shut.
381+
yield* advancePastResume; // 8 x 30s = 240_000ms
382+
assert.notInclude(types(yield* Ref.get(dispatched)), "thread.turn.start");
383+
384+
yield* advanceUntil(dispatchedIncludes(dispatched, "thread.turn.start"), "the resume turn");
336385
}).pipe(Effect.provide(AutoResumeReactorLive.pipe(Layer.provideMerge(deps))));
337386
}).pipe(Effect.scoped, Effect.provide(Layer.mergeAll(NodeServices.layer, TestClock.layer()))),
338387
);

‎apps/server/src/t3x/autoResume/Reactor.ts‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ const makeSupervisor = Effect.gen(function* () {
142142

143143
const plan = planSchedule({
144144
verdict,
145-
hasPending: record.pending !== null,
145+
pendingResumeAtMs: record.pending?.resumeAtMs ?? null,
146146
nowMs,
147147
firedRecently,
148148
firedInCapWindow,
@@ -156,6 +156,11 @@ const makeSupervisor = Effect.gen(function* () {
156156
const thread = snapshot.threads.find((t) => t.id === threadId);
157157
if (!thread || threadIsGone(thread) || !isClaudeThread(thread)) return;
158158

159+
// Replacing an existing arm (a later window superseded it — see decide.ts). The
160+
// fresh `captureBaseline` below is what re-baselines the resume onto whatever the
161+
// thread looks like now, so a supersede is also the re-arm path for #39.
162+
const superseded = record.pending !== null;
163+
159164
yield* store.schedule({
160165
threadId,
161166
resumeAtMs: plan.resumeAtMs,
@@ -165,11 +170,14 @@ const makeSupervisor = Effect.gen(function* () {
165170
});
166171

167172
const waitMinutes = Math.max(0, Math.round((plan.resumeAtMs - nowMs) / 60_000));
173+
const limitType = verdict.rateLimitType ?? "window";
168174
yield* appendActivity(
169175
threadId,
170176
"info",
171-
"t3x.auto-resume.scheduled",
172-
`Usage limit reached (${verdict.rateLimitType ?? "window"}). Auto-resume scheduled in ~${waitMinutes} min.`,
177+
superseded ? "t3x.auto-resume.rescheduled" : "t3x.auto-resume.scheduled",
178+
superseded
179+
? `Usage limit window pushed back (${limitType}). Auto-resume rescheduled to ~${waitMinutes} min from now.`
180+
: `Usage limit reached (${limitType}). Auto-resume scheduled in ~${waitMinutes} min.`,
173181
);
174182
});
175183

‎apps/server/src/t3x/autoResume/decide.test.ts‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,12 @@ const rejected = (o: Partial<RateLimitVerdict> = {}): RateLimitVerdict => ({
1414
});
1515

1616
// Base input = a fresh rejection, nothing pending, no prior fires, not capped.
17+
// With the default verdict (resetsAtMs 1_000_000) and nowMs 0 the computed resume is
18+
// 1_000_000 + 60_000 margin = 1_060_000; the pending-comparison tests are anchored on that.
1719
const plan = (o: Partial<PlanScheduleInput> = {}) =>
1820
planSchedule({
1921
verdict: rejected(),
20-
hasPending: false,
22+
pendingResumeAtMs: null,
2123
nowMs: 0,
2224
firedRecently: 0,
2325
firedInCapWindow: 0,
@@ -41,7 +43,36 @@ describe("planSchedule", () => {
4143
});
4244

4345
it("skips when a resume is already pending (dedupes telemetry re-emits)", () => {
44-
expect(plan({ hasPending: true })).toEqual({ kind: "skip", reason: "already-pending" });
46+
// Same window re-emitted: computed 1_060_000 is not later than the arm, so nothing moves.
47+
expect(plan({ pendingResumeAtMs: 1_060_000 })).toEqual({
48+
kind: "skip",
49+
reason: "already-pending",
50+
});
51+
});
52+
53+
// radroid/t3code#39: a second, longer limit landing on top of an armed shorter one used
54+
// to be dropped, so the arm fired into a window that was still shut.
55+
it("re-schedules when a concrete later reset window supersedes the pending arm", () => {
56+
const p = plan({ pendingResumeAtMs: 500_000 });
57+
expect(p.kind).toBe("schedule");
58+
if (p.kind !== "schedule") return;
59+
expect(p.resumeAtMs).toBe(1_000_000 + config.safetyMarginMs);
60+
});
61+
62+
it("keeps the existing arm when the new window opens earlier", () => {
63+
expect(plan({ pendingResumeAtMs: 5_000_000 })).toEqual({
64+
kind: "skip",
65+
reason: "already-pending",
66+
});
67+
});
68+
69+
// The churn guard. A ladder-derived time is `nowMs + delay`, so it is later on every
70+
// re-emit; if those superseded, a persistent limit would push the arm out forever and
71+
// post a reschedule note each time.
72+
it("never lets a backoff-ladder re-emit push out a pending arm", () => {
73+
expect(
74+
plan({ verdict: rejected({ resetsAtMs: null }), nowMs: 100_000, pendingResumeAtMs: 50_000 }),
75+
).toEqual({ kind: "skip", reason: "already-pending" });
4576
});
4677

4778
it("skips when the thread has hit the 24h cap (stops re-scheduling + misleading notes)", () => {

‎apps/server/src/t3x/autoResume/decide.ts‎

Lines changed: 25 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,11 @@ export interface SkipPlan {
2323

2424
export interface PlanScheduleInput {
2525
readonly verdict: RateLimitVerdict;
26-
/** Whether this thread already has a resume pending (one-per-thread invariant). */
27-
readonly hasPending: boolean;
26+
/**
27+
* `resumeAtMs` of this thread's pending resume, or null when nothing is armed.
28+
* Still one-pending-per-thread — a new plan replaces the arm rather than adding one.
29+
*/
30+
readonly pendingResumeAtMs: number | null;
2831
readonly nowMs: number;
2932
/** Fires for this thread within the recent backoff window — drives the ladder. */
3033
readonly firedRecently: number;
@@ -36,10 +39,19 @@ export interface PlanScheduleInput {
3639
/**
3740
* Decide whether/when to schedule a resume for a rejection.
3841
*
39-
* Dedup is purely "one pending per thread": while a resume is pending we skip every
40-
* telemetry re-emit (no churn). Once a resume fires, its pending is cleared, so the next
41-
* rejection re-arms naturally — and because `firedRecently` has incremented, its resume
42-
* is spaced out on the backoff ladder rather than tight-looping.
42+
* Dedup is "one pending per thread": while a resume is pending we skip every telemetry
43+
* re-emit (no churn). Once a resume fires, its pending is cleared, so the next rejection
44+
* re-arms naturally — and because `firedRecently` has incremented, its resume is spaced
45+
* out on the backoff ladder rather than tight-looping.
46+
*
47+
* One narrow exception (radroid/t3code#39): a rejection that names a CONCRETE reset time
48+
* LATER than the pending one supersedes it. A `seven_day` limit landing on top of an
49+
* armed `five_hour` used to be dropped outright, so the arm fired into a window that was
50+
* still shut and burned an attempt. The exception is deliberately restricted to
51+
* `windowOpensInFuture` — a ladder-derived time is `nowMs + delay`, which grows with
52+
* every re-emit, so allowing those to supersede would push the arm out forever and flood
53+
* the timeline with reschedule notes. An earlier reset time never supersedes either: the
54+
* existing arm is already the conservative choice.
4355
*
4456
* The 24h cap is checked HERE (at schedule time) as well as at fire time. Checking it at
4557
* schedule time stops a capped-out thread from re-scheduling — and re-posting a misleading
@@ -56,16 +68,20 @@ export interface PlanScheduleInput {
5668
* skip) or blocks re-arming a persistent limit. One-pending + backoff avoids both.
5769
*/
5870
export function planSchedule(input: PlanScheduleInput): SchedulePlan | SkipPlan {
59-
const { verdict, hasPending, nowMs, firedRecently, firedInCapWindow, config } = input;
71+
const { verdict, pendingResumeAtMs, nowMs, firedRecently, firedInCapWindow, config } = input;
6072

6173
if (!verdict.rejected) return { kind: "skip", reason: "not-rejected" };
62-
if (hasPending) return { kind: "skip", reason: "already-pending" };
63-
if (firedInCapWindow >= config.maxResumesPer24h) return { kind: "skip", reason: "capped" };
6474

6575
const windowOpensInFuture = verdict.resetsAtMs !== null && verdict.resetsAtMs > nowMs;
6676
const resumeAtMs = windowOpensInFuture
6777
? verdict.resetsAtMs! + config.safetyMarginMs
6878
: nowMs + backoffDelayMs(config.backoffLadderMs, firedRecently);
6979

80+
if (pendingResumeAtMs !== null) {
81+
const supersedes = windowOpensInFuture && resumeAtMs > pendingResumeAtMs;
82+
if (!supersedes) return { kind: "skip", reason: "already-pending" };
83+
}
84+
if (firedInCapWindow >= config.maxResumesPer24h) return { kind: "skip", reason: "capped" };
85+
7086
return { kind: "schedule", resumeAtMs };
7187
}

‎apps/server/src/t3x/autoResume/guards.test.ts‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,15 +160,38 @@ describe("cancelReason", () => {
160160
expect(cancelReason(base(), baseline())).toBeNull();
161161
});
162162

163-
it("detects a new user message", () => {
163+
// Regression for radroid/t3code#39. The removed `user-took-over` branch cancelled on
164+
// any newer user message. The message that trips it is typically "keep going" typed at
165+
// the usage-limit banner — and is itself rejected by the same limit, so it starts
166+
// nothing and the thread is left with no pending resume at all.
167+
it("does NOT cancel when a new user message arrived while the resume was pending", () => {
164168
const thread = makeThread({
165169
messages: [
166170
{ id: "u1", role: "user" },
167171
{ id: "u2", role: "user" },
168172
],
169173
latestTurnId: "turn-1",
170174
});
171-
expect(cancelReason(thread, baseline())).toBe("user-took-over");
175+
expect(cancelReason(thread, baseline())).toBeNull();
176+
});
177+
178+
// The baseline still records it — it is persisted with the pending resume and is what
179+
// makes a stranded arm diagnosable from the state file.
180+
it("still captures the newest user message id in the baseline", () => {
181+
expect(baseline().newestUserMessageId).toBe("u1");
182+
});
183+
184+
// What the removed branch was actually reaching for: a user who is driving right now.
185+
// That is `progressing`, and it still cancels.
186+
it("cancels a new user message that is actually being worked on", () => {
187+
const thread = makeThread({
188+
messages: [
189+
{ id: "u1", role: "user" },
190+
{ id: "u2", role: "user" },
191+
],
192+
status: "running",
193+
});
194+
expect(cancelReason(thread, baseline())).toBe("progressing");
172195
});
173196

174197
it("detects a new turn since scheduling", () => {

‎apps/server/src/t3x/autoResume/guards.ts‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,12 @@ import type { OrchestrationThread } from "@t3tools/contracts";
1212

1313
/**
1414
* Baseline captured when a resume is scheduled, re-checked immediately before dispatch
15-
* to detect that the thread moved on (user took over, a new turn ran, etc.).
15+
* to detect that the thread moved on.
16+
*
17+
* `newestUserMessageId` is recorded but is deliberately NOT a cancel condition — see the
18+
* block in `cancelReason` (radroid/t3code#39). It stays in the shape because it is part
19+
* of the persisted pending-resume record (`state.ts`), it is re-captured on every
20+
* (re)schedule, and it is what makes a stranded arm diagnosable from the state file.
1621
*/
1722
export interface GuardBaseline {
1823
readonly newestUserMessageId: string | null;
@@ -114,7 +119,6 @@ export type CancelReason =
114119
| "not-claude"
115120
| "progressing"
116121
| "awaiting-input"
117-
| "user-took-over"
118122
| "thread-advanced";
119123

120124
/**
@@ -129,7 +133,23 @@ export function cancelReason(
129133
if (!isClaudeThread(thread)) return "not-claude";
130134
if (threadIsProgressing(thread)) return "progressing";
131135
if (hasOpenBlockingRequest(thread.activities)) return "awaiting-input";
132-
if (newestUserMessageId(thread) !== baseline.newestUserMessageId) return "user-took-over";
136+
// A new user message does NOT cancel (radroid/t3code#39). This branch used to read
137+
// `newestUserMessageId(thread) !== baseline.newestUserMessageId` and return
138+
// "user-took-over", which is the same negative-evidence mistake #6 fixed one line
139+
// below: "a message exists that wasn't there when we armed" is not evidence that the
140+
// human took the wheel. In practice it is the opposite — the message that trips it is
141+
// typed the moment the usage-limit banner appears, which is exactly when someone is
142+
// stepping away ("keep going through the night"). That message is then usually rejected
143+
// by the same limit, so it starts nothing, and the wake tick destroys the only pending
144+
// resume. Measured on this install: 4 of 17 armed resumes (~24%) lost this way.
145+
//
146+
// Everything the branch was reaching for is still covered:
147+
// * the user is actively driving right now -> `progressing`
148+
// * the thread is blocked on a prompt -> `awaiting-input`
149+
// * a different turn is live at fire time -> `thread-advanced`
150+
// * the user wants no resume at all -> the per-thread switch, honoured
151+
// in `Reactor.fireOne`.
152+
//
133153
// Advancement needs POSITIVE evidence: a different, non-null turn id. The snapshot's
134154
// `latestTurn` is joined on `projection_threads.latest_turn_id`, which is populated
135155
// only while a turn is active — so a usage limit that lands mid-turn captures the

‎docs/t3x/loop/DESIGN.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,14 @@ is PubSub-backed, so a second subscriber does not steal auto-resume's events. Be
222222
that is rejected by a limit produces no `updatedAt` movement, so it takes a strike and the thread stops
223223
after two.
224224

225+
> **Update 2026-08-11 (#39).** The `user-took-over` branch quoted above is **gone** — a newer user
226+
> message no longer cancels a pending resume, because the same "keep going" message that tripped it is
227+
> typically the user stepping away, and it was destroying ~24% of armed resumes. So a loop nudge landing
228+
> mid-wait no longer destroys rate-limit recovery. **Guard #9 still stands**, for the other reason:
229+
> nudging a thread that is sitting inside a usage-limit window is pointless work. What changes is that
230+
> #9 is now a politeness rule rather than the only thing standing between a nudge and a stranded thread.
231+
> The rest of §6 — the second fiber, `rateLimitedUntilMs`, the strike interlock — is unaffected.
232+
225233
---
226234

227235
## 7. Budget visibility & settings

0 commit comments

Comments
 (0)