Skip to content

Commit 3a8554b

Browse files
olddonkeyclaude
andcommitted
fix(responses): record the serving route only after it serves
`updateReasoningReplayServingIdentity` compared and committed in one call, and `bindRouteReasoningReplayScope` calls it before the request goes out — so the candidate destination was recorded whether or not that request ever completed. turn 1 -> A succeeds record = A turn 2 -> B: A != B, strip A blobs record = B (committed too early) ... this request then fails (rate limit, transport, 5xx) turn 3 -> retry B: B == B, no strip but the transcript still carries A-minted blobs -> rejected The opaque-blob recovery rescues turn 3, so this degraded rather than broke: one wasted round trip and one turn of degraded reasoning on a path meant to be deterministic. The record's meaning was the defect — it should mean "this destination served this thread", and a request that never completed served nothing. Split the call in two. `reasoningReplayServingIdentityChanged` compares without writing; `commitReasoningReplayServingIdentity` records, and runs only at a successful terminal response. Bounded discipline is unchanged: same LRU/TTL and byte accounting, same refusal to record without a durable identity dimension, same fail-soft direction where no record still means keep the blobs. For bridged transports a terminal means `completed` or `incomplete`. For streamed passthrough it means a non-error upstream status before relay starts: waiting for SSE completion would retain request state for the stream's lifetime, and a later body failure does not undo that the destination accepted and served the turn. That boundary is stated in the code rather than implied. The two post-recovery re-records are gone — a successful recovery now reaches the same terminal commit as any other success. Regression test: A succeeds, an A->B turn strips and then fails, and the next B request for the same thread still strips. Verified it fails against the old code. Co-Authored-By: Claude Fable 5 <[email protected]>
1 parent be2cf59 commit 3a8554b

6 files changed

Lines changed: 198 additions & 65 deletions

File tree

‎src/responses/reasoning-replay-cache.ts‎

Lines changed: 30 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -133,32 +133,51 @@ function sweepExpiredServingIdentities(at: number): void {
133133
}
134134
}
135135

136+
function servingIdentityFor(
137+
scope: OcxReasoningReplayScopeRef | undefined,
138+
): { threadId: string; identity: string } | undefined {
139+
const threadId = scope?.clientThreadId;
140+
const identityTuple = tupleForServingIdentity(scope?.current);
141+
if (!nonEmpty(threadId) || !identityTuple) return undefined;
142+
return { threadId, identity: JSON.stringify(identityTuple) };
143+
}
144+
136145
/**
137-
* Compare this request's route with the last route recorded for its client thread, then
138-
* record the current route. A live mismatch means replayed opaque reasoning was minted by
139-
* another backend and must not be forwarded to this one.
146+
* Compare this request's route with the last successfully serving route for its client thread.
147+
* A live mismatch means replayed opaque reasoning was minted by another backend and must not be
148+
* forwarded to this one. Comparison deliberately does not refresh or replace the recorded route:
149+
* a failed candidate request did not serve the thread.
140150
*
141151
* Serving provenance uses restart-stable destination and credential dimensions so token
142152
* generations and other volatile credential material cannot create false route changes. Missing
143153
* durable identity, expired, or evicted state is deliberately unknown rather than a mismatch.
144154
* This store is process-local, so a backend switch spanning a proxy restart is not detected.
145155
*/
146-
export function updateReasoningReplayServingIdentity(
156+
export function reasoningReplayServingIdentityChanged(
147157
scope: OcxReasoningReplayScopeRef | undefined,
148158
): boolean {
149-
const threadId = scope?.clientThreadId;
150-
const identityTuple = tupleForServingIdentity(scope?.current);
151-
if (!nonEmpty(threadId) || !identityTuple) return false;
152-
const identity = JSON.stringify(identityTuple);
159+
const current = servingIdentityFor(scope);
160+
if (!current) return false;
161+
const at = now();
162+
sweepExpiredServingIdentities(at);
163+
const previous = servingIdentities.get(current.threadId);
164+
return previous !== undefined && previous.identity !== current.identity;
165+
}
153166

167+
/** Record the route only after it has successfully served the client thread. */
168+
export function commitReasoningReplayServingIdentity(
169+
scope: OcxReasoningReplayScopeRef | undefined,
170+
): void {
171+
const current = servingIdentityFor(scope);
172+
if (!current) return;
154173
const at = now();
155174
sweepExpiredServingIdentities(at);
156-
const previous = servingIdentities.get(threadId);
157-
const changed = previous !== undefined && previous.identity !== identity;
175+
const previous = servingIdentities.get(current.threadId);
176+
const { threadId, identity } = current;
158177
const bytes = Buffer.byteLength(JSON.stringify([threadId, identity]), "utf8");
159178
if (bytes > MAX_TOTAL_BYTES) {
160179
deleteServingIdentity(threadId);
161-
return false;
180+
return;
162181
}
163182

164183
if (previous) deleteServingIdentity(threadId);
@@ -179,7 +198,6 @@ export function updateReasoningReplayServingIdentity(
179198
if (oldestThreadId === undefined) break;
180199
deleteServingIdentity(oldestThreadId);
181200
}
182-
return changed;
183201
}
184202

185203
function processLocalIdentity(domain: string, material: string): string {

‎src/server/responses/core.ts‎

Lines changed: 48 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,14 @@ import {
1616
import { parseRequest } from "../../responses/parser";
1717
import {
1818
bindReasoningReplayScope,
19+
commitReasoningReplayServingIdentity,
1920
reasoningReplayCodexCredentialIdentity,
2021
reasoningReplayDestinationIdentity,
2122
durableReplayDestinationIdentity,
2223
durableReplayCredentialIdentity,
2324
reasoningReplayKeyCredentialIdentity,
2425
reasoningReplayOAuthCredentialIdentity,
25-
updateReasoningReplayServingIdentity,
26+
reasoningReplayServingIdentityChanged,
2627
} from "../../responses/reasoning-replay-cache";
2728
import { awaitThoughtSignatureDurability, thoughtSignatureReplaySalt } from "../../responses/thought-signature-replay";
2829
import { buildCompactV1Output, COMPACT_PROMPT, decodeCompactionSummary, extractCompactUserMessages } from "../../responses/compaction";
@@ -510,12 +511,20 @@ function bindRouteReasoningReplayScope(args: {
510511
);
511512
// Keep this sticky for the whole outbound request: a later auth/key rebind may compare equal
512513
// after the first mismatch, but it cannot make history minted by the prior route decodable.
513-
if (updateReasoningReplayServingIdentity(parsed._reasoningReplayScope)) {
514+
if (reasoningReplayServingIdentityChanged(parsed._reasoningReplayScope)) {
514515
parsed._stripReasoningEncryptedContent = true;
515516
}
516517
bindProviderContinuationForRoute(parsed, continuationOwner);
517518
}
518519

520+
function adapterResponseReachedServingTerminal(
521+
events: readonly AdapterEvent[],
522+
response: Readonly<Record<string, unknown>>,
523+
): boolean {
524+
return (response.status === "completed" || response.status === "incomplete")
525+
&& events.some(event => event.type === "done" || event.type === "incomplete");
526+
}
527+
519528
const OPAQUE_RESPONSES_INPUT_TYPES = new Set([
520529
"reasoning",
521530
"compaction",
@@ -2755,6 +2764,9 @@ async function handleResponsesInner(
27552764
// message, and leave Codex fataling on a missing compaction item (#422).
27562765
const routedCompaction = parsed._compactionRequest === true
27572766
&& !isCanonicalOpenAiForwardProvider(route.provider);
2767+
const commitReasoningReplayServingRoute = (): void => {
2768+
commitReasoningReplayServingIdentity(parsed._reasoningReplayScope);
2769+
};
27582770
if (routedCompaction) {
27592771
delete parsed.context.tools;
27602772
delete parsed._webSearch;
@@ -3349,11 +3361,6 @@ async function handleResponsesInner(
33493361
}
33503362
break;
33513363
}
3352-
// Binding normally records before the first send. Repeat it only after a successful recovery
3353-
// so an eviction during the extra round trip cannot leave the next cross-route turn cold.
3354-
if (opaqueBlobRecoveryGuard.attempted && upstreamResponse.ok) {
3355-
updateReasoningReplayServingIdentity(parsed._reasoningReplayScope);
3356-
}
33573364
const headers = sanitizePassthroughHeaders(upstreamResponse.headers);
33583365
const resolvedModel = headers.get("openai-model")?.trim();
33593366
if (resolvedModel) logCtx.resolvedModel = resolvedModel;
@@ -3477,6 +3484,10 @@ async function handleResponsesInner(
34773484
// devlog/_fin/260731_macos_rss_retention/100_darwin_eager_optin.md).
34783485
// The bundled known-bad runtime remains on tee by default on both platforms.
34793486
if (isEventStream && upstreamResponse.body) {
3487+
// For streamed passthrough, a successful terminal response means non-error upstream status
3488+
// before relay starts. Waiting for SSE completion would retain request state across the whole
3489+
// stream; a later body failure does not undo that this destination accepted and served the turn.
3490+
commitReasoningReplayServingRoute();
34803491
const terminalRepairPolicy = providerModelResponsesTerminalRepair(
34813492
route.providerName,
34823493
route.provider,
@@ -3762,6 +3773,7 @@ async function handleResponsesInner(
37623773
return formatErrorResponse(502, "upstream_error", undeclaredToolCallMessage(undeclared));
37633774
}
37643775
}
3776+
commitReasoningReplayServingRoute();
37653777
if (rememberPassthroughResponseChecked) {
37663778
try {
37673779
rememberPassthroughResponseChecked(
@@ -3843,6 +3855,9 @@ async function handleResponsesInner(
38433855
headers,
38443856
});
38453857
}
3858+
// An unclassified passthrough body is relayed directly and has no bounded completion observer;
3859+
// use the same non-error-status success boundary as SSE instead of retaining per-stream state.
3860+
commitReasoningReplayServingRoute();
38463861
const body = relayWithAbort(upstreamResponse.body, upstream);
38473862
const turnAc = new AbortController();
38483863
const tracked = body ? trackStreamLifetime(body, turnAc, undefined, options.turnAdmissionLease) : null;
@@ -3988,13 +4003,15 @@ async function handleResponsesInner(
39884003
retryOn429Policy: rateLimitRetryPolicyFor(route.provider),
39894004
...(options.onFirstOutput ? { onFirstOutput: options.onFirstOutput } : {}),
39904005
...(options.forceEmptyResponseId ? { forceEmptyResponseId: true } : {}),
3991-
onCompletedResponse: (response, providerState) =>
4006+
onCompletedResponse: (response, providerState) => {
4007+
commitReasoningReplayServingRoute();
39924008
rememberResponseState(
39934009
parsed._rawBody,
39944010
response,
39954011
continuationStateForResponse(providerState),
39964012
responseStateOptions(adapterNeedsForcedContinuation(adapter.name)),
3997-
),
4013+
);
4014+
},
39984015
});
39994016
if (imgResponse.body) {
40004017
const imgTurnAc = new AbortController();
@@ -4072,6 +4089,7 @@ async function handleResponsesInner(
40724089
return rotatedAdapter;
40734090
},
40744091
retryOn429Policy: rateLimitRetryPolicyFor(route.provider),
4092+
onCompletedResponse: commitReasoningReplayServingRoute,
40754093
});
40764094
// Register the sidecar stream as an active turn so drainAndShutdown waits for (or aborts)
40774095
// in-flight web-search turns instead of skipping them during graceful shutdown.
@@ -4235,15 +4253,17 @@ async function handleResponsesInner(
42354253
if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage;
42364254
}
42374255
},
4238-
...(routedCompaction ? {} : {
4239-
onCompletedResponse: (response: Record<string, unknown>, providerState?: OcxProviderContinuationState) =>
4256+
onCompletedResponse: (response: Record<string, unknown>, providerState?: OcxProviderContinuationState) => {
4257+
commitReasoningReplayServingRoute();
4258+
if (!routedCompaction) {
42404259
rememberResponseState(
42414260
parsed._rawBody,
42424261
response,
42434262
continuationStateForResponse(providerState),
42444263
responseStateOptions(adapterNeedsForcedContinuation(adapter.name)),
4245-
),
4246-
}),
4264+
);
4265+
}
4266+
},
42474267
},
42484268
);
42494269
const bridgeTurnAc = new AbortController();
@@ -4306,6 +4326,9 @@ async function handleResponsesInner(
43064326
// buildResponseJSON; bound the durability window before the JSON becomes
43074327
// externally visible.
43084328
await awaitThoughtSignatureDurability();
4329+
if (adapterResponseReachedServingTerminal(events, json)) {
4330+
commitReasoningReplayServingRoute();
4331+
}
43094332
return new Response(JSON.stringify(json), { headers: { "Content-Type": "application/json" } });
43104333
}
43114334

@@ -4699,11 +4722,6 @@ async function handleResponsesInner(
46994722
}
47004723
break;
47014724
}
4702-
// Binding normally records before the first send. Repeat it only after a successful recovery
4703-
// so an eviction during the extra round trip cannot leave the next cross-route turn cold.
4704-
if (opaqueBlobRecoveryGuard.attempted && upstreamResponse.ok) {
4705-
updateReasoningReplayServingIdentity(parsed._reasoningReplayScope);
4706-
}
47074725
if (!upstreamResponse.ok) {
47084726
if (options.comboAttempt) {
47094727
// No pre-read guard: `consumeComboFailure` -> `readBoundedResponseBody` reads
@@ -5114,18 +5132,20 @@ async function handleResponsesInner(
51145132
if (logCtx.activeAttempt) logCtx.activeAttempt.usage = usage;
51155133
}
51165134
},
5117-
// Compaction turns must NOT enter the continuation cache: _rawBody still holds the full
5118-
// PRE-compaction history, and a later previous_response_id expansion would rehydrate the
5119-
// giant stale chain Codex just replaced.
5120-
...(routedCompaction ? {} : {
5121-
onCompletedResponse: (response: Record<string, unknown>, providerState?: OcxProviderContinuationState) =>
5135+
onCompletedResponse: (response: Record<string, unknown>, providerState?: OcxProviderContinuationState) => {
5136+
commitReasoningReplayServingRoute();
5137+
// Compaction turns must NOT enter the continuation cache: _rawBody still holds the full
5138+
// PRE-compaction history, and a later previous_response_id expansion would rehydrate the
5139+
// giant stale chain Codex just replaced.
5140+
if (!routedCompaction) {
51225141
rememberResponseState(
51235142
parsed._rawBody,
51245143
response,
51255144
continuationStateForResponse(providerState),
51265145
responseStateOptions(activeAdapter.name === "kiro"),
5127-
),
5128-
}),
5146+
);
5147+
}
5148+
},
51295149
},
51305150
);
51315151
const bridgeTurnAc = new AbortController();
@@ -5200,6 +5220,9 @@ async function handleResponsesInner(
52005220
}
52015221
// #1926 gap 2: same buffered-path durability bound as the primary branch.
52025222
await awaitThoughtSignatureDurability();
5223+
if (adapterResponseReachedServingTerminal(events, json)) {
5224+
commitReasoningReplayServingRoute();
5225+
}
52035226
return new Response(JSON.stringify(json), { headers: { "Content-Type": "application/json" } });
52045227
}
52055228

‎src/web-search/loop.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,8 @@ export interface WebSearchLoopDeps {
310310
on429?: (retryAfterHeader: string | null) => ProviderAdapter | null;
311311
/** Opt-in same-target 429 policy (key-auth providers). When present, 429 replays on the SAME key before on429 rotation. */
312312
retryOn429Policy?: Required<RateLimitRetryPolicy> | null;
313+
/** Called only when the final bridged Responses stream reaches completed or incomplete. */
314+
onCompletedResponse?: (response: Record<string, unknown>) => void;
313315
}
314316

315317
/**
@@ -884,6 +886,7 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise<Respons
884886
...(deps.stallTimeoutSec !== undefined ? { stallTimeoutSec: deps.stallTimeoutSec } : {}),
885887
...(deps.onFirstOutput ? { onFirstOutput: deps.onFirstOutput } : {}),
886888
...(deps.onUsage ? { onUsage: deps.onUsage } : {}),
889+
...(deps.onCompletedResponse ? { onCompletedResponse: deps.onCompletedResponse } : {}),
887890
},
888891
);
889892
return new Response(sse, { headers: SSE_HEADERS });

‎structure/04_transports-and-sidecars.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -563,8 +563,12 @@ its summary survive; `status` has already been removed on every path. Missing, e
563563
identity state is unknown. The comparison uses the durable destination and credential identities
564564
with the provider, adapter, and model, so OAuth token-generation refreshes do not look like backend
565565
changes; when either durable dimension is unavailable it refuses to record rather than falling back
566-
to a volatile identity. This deterministic pre-flight is the primary path and covers threads the
567-
process has served while their record remains inside the TTL/LRU bounds. Missing, expired, evicted, and
566+
to a volatile identity. Route binding only compares: it does not replace the recorded identity until
567+
the destination successfully serves the turn. Bridged streams commit on a completed or incomplete
568+
terminal; native passthrough streams use the non-error upstream status before relay as their success
569+
boundary so the proxy does not retain request state across the whole stream. This deterministic
570+
pre-flight is the primary path and covers threads the process has served while their record remains
571+
inside the TTL/LRU bounds. Missing, expired, evicted, and
568572
pre-process history stays fail-soft on the first send. If a Responses upstream then returns its own
569573
self-identifying opaque-blob 4xx (`invalid_encrypted_content`, or xAI's two `invalid-argument`
570574
decoder errors), the proxy rebuilds once through the same sanitation path: reasoning
@@ -591,7 +595,7 @@ conversation's last-serving identity and cause a later main-model turn to strip
591595
- 목적과 의도: Keep same-backend opaque reasoning replay while preventing backend-private blobs and output-only fields from breaking the first turn after a route change.
592596
- 기존 구현 및 제약 조건: Reasoning-input sanitation already handled raw content and `ocxr1:` envelopes; the replay cache already supplied a bounded, thread-scoped physical-route identity, but no record connected that identity to native `encrypted_content` provenance.
593597
- 검토한 주요 대안: Strip every opaque blob, persist provenance across restarts, trust generic 4xx prose, rely only on a retry, or combine deterministic route comparison with a narrowly identified recovery.
594-
- 선택한 방식: Remove output-only `status` from every reasoning input item without changing the pre-existing raw-`content` blanking rule; compare and update a 64-entry/256 KiB/one-hour in-process serving-identity record using durable destination and credential dimensions at request time, pass a proven change into the Responses adapter before the first send, and use one self-identified opaque-blob recovery only when provenance was unknown.
598+
- 선택한 방식: Remove output-only `status` from every reasoning input item without changing the pre-existing raw-`content` blanking rule; compare a 64-entry/256 KiB/one-hour in-process serving-identity record using durable destination and credential dimensions before sending, commit it only after the selected destination succeeds, pass a proven change into the Responses adapter before the first send, and use one self-identified opaque-blob recovery only when provenance was unknown.
595599
- 다른 대안 대신 이 방식을 선택한 이유: The former blob/status coupling was defensive rather than observed, and live backends showed that removing `status` preserves same-backend Grok replay while allowing cold cross-backend requests to reach blob validation. Unknown provenance can still be valid after restart, durable storage is unnecessary for this bounded compatibility hint, and deterministic pre-flight avoids the extra paid or stateful upstream attempt whenever the process has evidence. The upstream's narrow error identity supplies authoritative evidence only for histories the process could not observe.
596600
- 장점, 단점 및 영향: Same-route and unknown replay retain cached reasoning on the first send without replaying an output-only field, known cross-route replay keeps the reasoning item without its undecodable blob, and a cold cross-route replay can reach opaque-blob recovery instead of failing early on `status`. A repeated blob rejection is surfaced unchanged after exactly one recovery attempt.
597601

0 commit comments

Comments
 (0)