Skip to content

Commit 854ff4c

Browse files
ZaynJarvisktz03jiale-li-orionclaude
authored
fix(pi-extension): replay offline backlog through the batch endpoint (#4692)
* fix(pi-extension): replay offline backlog through the batch endpoint After a server outage pi's local pending queue can hold hundreds of addMessage entries. The takeover barrier requires that queue to be empty for the session, but replayed it with the shared replayPending(): one POST per entry, at most one replay window (50) per attempt. At the ~0.67s/entry measured in #4504 a 670-entry backlog needed 14 takeover attempts of ~33s each, and pendingTokens kept growing meanwhile. - flushForTakeover drains the session's backlog via /messages/batch (sendSessionMessages, BATCH_LIMIT=100), so the whole backlog clears in a handful of requests within one attempt. Entries are claimed one batch at a time; a failed batch costs a retry only for the entries it contained, the rest stay untouched. - syncBranch sends the whole turn in one batch request instead of one POST per payload, with retryable failures queued as before. - pi's shared dir now vendors batch-send.mjs (already used by the claude-code, codex, opencode and zcode plugins). Fixes #4504 Co-Authored-By: ktz03 <[email protected]> Co-Authored-By: jiale li <[email protected]> Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01Uued6mAGJ4jfTW4XRmNWwL * fix(pi-extension): enqueue non-retryable batch failures and bound drain (#4702) Address #4692 review nits from now-ing and jiale-li-orion: - enqueueRemainder always queues on enqueue-on-failure (incl. 400/403) so the sync watermark advances - drainSessionBacklog soft-bounded by OPENVIKING_PENDING_DRAIN_BUDGET_MS / MAX_BATCHES - tests for watermark enqueue and maxBatches stop * fix(pi-extension): keep batch-send drop policy, advance watermark in pi #4702 made sendSessionMessages enqueue payloads after a non-retryable rejection so pi's sync watermark would advance. That reverses a policy pinned by the shared batch-send tests (poison payloads are dropped, not queued) for every harness, and the other vendored copies were not regenerated. Keep the shared policy and fix the watermark where the need is: pi's sendPayloads counts non-retryable drops as accepted, matching the outcome replayPending() applies to such entries. Co-Authored-By: ktz03 <[email protected]> Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01Uued6mAGJ4jfTW4XRmNWwL --------- Co-authored-by: ktz03 <[email protected]> Co-authored-by: jiale li <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 37db9c8 commit 854ff4c

6 files changed

Lines changed: 373 additions & 18 deletions

File tree

‎examples/memory-plugin-shared/sync.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ const OPENCODE_SHARED_FILES = [...HOOK_SHARED_FILES, ...SETUP_WIZARD_SHARED_FILE
5252
// dsh and zcode ship no setup entry point, so nothing there calls the wizard.
5353
const ZCODE_SHARED_FILES = [...HOOK_SHARED_FILES, ...MCP_PROXY_SHARED_FILES, ...BATCH_SHARED_FILES, ...ASYNC_WRITE_SHARED_FILES, "agent-hook-runtime.mjs", "agent-uri-guard.mjs"];
5454
const DSH_SHARED_FILES = [...HOOK_SHARED_FILES, ...MCP_PROXY_SHARED_FILES];
55-
const PI_SHARED_FILES = [...HOOK_SHARED_FILES, ...SETUP_WIZARD_SHARED_FILES];
55+
const PI_SHARED_FILES = [...HOOK_SHARED_FILES, ...SETUP_WIZARD_SHARED_FILES, ...BATCH_SHARED_FILES];
5656
// Agent Plugins 1.0 has no hooks: it is the proxy and nothing else.
5757
const AGENT_PLUGINS_SHARED_FILES = ["credentials.mjs", "debug-log.mjs", ...MCP_PROXY_SHARED_FILES];
5858
// openclaw assembles recall server-side, so it takes the recall pair alone.
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
export const BATCH_LIMIT: number;
2+
export function sendSessionMessages(
3+
fetchJSON: (path: string, init?: any) => Promise<{ ok: boolean; status?: number; result?: any; error?: any }>,
4+
sessionId: string,
5+
payloads: Array<Record<string, any>>,
6+
opts?: { enqueueOnRetryable?: boolean; onSent?: (count: number) => void | Promise<void> },
7+
): Promise<{
8+
sent: number;
9+
queued: number;
10+
enqueueFailed: number;
11+
failed: number;
12+
retryable: boolean;
13+
usedBatch: boolean;
14+
lastError: any;
15+
}>;

‎examples/pi-coding-agent-extension/shared/batch-send.mjs‎

Lines changed: 121 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/pi-coding-agent-extension/shared/pending-queue.d.mts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,7 @@ export function replayPending(
44
fetchJSON: (path: string, init?: any) => Promise<{ ok: boolean; status?: number; result?: any; error?: any }>,
55
log: (stage: string, data?: any) => void,
66
): Promise<{ replayed: number; failed: number; skipped: number; deferred: number }>;
7+
export function claimForReplay(filename: string): Promise<string | null>;
8+
export function dequeue(filename: string): Promise<boolean>;
9+
export function incrementRetry(filename: string, entry: Record<string, any>): Promise<boolean>;
710
export function cleanStale(): Promise<number>;

‎examples/pi-coding-agent-extension/sync.ts‎

Lines changed: 115 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@ import type { OVClient } from "./client.js";
22
import { createLogger } from "./shared/debug-log.mjs";
33
import type { OVConfig } from "./config.js";
44
import { deriveHarnessSessionId } from "./shared/session-model.mjs";
5-
import { enqueue, listPending, replayPending } from "./shared/pending-queue.mjs";
5+
import { claimForReplay, dequeue, enqueue, incrementRetry, listPending, replayPending } from "./shared/pending-queue.mjs";
6+
import { BATCH_LIMIT, sendSessionMessages } from "./shared/batch-send.mjs";
67
import { extractBranchCapturePayloads } from "./lib/capture-adapter.mjs";
78
import { countUndeliveredForSession, estimatePayloadTokens } from "./lib/takeover-core.mjs";
89

@@ -61,26 +62,101 @@ export class SyncManager {
6162

6263
async flushForTakeover(): Promise<boolean> {
6364
if (!this.ovSessionId) return false;
64-
await this.replayPending();
65+
// Drain is bounded (time / max batches). Remaining undelivered entries —
66+
// including any still claimed as `.processing` — keep the barrier closed
67+
// until a later turn finishes draining them.
68+
if (this.client.connected) await this.drainSessionBacklog();
6569
const pending = await listPending();
6670
return countUndeliveredForSession(pending, this.ovSessionId) === 0;
6771
}
6872

73+
/**
74+
* Replay this session's queued addMessage entries through the batch
75+
* endpoint, BATCH_LIMIT per request. The shared replayPending() sends one
76+
* request per entry and stops after one replay window, which is what let a
77+
* large offline backlog block the takeover barrier for many turns (#4504).
78+
* Entries are claimed one batch at a time so a failed batch only costs a
79+
* retry for the entries it contained.
80+
*
81+
* Bounded per call via OPENVIKING_PENDING_DRAIN_BUDGET_MS (default 60s) and
82+
* optional OPENVIKING_PENDING_DRAIN_MAX_BATCHES so a huge backlog cannot
83+
* block turn_end for an unbounded wall time; remainder drains on later turns.
84+
*/
85+
private async drainSessionBacklog(): Promise<void> {
86+
const sid = this.ovSessionId;
87+
if (!sid) return;
88+
const budgetRaw = Number(process.env.OPENVIKING_PENDING_DRAIN_BUDGET_MS);
89+
const timeBudgetMs = Number.isFinite(budgetRaw) && budgetRaw >= 0 ? budgetRaw : 60_000;
90+
const maxRaw = Number(process.env.OPENVIKING_PENDING_DRAIN_MAX_BATCHES);
91+
const maxBatches =
92+
Number.isFinite(maxRaw) && maxRaw > 0 ? Math.floor(maxRaw) : Number.POSITIVE_INFINITY;
93+
const started = Date.now();
94+
let batches = 0;
95+
96+
const backlog = (await listPending()).filter(
97+
({ entry }) => entry?.type === "addMessage" && entry.sessionId === sid,
98+
);
99+
for (let start = 0; start < backlog.length; start += BATCH_LIMIT) {
100+
if (batches >= maxBatches || Date.now() - started >= timeBudgetMs) {
101+
this.logger.log("drain", {
102+
session: sid,
103+
stopped: batches >= maxBatches ? "max-batches" : "time-budget",
104+
batches,
105+
remaining: backlog.length - start,
106+
elapsedMs: Date.now() - started,
107+
});
108+
return;
109+
}
110+
111+
const claimed: Array<{ filename: string; entry: any }> = [];
112+
for (const { filename, entry } of backlog.slice(start, start + BATCH_LIMIT)) {
113+
const name = await claimForReplay(filename);
114+
if (name) claimed.push({ filename: name, entry });
115+
}
116+
if (claimed.length === 0) continue;
117+
batches += 1;
118+
119+
let delivered = 0;
120+
await sendSessionMessages(
121+
this.fetchJSON,
122+
sid,
123+
claimed.map(({ entry }) => entry.payload),
124+
{
125+
onSent: async (count: number) => {
126+
for (let i = 0; i < count; i++) await dequeue(claimed[delivered++].filename);
127+
},
128+
},
129+
);
130+
if (delivered === claimed.length) continue;
131+
132+
// Undelivered entries stay queued with one more retry; incrementRetry
133+
// drops them once the retry budget is exhausted.
134+
for (const { filename, entry } of claimed.slice(delivered)) {
135+
await incrementRetry(filename, entry);
136+
}
137+
this.logger.log("drain", {
138+
session: sid,
139+
delivered,
140+
retried: claimed.length - delivered,
141+
});
142+
return;
143+
}
144+
}
145+
146+
private fetchJSON = (path: string, init?: any) => this.client.fetchJSON(path, init, 10000);
147+
69148
async syncBranch(branch: any[]): Promise<SyncBranchResult> {
70149
if (!this.ovSessionId) return { added: 0, tokens: 0, allDelivered: true };
71150

72151
const extracted = extractBranchCapturePayloads(branch, this.syncedEntryCount, this.config);
73152
if (extracted.resetWatermark) this.syncedEntryCount = 0;
74-
let added = 0;
153+
const sent = await this.sendPayloads(extracted.payloads);
154+
const added = sent.accepted;
75155
let tokens = 0;
76-
let allDelivered = true;
77-
for (const payload of extracted.payloads) {
78-
const result = await this.addPayload(payload);
79-
if (!result.accepted) break;
80-
added++;
156+
for (const payload of extracted.payloads.slice(0, added)) {
81157
tokens += estimatePayloadTokens(payload);
82-
allDelivered = allDelivered && result.delivered;
83158
}
159+
const allDelivered = sent.delivered === added;
84160
if (added === extracted.payloads.length) {
85161
this.syncedEntryCount = extracted.nextEntryCount;
86162
}
@@ -91,11 +167,36 @@ export class SyncManager {
91167
}
92168

93169
async addPayload(payload: any): Promise<AddPayloadResult> {
94-
if (!this.ovSessionId) return { accepted: false, delivered: false };
95-
const ok = await this.client.addMessagePayload(this.ovSessionId, payload);
96-
if (ok) return { accepted: true, delivered: true };
97-
await enqueue("addMessage", this.ovSessionId, payload);
98-
return { accepted: true, delivered: false };
170+
const sent = await this.sendPayloads([payload]);
171+
return { accepted: sent.accepted === 1, delivered: sent.delivered === 1 };
172+
}
173+
174+
/**
175+
* Send payloads in one batch request; retryable failures are queued to disk.
176+
* Returns how many payloads were accepted (sent or queued, always a prefix)
177+
* and how many of those were delivered to the server.
178+
*/
179+
private async sendPayloads(payloads: any[]): Promise<{ accepted: number; delivered: number }> {
180+
if (!this.ovSessionId || payloads.length === 0) return { accepted: 0, delivered: 0 };
181+
const res = await sendSessionMessages(this.fetchJSON, this.ovSessionId, payloads, {
182+
enqueueOnRetryable: true,
183+
});
184+
if (res.failed > 0 || res.enqueueFailed > 0) {
185+
this.logger.log("send", {
186+
session: this.ovSessionId,
187+
sent: res.sent,
188+
queued: res.queued,
189+
failed: res.failed,
190+
enqueueFailed: res.enqueueFailed,
191+
error: res.lastError?.message || res.lastError?.code || "unknown",
192+
});
193+
}
194+
// A non-retryable rejection (4xx) drops the remaining payloads, the same
195+
// outcome replayPending() applies to such entries. Count them as accepted
196+
// so the watermark still advances past them; otherwise the next turn would
197+
// re-extract and re-send the payloads that were already delivered.
198+
const dropped = res.retryable ? 0 : res.failed;
199+
return { accepted: res.sent + res.queued + dropped, delivered: res.sent };
99200
}
100201

101202
async commitIfNeeded(): Promise<void> {

0 commit comments

Comments
 (0)