Skip to content

Commit 1e28e62

Browse files
authored
fix(devin): read usage from ModelUsageStats and classify cloud failures (#4419)
* fix(devin): read usage from ModelUsageStats and classify cloud failures A cached Devin turn reported a bare token total with no cached subset, so its log row looked like a smaller request than it was. The decoder was reading GetChatMessageResponse field 28. Field 28 is response_dimension_groups, the rows the IDE renders; field 7 is ModelUsageStats, the per-turn accounting. The old path worked by accident: ResponseDimension.uid is that message's field 5, which the entry walker treats as a metric id, so cache numbers appeared only when the service happened to render cache rows. Field 7 carries cache read and cache write unconditionally. Both fields arrive in the same message and the adapter keeps the last usage event, so decoding both is not enough: field 7 now suppresses field 28 within a message and is yielded last, and it needs its own uint64 decoder because the field-28 walker reads a fixed32 float out of a sub-message. Whether Cognition's input_tokens already includes cache is unsettled, and guessing inclusive is the expensive error: normalizeCostTokens only rejects read + write > input, so an inflated input passes validation and bills cached tokens at the uncached rate. The mapping is therefore derived from the frame. Both branches agree on the 58k-prompt case that prompted this. Two further classification defects. CloudChatError carried no HTTP status, so inferHttpStatusFromAdapterMessage turned an upstream 429 into a 502 and core's failover never rotated or backed off. And a cancelled turn said "Devin turn was aborted.", which isClientClosedMessage does not recognise, so a client hanging up was logged as an upstream failure; it now emits the phrase the classifier knows, with status 499. Usage frames are merged per field instead of replaced, because the counters are cumulative and a later partial frame used to zero an earlier count. * fix(devin): report field-7 usage ahead of finish and derive the merged total Review follow-ups on the usage decode. The authoritative ModelUsageStats event was yielded after the rest of the frame, so a frame that also carried finish reported usage behind the turn's end. It is now yielded first, which makes the order independent of where the service places the field. mergeDevinUsage took the max of two totals alongside the per-field maxima, which can leave totalTokens different from input + output; the cost and log paths read that total. The total is now derived from the merged counts. Regression coverage for what the change is actually for: field 7 suppressing the display rows within one frame and landing before finish, the display rows still decoding when no field 7 is present, a partial frame not zeroing an earlier count, and an HTTP status becoming a structured classification. A Connect trailer still carries no HTTP status, so a cap delivered that way keeps the older message-inference path. That is noted at the throw site as a follow-up rather than silently left open.
1 parent 7b3c4e9 commit 1e28e62

3 files changed

Lines changed: 287 additions & 9 deletions

File tree

‎src/adapters/devin.ts‎

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,53 @@ import { getCachedCatalog } from "./devin/cloud-direct/catalog";
1313
import { buildNonOpenAIToolCatalogNudgeForTools } from "./tool-catalog-nudge";
1414
import { DEVIN_DEFAULT_API_SERVER, resolveDevinApiServer } from "../oauth/devin";
1515

16+
/**
17+
* Combine two usage frames from one turn by keeping the larger count per field.
18+
*
19+
* Devin's counters are cumulative within a turn, so a frame that reports less
20+
* than an earlier one is reporting a subset, not a correction.
21+
*/
22+
export function mergeDevinUsage(previous: OcxUsage, next: OcxUsage): OcxUsage {
23+
const keys = [
24+
"inputTokens", "outputTokens",
25+
"cachedInputTokens", "cacheReadInputTokens", "cacheCreationInputTokens",
26+
"reasoningOutputTokens",
27+
] as const;
28+
const merged: OcxUsage = { ...previous, ...next };
29+
for (const key of keys) {
30+
const a = previous[key];
31+
const b = next[key];
32+
if (typeof a === "number" && typeof b === "number") merged[key] = Math.max(a, b);
33+
else if (typeof a === "number" && b === undefined) merged[key] = a;
34+
}
35+
// totalTokens is derived, not merged. Taking the max of two totals alongside
36+
// per-field maxima can leave total !== input + output, and the cost and log
37+
// paths read the total.
38+
const total = (merged.inputTokens ?? 0) + (merged.outputTokens ?? 0);
39+
if (total > 0) merged.totalTokens = total;
40+
return merged;
41+
}
42+
43+
/**
44+
* The wording `isClientClosedMessage` recognises.
45+
*
46+
* "Devin turn was aborted." matched nothing, so a cancelled turn fell through to
47+
* the default inference and was logged as a 502 upstream failure rather than as
48+
* the client hanging up.
49+
*/
50+
const DEVIN_CLIENT_CLOSED_MESSAGE = "client closed request";
51+
52+
/** Map a cloud-direct failure onto the structured fields the error event carries. */
53+
export function devinErrorClassification(error: unknown): { status?: number; errorType?: string; retryable?: boolean } {
54+
const status = error instanceof CloudChatError ? error.status : undefined;
55+
if (status === undefined) return {};
56+
if (status === 401) return { status, errorType: "authentication_error", retryable: false };
57+
if (status === 403) return { status, errorType: "permission_error", retryable: false };
58+
if (status === 429) return { status, errorType: "rate_limit_error", retryable: true };
59+
if (status >= 500) return { status, retryable: true };
60+
return { status, retryable: false };
61+
}
62+
1663
export const DEVIN_API_SERVER = DEVIN_DEFAULT_API_SERVER;
1764

1865
const EFFORT_SUFFIXES = new Set(["low", "medium", "high", "xhigh", "max", "none", "1m", "max-1m", "none-1m", "fast"]);
@@ -210,7 +257,7 @@ export function createDevinAdapter(
210257

211258
async runTurn(parsed: OcxParsedRequest, incoming: IncomingMeta, emit: (event: AdapterEvent) => void) {
212259
if (incoming.abortSignal?.aborted) {
213-
emit({ type: "error", message: "Devin turn was aborted before start." });
260+
emit({ type: "error", message: DEVIN_CLIENT_CLOSED_MESSAGE, status: 499, retryable: false });
214261
return;
215262
}
216263
let apiKey: string;
@@ -272,7 +319,7 @@ export function createDevinAdapter(
272319
// Say what happened instead, the way the other runTurn-only adapter
273320
// does, and carry any usage already seen.
274321
closeOpenTool();
275-
emit({ type: "error", message: "Devin turn was aborted.", ...(usage ? { usage } : {}) });
322+
emit({ type: "error", message: DEVIN_CLIENT_CLOSED_MESSAGE, status: 499, retryable: false, ...(usage ? { usage } : {}) });
276323
return;
277324
}
278325
if (event.kind === "text") {
@@ -305,35 +352,46 @@ export function createDevinAdapter(
305352
}
306353
if (event.kind === "usage") {
307354
const total = event.totalTokens ?? ((event.promptTokens ?? 0) + (event.completionTokens ?? 0));
308-
usage = {
355+
const next: OcxUsage = {
309356
inputTokens: event.promptTokens ?? 0,
310357
outputTokens: event.completionTokens ?? 0,
311358
...(total > 0 ? { totalTokens: total } : {}),
312359
...(event.cachedInputTokens !== undefined ? { cachedInputTokens: event.cachedInputTokens } : {}),
313360
...(event.cacheCreationInputTokens !== undefined ? { cacheCreationInputTokens: event.cacheCreationInputTokens } : {}),
314361
...(event.reasoningTokens !== undefined ? { reasoningOutputTokens: event.reasoningTokens } : {}),
315362
};
363+
// Merge rather than replace. A turn can carry more than one usage
364+
// frame, and the counters are cumulative, so a later partial frame
365+
// that omits a field used to zero a count the earlier frame had
366+
// already reported.
367+
usage = usage ? mergeDevinUsage(usage, next) : next;
316368
continue;
317369
}
318370
}
319371
closeOpenTool();
320372
if (incoming.abortSignal?.aborted) {
321-
emit({ type: "error", message: "Devin turn was aborted.", ...(usage ? { usage } : {}) });
373+
emit({ type: "error", message: DEVIN_CLIENT_CLOSED_MESSAGE, status: 499, retryable: false, ...(usage ? { usage } : {}) });
322374
} else {
323375
emit({ type: "done", ...(usage ? { usage } : {}), ...(stopReason ? { stopReason } : {}) });
324376
}
325377
} catch (error) {
326378
closeOpenTool();
327379
if (incoming.abortSignal?.aborted) {
328-
emit({ type: "error", message: "Devin turn was aborted.", ...(usage ? { usage } : {}) });
380+
emit({ type: "error", message: DEVIN_CLIENT_CLOSED_MESSAGE, status: 499, retryable: false, ...(usage ? { usage } : {}) });
329381
return;
330382
}
331383
const message = error instanceof CloudChatError
332384
? ("Devin cloud error" + (error.code ? " " + error.code : "") + ": " + error.message)
333385
: error instanceof Error ? error.message : String(error);
334386
// Usage that already arrived is still real; dropping it loses the
335387
// accounting for a turn that did most of its work before failing.
336-
emit({ type: "error", message, ...(usage ? { usage } : {}) });
388+
emit({
389+
type: "error",
390+
message,
391+
...devinErrorClassification(error),
392+
...(error instanceof CloudChatError && error.code ? { code: error.code } : {}),
393+
...(usage ? { usage } : {}),
394+
});
337395
}
338396
},
339397
};

‎src/adapters/devin/cloud-direct/chat.ts‎

Lines changed: 105 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -669,7 +669,30 @@ function buildGetChatMessageRequest(args: BuildArgs): Buffer {
669669
* any non-zero to 'tool_calls' for now (and let the caller fall back to
670670
* 'stop' if no tool_call deltas were emitted).
671671
*/
672-
function* decodeChatFrame(proto: Buffer): Generator<CloudChatEvent> {
672+
export function* decodeChatFrame(proto: Buffer): Generator<CloudChatEvent> {
673+
// Field 7 is `ModelUsageStats`, the authoritative per-turn accounting, and
674+
// field 28 is `response_dimension_groups` — the rows the IDE renders. The
675+
// decoder below reads 28 because a capture happened to expose metric-looking
676+
// strings there (`ResponseDimension.uid` is its field 5, which is what the
677+
// entry walker treats as `metric_id`), and that works only when the service
678+
// chose to render cache rows. Field 7 carries cache read and cache write
679+
// unconditionally, which is why a cached Devin turn used to report a bare
680+
// total with no cached subset.
681+
//
682+
// Both fields arrive in the same message and the adapter keeps the last usage
683+
// event it sees, so this cannot be a plain "decode both": field 7 has to
684+
// suppress field 28 within the message. It is yielded before the rest of the
685+
// frame rather than after it, so a frame that also carries finish (field 5)
686+
// still reports usage ahead of the turn's end, and the order does not depend
687+
// on where the service happens to place the field.
688+
let authoritativeUsage: CloudChatEvent | null = null;
689+
for (const f of iterFields(proto)) {
690+
if (f.num === 7 && f.wire === 2 && Buffer.isBuffer(f.value)) {
691+
authoritativeUsage = decodeModelUsageStats(f.value as Buffer);
692+
if (authoritativeUsage) break;
693+
}
694+
}
695+
if (authoritativeUsage) yield authoritativeUsage;
673696
for (const f of iterFields(proto)) {
674697
if (f.num === 3 && f.wire === 2 && Buffer.isBuffer(f.value)) {
675698
// Visible delta_text — what the user should SEE in the chat.
@@ -740,6 +763,7 @@ function* decodeChatFrame(proto: Buffer): Generator<CloudChatEvent> {
740763
// else stays 'stop' for 0/2/4-9/12/13
741764
yield { kind: 'finish', reason };
742765
} else if (f.num === 28 && f.wire === 2 && Buffer.isBuffer(f.value)) {
766+
if (authoritativeUsage) continue;
743767
const usage = decodeUsageBlock(f.value as Buffer);
744768
if (usage) yield usage;
745769
}
@@ -834,6 +858,68 @@ function decodeUsageBlock(buf: Buffer): CloudChatEvent | null {
834858
};
835859
}
836860

861+
/**
862+
* `exa.codeium_common_pb.ModelUsageStats` at GetChatMessageResponse field 7.
863+
*
864+
* ModelUsageStats {
865+
* #2 input_tokens uint64
866+
* #3 output_tokens uint64
867+
* #4 cache_write_tokens uint64
868+
* #5 cache_read_tokens uint64
869+
* }
870+
*
871+
* Plain varints, so the field-28 entry walker — which descends a
872+
* length-delimited sub-message and reads a fixed32 float — cannot read this at
873+
* all. It needs its own decoder.
874+
*
875+
* Whether Cognition's `input_tokens` already includes the cached tokens is not
876+
* settled. oh-my-pi sums all four into its total, which suggests exclusive, but
877+
* that is their convention rather than a measurement of this field. Guessing
878+
* wrong in the inclusive direction is the expensive mistake: `normalizeCostTokens`
879+
* only rejects `read + write > input`, so an inflated input passes validation and
880+
* bills cached tokens at the uncached rate.
881+
*
882+
* So the shape is derived from the frame instead of assumed. An input that
883+
* already covers the cache is left alone; one that cannot possibly cover it is
884+
* folded. Both branches agree on the case that motivated this — a 58k prompt
885+
* that is 57k cache read and 1k fresh reads as 58k with a 57k cached subset —
886+
* and neither can emit `read + write > input`. Replace the derivation with a
887+
* fixed mapping once a live frame settles the question.
888+
*/
889+
export function decodeModelUsageStats(buf: Buffer): CloudChatEvent | null {
890+
let wireInput: number | undefined;
891+
let output: number | undefined;
892+
let cacheWrite: number | undefined;
893+
let cacheRead: number | undefined;
894+
for (const f of iterFields(buf)) {
895+
if (f.wire !== 0) continue;
896+
const n = Number(f.value);
897+
if (!Number.isFinite(n) || n < 0) continue;
898+
if (f.num === 2) wireInput = n;
899+
else if (f.num === 3) output = n;
900+
else if (f.num === 4) cacheWrite = n;
901+
else if (f.num === 5) cacheRead = n;
902+
}
903+
if (wireInput === undefined && output === undefined && cacheRead === undefined && cacheWrite === undefined) {
904+
return null;
905+
}
906+
const read = cacheRead ?? 0;
907+
const write = cacheWrite ?? 0;
908+
const rawInput = wireInput ?? 0;
909+
const promptTokens = rawInput >= read + write ? rawInput : rawInput + read + write;
910+
const completionTokens = output ?? 0;
911+
const total = promptTokens + completionTokens;
912+
return {
913+
kind: 'usage',
914+
promptTokens,
915+
completionTokens,
916+
totalTokens: total > 0 ? total : undefined,
917+
cachedInputTokens: cacheRead,
918+
cacheCreationInputTokens: cacheWrite,
919+
reasoningTokens: undefined,
920+
};
921+
}
922+
837923
// ----------------------------------------------------------------------------
838924
// Public API: streamChat
839925
// ----------------------------------------------------------------------------
@@ -864,7 +950,18 @@ export interface CloudChatRequest {
864950
}
865951

866952
export class CloudChatError extends Error {
867-
constructor(message: string, public readonly code?: string, public readonly traceId?: string) {
953+
constructor(
954+
message: string,
955+
public readonly code?: string,
956+
public readonly traceId?: string,
957+
/**
958+
* Upstream HTTP status, when the failure was a status line rather than a
959+
* Connect trailer. Without it the adapter's message reaches
960+
* `inferHttpStatusFromAdapterMessage`, which does not parse `HTTP 429`, so
961+
* a live rate limit was classified 502 and core's failover never rotated.
962+
*/
963+
public readonly status?: number,
964+
) {
868965
super(message);
869966
this.name = 'CloudChatError';
870967
}
@@ -987,7 +1084,12 @@ export async function* streamChatEvents(req: CloudChatRequest): AsyncGenerator<C
9871084
// The body is not echoed into the message. This error reaches the adapter's
9881085
// error event and /api/logs, and a Connect error can quote the request that
9891086
// produced it - which is the request holding the api_key.
990-
throw new CloudChatError(`GetChatMessage failed (HTTP ${resp.status})`, undefined);
1087+
//
1088+
// Only the HTTP status line is carried here. A Connect EOS trailer that
1089+
// reports resource_exhausted or unavailable still arrives without a status,
1090+
// so a cap delivered that way keeps the older message-inference path.
1091+
// Mapping trailer codes onto HTTP statuses is deliberately a follow-up.
1092+
throw new CloudChatError(`GetChatMessage failed (HTTP ${resp.status})`, undefined, undefined, resp.status);
9911093
}
9921094
if (!resp.body) {
9931095
throw new CloudChatError('GetChatMessage response had no body stream');

‎tests/providers/devin-hardening.test.ts‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ import { DEVIN_DEFAULT_API_SERVER, resolveDevinApiBaseUrl, validateDevinApiBaseU
55
import { registerUser } from "../../src/oauth/devin/register-user";
66
import { anySignal } from "../../src/lib/abort";
77
import { buildGetChatMessageRequestForTests } from "../../src/adapters/devin/cloud-direct/chat";
8+
import { decodeModelUsageStats } from "../../src/adapters/devin/cloud-direct/chat";
9+
import { CloudChatError, decodeChatFrame } from "../../src/adapters/devin/cloud-direct/chat";
10+
import { devinErrorClassification, mergeDevinUsage } from "../../src/adapters/devin";
811
import { iterFields } from "../../src/adapters/devin/cloud-direct/wire";
912
import { buildMetadata, normalizeDevinSessionToken } from "../../src/adapters/devin/cloud-direct/metadata";
1013

@@ -280,3 +283,118 @@ describe("devin session-token normalization", () => {
280283
}
281284
});
282285
});
286+
287+
describe("devin ModelUsageStats decode (response field 7)", () => {
288+
function varint(num: number, value: number): Buffer {
289+
const out: number[] = [(num << 3) | 0];
290+
let v = value;
291+
do { const b = v & 0x7f; v = Math.floor(v / 128); out.push(v > 0 ? b | 0x80 : b); } while (v > 0);
292+
return Buffer.from(out);
293+
}
294+
const stats = (input: number, output: number, write: number, read: number) =>
295+
Buffer.concat([varint(2, input), varint(3, output), varint(4, write), varint(5, read)]);
296+
297+
test("an exclusive frame folds cache into the inclusive input this repo reports", () => {
298+
// 1k fresh + 57k cache read is the 58k prompt the user sees as one number.
299+
const u = decodeModelUsageStats(stats(1_000, 200, 0, 57_000));
300+
expect(u?.promptTokens).toBe(58_000);
301+
expect(u?.cachedInputTokens).toBe(57_000);
302+
expect(u?.totalTokens).toBe(58_200);
303+
});
304+
305+
test("an already-inclusive frame is left alone rather than inflated", () => {
306+
const u = decodeModelUsageStats(stats(58_000, 200, 0, 57_000));
307+
expect(u?.promptTokens).toBe(58_000);
308+
expect(u?.cachedInputTokens).toBe(57_000);
309+
// normalizeCostTokens only rejects read + write > input, so an inflated
310+
// input would pass validation and bill cache at the uncached rate.
311+
expect(u!.cachedInputTokens! + (u!.cacheCreationInputTokens ?? 0)).toBeLessThanOrEqual(u!.promptTokens!);
312+
});
313+
314+
test("cache write counts as prompt too, and an empty message decodes to nothing", () => {
315+
const u = decodeModelUsageStats(stats(1_000, 0, 4_000, 0));
316+
expect(u?.promptTokens).toBe(5_000);
317+
expect(u?.cacheCreationInputTokens).toBe(4_000);
318+
expect(decodeModelUsageStats(Buffer.alloc(0))).toBeNull();
319+
});
320+
});
321+
322+
describe("devin frame-level usage precedence and classification", () => {
323+
// Tags above 15 need a multi-byte varint: field 28 wire 2 is 226, and
324+
// writing that as one raw byte sets the continuation bit and swallows the
325+
// next byte.
326+
function uvarint(value: number): number[] {
327+
const out: number[] = [];
328+
let v = value;
329+
do { const b = v & 0x7f; v = Math.floor(v / 128); out.push(v > 0 ? b | 0x80 : b); } while (v > 0);
330+
return out;
331+
}
332+
function varint(num: number, value: number): Buffer {
333+
return Buffer.from([...uvarint((num << 3) | 0), ...uvarint(value)]);
334+
}
335+
function lenDelim(num: number, payload: Buffer): Buffer {
336+
return Buffer.concat([Buffer.from([...uvarint((num << 3) | 2), ...uvarint(payload.length)]), payload]);
337+
}
338+
// ResponseDimensionGroup carrying a cumulative metric whose uid reads like a
339+
// metric id — the shape the old decoder mined for usage.
340+
function displayGroup(uid: string, value: number): Buffer {
341+
const f32 = Buffer.alloc(5);
342+
f32.writeUInt8((2 << 3) | 5, 0);
343+
f32.writeFloatLE(value, 1);
344+
const entry = Buffer.concat([lenDelim(4, f32), lenDelim(5, Buffer.from(uid, "utf8"))]);
345+
return lenDelim(2, entry);
346+
}
347+
348+
test("field 7 suppresses the display rows and is reported before finish", () => {
349+
const stats = Buffer.concat([varint(2, 1_000), varint(3, 200), varint(4, 0), varint(5, 57_000)]);
350+
const frame = Buffer.concat([
351+
lenDelim(7, stats),
352+
varint(5, 2), // stop_reason STOP_PATTERN
353+
lenDelim(28, displayGroup("input_tokens", 999)), // the wrong, display-derived number
354+
]);
355+
const events = [...decodeChatFrame(frame)];
356+
const usages = events.filter(e => e.kind === "usage");
357+
expect(usages).toHaveLength(1);
358+
expect(usages[0]!.promptTokens).toBe(58_000);
359+
expect(usages[0]!.cachedInputTokens).toBe(57_000);
360+
// Ahead of finish, so ordering does not depend on where the service puts
361+
// the field.
362+
expect(events.findIndex(e => e.kind === "usage"))
363+
.toBeLessThan(events.findIndex(e => e.kind === "finish"));
364+
});
365+
366+
test("a frame with no field 7 still falls back to the display rows", () => {
367+
const frame = lenDelim(28, Buffer.concat([
368+
displayGroup("input_tokens", 4_000),
369+
displayGroup("output_tokens", 100),
370+
]));
371+
const usages = [...decodeChatFrame(frame)].filter(e => e.kind === "usage");
372+
expect(usages).toHaveLength(1);
373+
expect(usages[0]!.promptTokens).toBe(4_000);
374+
});
375+
});
376+
377+
describe("devin usage merging and error classification", () => {
378+
test("a later partial frame cannot zero an earlier count, and the total stays derived", () => {
379+
const merged = mergeDevinUsage(
380+
{ inputTokens: 58_000, outputTokens: 200, totalTokens: 58_200, cachedInputTokens: 57_000 },
381+
{ inputTokens: 58_000, outputTokens: 900 },
382+
);
383+
expect(merged.cachedInputTokens).toBe(57_000);
384+
expect(merged.outputTokens).toBe(900);
385+
// Taking the max of two totals alongside per-field maxima would leave
386+
// 58,200 here, which no longer equals input + output.
387+
expect(merged.totalTokens).toBe(58_900);
388+
});
389+
390+
test("an HTTP status on the cloud error becomes a structured classification", () => {
391+
expect(devinErrorClassification(new CloudChatError("x", undefined, undefined, 429)))
392+
.toEqual({ status: 429, errorType: "rate_limit_error", retryable: true });
393+
expect(devinErrorClassification(new CloudChatError("x", undefined, undefined, 401)))
394+
.toEqual({ status: 401, errorType: "authentication_error", retryable: false });
395+
expect(devinErrorClassification(new CloudChatError("x", undefined, undefined, 503)))
396+
.toEqual({ status: 503, retryable: true });
397+
// A Connect trailer carries no status, so it keeps the older inference path.
398+
expect(devinErrorClassification(new CloudChatError("x", "resource_exhausted"))).toEqual({});
399+
});
400+
});

0 commit comments

Comments
 (0)