Skip to content

Commit 4973cae

Browse files
committed
fix(genkit): demote every gen_ai.cost.* / llm.cost.* key on non-LLM spans
Installed-review N3. isPromotedUsageKey matched only gen_ai.cost.total and llm.cost.total exactly, so cost parts (gen_ai.cost.input/output, llm.cost.prompt/completion) set through fi-core context attributes or Genkit telemetryLabels stayed unprefixed on flow/step/generate/tool spans, where fi-collector would sum them into the trace cost again. Match the cost families by prefix (gen_ai.cost., llm.cost.), like the token families. Model spans keep the keys unprefixed. Tests (written first), mapping.test.ts: - every gen_ai.cost.* / llm.cost.* key is promoted (parts and totals); look-alikes (gen_ai.costs, llm.costing, genkit.usage.*) are not. - cost parts on a flow span, from span and from context attributes, move to genkit.usage.<key>. - a model span keeps them unprefixed (control). RED on b5cb204: the first two failed (2 failed, 37 passed in the file). GREEN: jest 68 passed; contract 20 passed (Node 26.8.1). The contract helper's promoted-key check now uses the same prefixes. README: the promoted-key list names the cost prefixes. approved: Nikhil 2026-10-03 blanket Refs: TH-8240
1 parent b5cb204 commit 4973cae

4 files changed

Lines changed: 42 additions & 6 deletions

File tree

‎typescript/packages/traceai_genkit/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ Metrics: `enableTelemetry({ spanProcessors })` passes no metric reader, and the
9898

9999
Unavailable at 1.42.0, and not set: provider, cost, user id (unless the app sets it), retriever documents, and every `GenerationUsage` field the inventory did not see on a span (`thoughtsTokens`, `cachedContentTokens`, the character / image / video / audio counters, `custom`). Those stay inside `genkit:output`.
100100

101-
Tokens are written only on model call spans. The `generate` span's `genkit:output` repeats the last turn's usage; it stays inside that JSON and is not promoted. If any non-model span carries a key the collector promotes (`gen_ai.usage.*`, `llm.token_count.*`, `llm.usage.*`, `gen_ai.cost.total`, `llm.cost.total`), the processor moves it to `genkit.usage.<key>`, so the trace-wide token sum equals the model calls.
101+
Tokens are written only on model call spans. The `generate` span's `genkit:output` repeats the last turn's usage; it stays inside that JSON and is not promoted. If any non-model span carries a key the collector promotes (`gen_ai.usage.*`, `llm.token_count.*`, `llm.usage.*`, `gen_ai.cost.*`, `llm.cost.*`; the cost prefixes cover the parts such as `gen_ai.cost.input` and `llm.cost.completion` as well as the totals), the processor moves it to `genkit.usage.<key>`, so the trace-wide token and cost sums equal the model calls.
102102

103103
Errors: Genkit sets the OTel status to `ERROR` and records an `exception` event on every span the error passes through (`core/src/tracing/instrumentation.ts:153-172`). Both are exported; with content off, schema-validation data is cut from them (see [Content](#content)). `genkit:isFailureSource` marks the first failing span.
104104

‎typescript/packages/traceai_genkit/contract/test_harness_contract.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,8 @@
6565
"gen_ai.usage.input_tokens",
6666
"gen_ai.usage.output_tokens",
6767
"gen_ai.usage.total_tokens",
68-
"gen_ai.cost.total",
69-
"llm.cost.total",
7068
}
71-
PROMOTED_PREFIXES = ("gen_ai.usage.", "llm.token_count.", "llm.usage.")
69+
PROMOTED_PREFIXES = ("gen_ai.usage.", "llm.token_count.", "llm.usage.", "gen_ai.cost.", "llm.cost.")
7270
# Inventory at genkit 1.42.0 (contract/inventory.mjs): mockModel spans carry
7371
# usage.inputTokens/outputTokens/totalTokens in genkit:output; only those are
7472
# mapped. thoughtsTokens and cachedContentTokens exist in GenerationUsageSchema

‎typescript/packages/traceai_genkit/src/__tests__/mapping.test.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,42 @@ describe("model call spans", () => {
188188
expect(isPromotedUsageKey(key)).toBe(false);
189189
}
190190
});
191+
192+
it("treats every gen_ai.cost.* and llm.cost.* key as promoted, cost parts included", () => {
193+
// fi-collector promotes the cost parts too when no total is present (DeriveHotKeys).
194+
for (const key of [
195+
"gen_ai.cost.input",
196+
"gen_ai.cost.output",
197+
"gen_ai.cost.total",
198+
"llm.cost.prompt",
199+
"llm.cost.completion",
200+
"llm.cost.total",
201+
"gen_ai.cost.some_future_part",
202+
]) {
203+
expect(isPromotedUsageKey(key)).toBe(true);
204+
}
205+
for (const key of ["genkit.usage.gen_ai.cost.input", "gen_ai.costs", "llm.costing", "cost.total"]) {
206+
expect(isPromotedUsageKey(key)).toBe(false);
207+
}
208+
});
209+
210+
it("moves cost parts on a non-model span to genkit.usage.*, from span and context attributes", () => {
211+
const out = mapGenkitAttributes(
212+
{ ...flowAttributes(), "gen_ai.cost.input": 0.01, "llm.cost.completion": 0.02 },
213+
{ contextAttributes: { "gen_ai.cost.output": 0.03, "llm.cost.prompt": 0.04 } },
214+
);
215+
expect(Object.keys(out).filter((k) => k.startsWith("gen_ai.cost.") || k.startsWith("llm.cost."))).toEqual([]);
216+
expect(out["genkit.usage.gen_ai.cost.input"]).toBe(0.01);
217+
expect(out["genkit.usage.llm.cost.completion"]).toBe(0.02);
218+
expect(out["genkit.usage.gen_ai.cost.output"]).toBe(0.03);
219+
expect(out["genkit.usage.llm.cost.prompt"]).toBe(0.04);
220+
});
221+
222+
it("keeps cost parts unprefixed on a model span", () => {
223+
const out = mapGenkitAttributes(modelAttributes(), { contextAttributes: { "gen_ai.cost.input": 0.05 } });
224+
expect(out["gen_ai.cost.input"]).toBe(0.05);
225+
expect(out["genkit.usage.gen_ai.cost.input"]).toBeUndefined();
226+
});
191227
});
192228

193229
describe("tool spans", () => {

‎typescript/packages/traceai_genkit/src/attributes.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,8 @@ export const GENKIT_USAGE_PREFIX = "genkit.usage.";
172172

173173
/**
174174
* True for keys fi-collector promotes into token/cost columns on any span kind.
175+
* Cost is matched by prefix: with no total present the collector also promotes
176+
* the cost parts (`gen_ai.cost.input`/`output`, `llm.cost.prompt`/`completion`).
175177
*/
176178
export function isPromotedUsageKey(key: string): boolean {
177179
return (
@@ -181,7 +183,7 @@ export function isPromotedUsageKey(key: string): boolean {
181183
key.startsWith("gen_ai.usage.") ||
182184
key.startsWith("llm.token_count.") ||
183185
key.startsWith("llm.usage.") ||
184-
key === "gen_ai.cost.total" ||
185-
key === "llm.cost.total"
186+
key.startsWith("gen_ai.cost.") ||
187+
key.startsWith("llm.cost.")
186188
);
187189
}

0 commit comments

Comments
 (0)