Skip to content

Commit 33be4ef

Browse files
nik13claude
andcommitted
fix(genkit): bare model name and provider on model spans so Future AGI prices them
gen_ai.request.model carried the registered action name ("googleai/gemini-3.5-flash"). fi-collector prices a span by an exact model-name lookup in its litellm table, which lists "gemini-3.5-flash", so every Genkit model span showed cost 0 and no provider. Model spans now get the vendor model id without the plugin namespace on gen_ai.request.model and gen_ai.response.model, and gen_ai.provider.name from the namespace for the model plugins whose namespaces were read from their published packages (googleai -> google, vertexai -> vertexai, openai, deepseek, xai, anthropic, ollama, mistral -> mistralai, groq, aws-bedrock -> aws, azure-openai -> azure). Other namespaces leave the provider unset. The full registered name stays on genkit:name. Live check with gemini-3.5-flash: model gemini-3.5-flash, provider google, cost 0.000234 / 0.000357 / 0.000147 in ClickHouse and the UI (was 0). README: replace the retired gemini-2.5-flash example, document the provider map and how cost is computed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 53d9c8f commit 33be4ef

6 files changed

Lines changed: 98 additions & 8 deletions

File tree

‎typescript/packages/traceai_genkit/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,8 @@ Cost: the Future AGI export adds no Google Cloud cost. The Firebase / Google Clo
9696
| Future AGI key | Source | Where |
9797
|---|---|---|
9898
| `fi.span.kind`, `gen_ai.span.kind` | table above | every Genkit span |
99-
| `gen_ai.request.model` | `genkit:name` of the model action span (the registered name, e.g. `googleai/gemini-2.5-flash`; not parsed) | model spans |
99+
| `gen_ai.request.model`, `gen_ai.response.model` | `genkit:name` of the model action span without its plugin namespace: `googleai/gemini-3.5-flash` becomes `gemini-3.5-flash`, the name Future AGI's price table uses. The full registered name stays on `genkit:name`. Genkit reports no resolved model version, so both keys hold the requested model | model spans |
100+
| `gen_ai.provider.name` | the plugin namespace: `googleai` → `google`, `vertexai` → `vertexai`, `openai` → `openai`, `anthropic` → `anthropic`, `deepseek` → `deepseek`, `xai` → `xai`, `ollama` → `ollama`, `mistral` → `mistralai`, `groq` → `groq`, `aws-bedrock` → `aws`, `azure-openai` → `azure`. Any other namespace: not set | model spans |
100101
| `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` | `usage.inputTokens` / `outputTokens` / `totalTokens` in the model span's `genkit:output` | model spans only |
101102
| `gen_ai.usage.output_tokens.reasoning` / `gen_ai.usage.cache_read_tokens` | `usage.thoughtsTokens` / `cachedContentTokens`, when the model plugin reports them (the Google AI / Vertex AI plugin does; Gemini's `outputTokens` excludes thinking tokens, which are in `totalTokens`) | model spans only |
102103
| `gen_ai.response.finish_reasons` | `finishReason` in the model span's `genkit:output` | model spans |
@@ -106,7 +107,7 @@ Cost: the Future AGI export adds no Google Cloud cost. The Firebase / Google Clo
106107
| `input.value` / `output.value` (+ `application/json` mime types) | `genkit:input` / `genkit:output` | only with `captureContent: true` |
107108
| `genkit:*` | passed through, except the content keys below | every Genkit span |
108109

109-
Unavailable at 1.42.0, and not set: provider, cost, user id (unless the app sets it), retriever documents, and the other `GenerationUsage` fields (the character / image / video / audio counters, `custom`). Those stay inside `genkit:output`.
110+
Cost is not set by this package: fi-collector prices each model span from its tokens and `gen_ai.request.model` (an exact lookup in its litellm price table), so a model the table lists, such as `gemini-3.5-flash`, gets a cost and one it does not list shows none. Unavailable at 1.42.0, and not set: user id (unless the app sets it), retriever documents, and the other `GenerationUsage` fields (the character / image / video / audio counters, `custom`). Those stay inside `genkit:output`.
110111

111112
Tokens are written only on model call spans. The `generate` span's `genkit:output` repeats the last turn's usage; it stays inside that JSON and is not promoted. If any non-model span carries a key the collector promotes (`gen_ai.usage.*`, `llm.token_count.*`, `llm.usage.*`, `gen_ai.cost.*`, `llm.cost.*`; the cost prefixes cover the parts such as `gen_ai.cost.input` and `llm.cost.completion` as well as the totals), the processor moves it to `genkit.usage.<key>`, so the trace-wide token and cost sums equal the model calls.
112113

‎typescript/packages/traceai_genkit/contract/test_harness_contract.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -307,7 +307,9 @@ def kinds(span: Dict[str, Any]) -> tuple:
307307
assert usage == [(11, 3, 14), (21, 4, 25)]
308308
for model in models:
309309
a = _attrs(model)
310-
assert a["gen_ai.request.model"] == "contract/tool-model"
310+
assert a["gen_ai.request.model"] == "tool-model"
311+
assert "gen_ai.provider.name" not in a # "contract" is not a known model plugin
312+
assert a["genkit:name"] == "contract/tool-model"
311313
assert a["gen_ai.response.finish_reasons"] == ["stop"]
312314
for key in UNAVAILABLE_ON_FIXTURE:
313315
assert key not in a

‎typescript/packages/traceai_genkit/src/__tests__/mapping.test.ts‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ describe("genkitSpanKind: kind map from the 1.42.0 type-string inventory", () =>
9292
describe("model call spans", () => {
9393
it("copies model name, usage and finish reason from the model action span", () => {
9494
const out = mapGenkitAttributes(modelAttributes());
95-
expect(out["gen_ai.request.model"]).toBe("fixture/mock");
95+
expect(out["gen_ai.request.model"]).toBe("mock");
9696
expect(out["gen_ai.usage.input_tokens"]).toBe(11);
9797
expect(out["gen_ai.usage.output_tokens"]).toBe(3);
9898
expect(out["gen_ai.usage.total_tokens"]).toBe(14);
@@ -147,12 +147,52 @@ describe("model call spans", () => {
147147
expect(GENKIT_USAGE_NOT_MAPPED).not.toContain("cachedContentTokens");
148148
});
149149

150+
it.each([
151+
["googleai/gemini-3.5-flash", "gemini-3.5-flash", "google"],
152+
["vertexai/gemini-3.5-flash", "gemini-3.5-flash", "vertexai"],
153+
["vertexai/claude-sonnet-5", "claude-sonnet-5", "vertexai"],
154+
["openai/gpt-4o-mini", "gpt-4o-mini", "openai"],
155+
["anthropic/claude-haiku-4-5", "claude-haiku-4-5", "anthropic"],
156+
["deepseek/deepseek-chat", "deepseek-chat", "deepseek"],
157+
["xai/grok-4.5", "grok-4.5", "xai"],
158+
["ollama/llama3.2", "llama3.2", "ollama"],
159+
["mistral/mistral-large-latest", "mistral-large-latest", "mistralai"],
160+
["groq/llama-3.3-70b-versatile", "llama-3.3-70b-versatile", "groq"],
161+
["aws-bedrock/amazon.nova-pro-v1:0", "amazon.nova-pro-v1:0", "aws"],
162+
["azure-openai/gpt-4o", "gpt-4o", "azure"],
163+
])("%s: model %s, provider %s (the collector prices by the bare model name)", (genkitName, model, provider) => {
164+
const out = mapGenkitAttributes({ ...modelAttributes(), "genkit:name": genkitName });
165+
expect(out["gen_ai.request.model"]).toBe(model);
166+
expect(out["gen_ai.response.model"]).toBe(model);
167+
expect(out["gen_ai.provider.name"]).toBe(provider);
168+
// The full registered name stays on the passed-through Genkit key.
169+
expect(out["genkit:name"]).toBe(genkitName);
170+
});
171+
172+
it("strips an unknown plugin namespace but leaves the provider unset", () => {
173+
const out = mapGenkitAttributes({ ...modelAttributes(), "genkit:name": "together/meta-llama/Llama-3-70b" });
174+
expect(out["gen_ai.request.model"]).toBe("meta-llama/Llama-3-70b");
175+
expect(out["gen_ai.provider.name"]).toBeUndefined();
176+
});
177+
178+
it("keeps a model name with no namespace as is, with no provider", () => {
179+
const out = mapGenkitAttributes({ ...modelAttributes(), "genkit:name": "my-model" });
180+
expect(out["gen_ai.request.model"]).toBe("my-model");
181+
expect(out["gen_ai.provider.name"]).toBeUndefined();
182+
});
183+
184+
it("sets no model or provider on non-model spans", () => {
185+
const out = mapGenkitAttributes({ ...toolAttributes(), "genkit:name": "googleai/gemini-3.5-flash" });
186+
expect(out["gen_ai.request.model"]).toBeUndefined();
187+
expect(out["gen_ai.provider.name"]).toBeUndefined();
188+
});
189+
150190
it("records usage as unavailable (absent) when the model reported none", () => {
151191
const out = mapGenkitAttributes(modelAttributes(null));
152192
for (const key of Object.keys(out)) {
153193
expect(isPromotedUsageKey(key)).toBe(false);
154194
}
155-
expect(out["gen_ai.request.model"]).toBe("fixture/mock");
195+
expect(out["gen_ai.request.model"]).toBe("mock");
156196
});
157197

158198
it("does not throw on a truncated or non-JSON genkit:output", () => {

‎typescript/packages/traceai_genkit/src/attributes.ts‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
* contract/inventory.mjs (Genkit's own `mockModel` from `genkit/testing`).
77
* Nothing here is a guessed path.
88
*/
9-
import { FISpanKind, SemanticConventions } from "@traceai/fi-semantic-conventions";
9+
import { FISpanKind, LLMProvider, SemanticConventions } from "@traceai/fi-semantic-conventions";
1010

1111
/**
1212
* Attribute keys Genkit 1.42.0 writes on its spans.
@@ -163,6 +163,43 @@ export const GENKIT_USAGE_NOT_MAPPED: readonly string[] = [
163163
"custom",
164164
];
165165

166+
/**
167+
* Model plugin namespace (the part of a model action name before the first
168+
* "/") -> `gen_ai.provider.name`. Only namespaces read from the published
169+
* plugins are listed: googleai / vertexai (@genkit-ai/google-genai), openai /
170+
* deepseek / xai (@genkit-ai/compat-oai), anthropic (genkitx-anthropic),
171+
* ollama (genkitx-ollama), mistral (genkitx-mistral), groq (genkitx-groq),
172+
* aws-bedrock (genkitx-aws-bedrock), azure-openai (genkitx-azure-openai). Any
173+
* other namespace leaves the provider unset. Values are the fi-semantic-conventions
174+
* LLMProvider values; "vertexai" is the Future AGI backend's provider value
175+
* for Vertex AI (FiLLMProviderValues.VERTEXAI), which LLMProvider lacks.
176+
*/
177+
export const MODEL_NAMESPACE_TO_PROVIDER: Readonly<Record<string, string>> = {
178+
googleai: LLMProvider.GOOGLE,
179+
vertexai: "vertexai",
180+
openai: LLMProvider.OPENAI,
181+
deepseek: LLMProvider.DEEPSEEK,
182+
xai: LLMProvider.XAI,
183+
anthropic: LLMProvider.ANTHROPIC,
184+
ollama: LLMProvider.OLLAMA,
185+
mistral: LLMProvider.MISTRALAI,
186+
groq: LLMProvider.GROQ,
187+
"aws-bedrock": LLMProvider.AWS,
188+
"azure-openai": LLMProvider.AZURE,
189+
};
190+
191+
/**
192+
* Split a registered model action name ("googleai/gemini-3.5-flash") into the
193+
* plugin namespace and the vendor model id. fi-collector prices a span by an
194+
* exact model-name lookup in its litellm table, which has "gemini-3.5-flash"
195+
* and not "googleai/gemini-3.5-flash". A name with no "/" has no namespace.
196+
*/
197+
export function splitModelName(name: string): { namespace?: string; model: string } {
198+
const at = name.indexOf("/");
199+
if (at <= 0 || at === name.length - 1) return { model: name };
200+
return { namespace: name.slice(0, at), model: name.slice(at + 1) };
201+
}
202+
166203
/** The Future AGI span-kind keys. fi-collector reads `fi.span.kind`, then `gen_ai.span.kind`. */
167204
export const FI_SPAN_KIND = SemanticConventions.FI_SPAN_KIND;
168205
export const GEN_AI_SPAN_KIND = "gen_ai.span.kind";

‎typescript/packages/traceai_genkit/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@ export {
2020
GENKIT_USAGE_PREFIX,
2121
GenkitAttributes,
2222
GenkitSpanType,
23+
MODEL_NAMESPACE_TO_PROVIDER,
2324
SUBTYPE_TO_KIND,
2425
TYPE_TO_KIND,
2526
USAGE_TO_ATTRIBUTE,
2627
isPromotedUsageKey,
28+
splitModelName,
2729
} from "./attributes";
2830
export { flushOnSignals, type FlushOnSignalsOptions, type SignalTarget } from "./signals";

‎typescript/packages/traceai_genkit/src/mapping.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,12 @@ import {
99
GENKIT_USAGE_PREFIX,
1010
GenkitAttributes,
1111
GenkitSpanType,
12+
MODEL_NAMESPACE_TO_PROVIDER,
1213
SUBTYPE_TO_KIND,
1314
TYPE_TO_KIND,
1415
USAGE_TO_ATTRIBUTE,
1516
isPromotedUsageKey,
17+
splitModelName,
1618
} from "./attributes";
1719

1820
export interface MapOptions {
@@ -67,8 +69,14 @@ function modelCallAttributes(source: Attributes): Attributes {
6769
const out: Attributes = {};
6870
const name = source[GenkitAttributes.NAME];
6971
if (typeof name === "string" && name.length > 0) {
70-
// The registered model action name, e.g. "googleai/gemini-2.5-flash" (action.ts:535, 572).
71-
out[SemanticConventions.LLM_MODEL_NAME] = name;
72+
// The registered model action name, e.g. "googleai/gemini-3.5-flash" (action.ts:535, 572).
73+
// The model keys get the vendor id without the plugin namespace, so fi-collector can price
74+
// the call; the full name stays on genkit:name. Genkit reports no resolved model version.
75+
const { namespace, model } = splitModelName(name);
76+
out[SemanticConventions.LLM_MODEL_NAME] = model;
77+
out[SemanticConventions.GEN_AI_RESPONSE_MODEL] = model;
78+
const provider = namespace === undefined ? undefined : MODEL_NAMESPACE_TO_PROVIDER[namespace];
79+
if (provider !== undefined) out[SemanticConventions.LLM_PROVIDER] = provider;
7280
}
7381
const response = parseJson(source[GenkitAttributes.OUTPUT]);
7482
if (!isRecord(response)) {

0 commit comments

Comments
 (0)