Skip to content

Commit b5cb204

Browse files
committed
fix(genkit): cut schema-validation data from status and exception events
Installed-review N1. Genkit's ValidationError embeds the rejected data in its message after "Provided data:" (@genkit-ai/core 1.42.0 src/schema.ts:78) and instrumentation.ts:156-162 copies e.message to the span status and an exception event on every span the error passes through. The processor passed status and events through unchanged, so with captureContent off (default) a structured ai.generate whose model output fails the schema exported that output to Future AGI. With captureContent off, toExportableSpan now cuts status.message and the exception.message / exception.stacktrace attributes of exception events at "Provided data:" and appends "[data redacted: captureContent is off]". exception.type, the event and every other event are kept. New objects are built; Genkit's span (read by the Dev UI exporter) is not modified. With captureContent true both pass through unchanged. Tests (written first): - jest processor.test.ts: marker in status.message and an exception event, off redacts / on keeps / no-marker unchanged. RED on 24a2929: the off case failed (marker present), 16 others passed. - contract: new `schema` journey (ai.generate with an output schema, the fake mockModel returns non-matching JSON). Asserts the model output text and its marker appear nowhere in the export, both error spans end with the note, exception.type kept ("400", GenkitError.code). RED on 24a2929 (marker found in the exported exception event); opt-in control passed. - GREEN: jest 65 passed; contract 20 passed (Node 26.8.1, no matrix). README: Content table row and paragraph for error-message redaction; the Errors line no longer says status/events pass through unchanged. approved: Nikhil 2026-10-03 blanket Refs: TH-8240
1 parent 24a2929 commit b5cb204

5 files changed

Lines changed: 222 additions & 9 deletions

File tree

‎typescript/packages/traceai_genkit/README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ Unavailable at 1.42.0, and not set: provider, cost, user id (unless the app sets
100100

101101
Tokens are written only on model call spans. The `generate` span's `genkit:output` repeats the last turn's usage; it stays inside that JSON and is not promoted. If any non-model span carries a key the collector promotes (`gen_ai.usage.*`, `llm.token_count.*`, `llm.usage.*`, `gen_ai.cost.total`, `llm.cost.total`), the processor moves it to `genkit.usage.<key>`, so the trace-wide token sum equals the model calls.
102102

103-
Errors: Genkit sets the OTel status to `ERROR` and records an `exception` event on every span the error passes through (`core/src/tracing/instrumentation.ts:153-172`); both pass through unchanged. `genkit:isFailureSource` marks the first failing span.
103+
Errors: Genkit sets the OTel status to `ERROR` and records an `exception` event on every span the error passes through (`core/src/tracing/instrumentation.ts:153-172`). Both are exported; with content off, schema-validation data is cut from them (see [Content](#content)). `genkit:isFailureSource` marks the first failing span.
104104

105105
### Session
106106

@@ -122,9 +122,12 @@ Content is off by default. That is stricter than Genkit, which records inputs an
122122
| `genkit:input`, `genkit:output`, `genkit:init` | dropped | exported, and copied to `input.value` / `output.value` |
123123
| `genkit:metadata:interrupt`, `genkit:metadata:resumed` (tool interrupts) | dropped | exported |
124124
| `genkit:metadata:context` (request context; Genkit redacts `auth` and `secrets`, headers can remain) | dropped | dropped |
125+
| Span status message, `exception.message` and `exception.stacktrace` of `exception` events | cut at `Provided data:`, followed by `[data redacted: captureContent is off]` | exported unchanged |
125126

126127
Usage and finish reason are read from `genkit:output` before it is dropped, so tokens are present with content off.
127128

129+
Genkit's schema-validation error embeds the rejected data in its message: `Schema validation failed. Parse Errors: ... Provided data: <the JSON> Required JSON schema: ...` (`@genkit-ai/core` `src/schema.ts:78`). It is raised, for example, when `ai.generate({ output: { schema } })` gets model output that does not match the schema (the contract `schema` journey), and Genkit copies the message to the status and the exception event of every span the error passes through. With content off, the processor keeps the text before `Provided data:` (status, parse errors and paths), drops the rest (data, schema and, in the stack trace, the stack frames) and appends `[data redacted: captureContent is off]`. `exception.type` and the event itself are kept. Other error messages are exported as thrown: an error your own code throws with user input in its message is not redacted.
130+
128131
The processor never changes Genkit's span. Genkit's Dev UI and any other exporter see the original attributes; Future AGI gets a mapped copy.
129132

130133
## Flush and shutdown
@@ -163,4 +166,4 @@ PYTHONPATH=python/tests uv run --no-project --python 3.11 --with pytest --with p
163166
pytest typescript/packages/traceai_genkit/contract -q -p no:cacheprovider --noconftest -o addopts=''
164167
```
165168

166-
The contract suite builds the package, runs `contract/run_fixture.mjs` against the real fi-core exporter and the shared receiver, and checks the collector path and headers, the resource, span names, kinds and parenting, model and tokens, the trace token sum, content off by default with an opt-in control run, one closed model span for a streamed flow, error status and exception events, SIGTERM flushing (with a control that shows a plain listener loses the race), a collector that is down, Genkit's Dev UI reflection server and telemetry export in `GENKIT_ENV=dev`, the inventory drift check, the packed tarball, and the CJS / ESM entry points. `NODE_BINARY` selects the node binary; `TRACEAI_NODE_MATRIX` (path-separated) widens the entry-point check.
169+
The contract suite builds the package, runs `contract/run_fixture.mjs` against the real fi-core exporter and the shared receiver, and checks the collector path and headers, the resource, span names, kinds and parenting, model and tokens, the trace token sum, content off by default with an opt-in control run, one closed model span for a streamed flow, error status and exception events, schema-validation data cut from status and exception events with content off (and kept after opt-in), SIGTERM flushing (with a control that shows a plain listener loses the race), a collector that is down, Genkit's Dev UI reflection server and telemetry export in `GENKIT_ENV=dev`, the inventory drift check, the packed tarball, and the CJS / ESM entry points. `NODE_BINARY` selects the node binary; `TRACEAI_NODE_MATRIX` (path-separated) widens the entry-point check.

‎typescript/packages/traceai_genkit/contract/run_fixture.mjs‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
* Env:
1010
* FI_BASE_URL, FI_API_KEY, FI_SECRET_KEY, FI_PROJECT_NAME read by fi-core register()
1111
* GENKIT_TELEMETRY_SERVER optional; Genkit's own dev-UI trace exporter target
12-
* JOURNEY tool | stream | error | agent | sigterm | devserver
12+
* JOURNEY tool | stream | error | schema | agent | sigterm | devserver
1313
* CAPTURE_CONTENT=1 captureContent: true
1414
* SESSION_ID wrap the flow call in fi-core setSession()
1515
* FI_GLOBAL_PROVIDER=1 misconfiguration control: register() as the global provider
@@ -107,6 +107,20 @@ const failingFlow = ai.defineFlow({ name: 'failingFlow', inputSchema: z.string()
107107
return response.text;
108108
});
109109

110+
// Structured output the schema rejects: `answer` must be a string. Genkit's
111+
// ValidationError (@genkit-ai/core src/schema.ts:78) embeds the model output
112+
// after "Provided data:" and instrumentation.ts:156-162 writes that message to
113+
// the generate and flow spans' status and exception events.
114+
export const STRUCTURED_OUTPUT_TEXT = JSON.stringify({ answer: 42, note: 'SECRET_STRUCTURED_OUTPUT_MARKER' });
115+
const structuredModel = mockModel(ai, {
116+
name: 'contract/structured-model',
117+
respond: () => ({ text: STRUCTURED_OUTPUT_TEXT, usage: { inputTokens: 7, outputTokens: 5, totalTokens: 12 } }),
118+
});
119+
const structuredFlow = ai.defineFlow({ name: 'structuredFlow', inputSchema: z.string() }, async (question) => {
120+
const response = await ai.generate({ model: structuredModel, prompt: question, output: { schema: z.object({ answer: z.string() }) } });
121+
return response.output;
122+
});
123+
110124
function withSession(fn) {
111125
const sessionId = process.env.SESSION_ID;
112126
if (!sessionId) return fn();
@@ -149,6 +163,13 @@ if (journey === 'tool') {
149163
} catch (error) {
150164
await flushAndExit({ error: String(error && error.message) });
151165
}
166+
} else if (journey === 'schema') {
167+
try {
168+
const result = await structuredFlow(prompt);
169+
await flushAndExit({ result, error: null });
170+
} catch (error) {
171+
await flushAndExit({ error: String(error && error.message), modelOutput: STRUCTURED_OUTPUT_TEXT });
172+
}
152173
} else if (journey === 'agent') {
153174
// Beta agents (genkit/beta defineAgent): Genkit tags the agent span with
154175
// genkit:metadata:agent:sessionId (@genkit-ai/ai src/agent.ts:1061-1063).

‎typescript/packages/traceai_genkit/contract/test_harness_contract.py‎

Lines changed: 64 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,14 @@
5353
PROJECT = "th8240-harness-contract"
5454
PLACEHOLDER_API_KEY = "contract-api-key-PLACEHOLDER"
5555
PLACEHOLDER_SECRET_KEY = "contract-secret-key-PLACEHOLDER"
56-
CONTENT_MARKERS = ("SECRET_PROMPT_MARKER", "SECRET_OUTPUT_MARKER", "SECRET_TOOL_OUTPUT_MARKER", "SECRET_CHUNK_MARKER")
56+
CONTENT_MARKERS = (
57+
"SECRET_PROMPT_MARKER",
58+
"SECRET_OUTPUT_MARKER",
59+
"SECRET_TOOL_OUTPUT_MARKER",
60+
"SECRET_CHUNK_MARKER",
61+
"SECRET_STRUCTURED_OUTPUT_MARKER",
62+
)
63+
REDACTED_NOTE = "[data redacted: captureContent is off]"
5764
PROMOTED_EXACT = {
5865
"gen_ai.usage.input_tokens",
5966
"gen_ai.usage.output_tokens",
@@ -413,6 +420,62 @@ def test_throwing_action_marks_error_with_exception_event(built_package: Path) -
413420
assert "genkit:isFailureSource" not in _attrs(root)
414421

415422

423+
def _exception_events(span: Dict[str, Any]) -> List[Dict[str, Any]]:
424+
return [
425+
{i["key"]: _value(i["value"]) for i in e.get("attributes", [])} for e in span.get("events", []) if e["name"] == "exception"
426+
]
427+
428+
429+
def _run_schema_journey(extra: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
430+
with Receiver() as receiver:
431+
result = run([NODE, str(FIXTURE)], env=_fixture_env(receiver.origin, "schema", extra), stdin=None, timeout=90)
432+
_check(result, "node run_fixture.mjs (schema)")
433+
spans = receiver.spans()
434+
output = _last_json_line(result.stdout)
435+
# Control: Genkit's ValidationError really carried the model output after "Provided data:".
436+
assert "Schema validation failed" in output["error"], output
437+
assert "Provided data:" in output["error"] and "SECRET_STRUCTURED_OUTPUT_MARKER" in output["error"], output
438+
return {"output": output, "spans": spans}
439+
440+
441+
def test_schema_validation_error_does_not_export_model_output_with_content_off(built_package: Path) -> None:
442+
"""N1: ai.generate with an output schema gets non-matching JSON from the model. Genkit's
443+
ValidationError embeds that output in the error message, which Genkit writes to span status
444+
and exception events. With captureContent off the model output must appear nowhere."""
445+
run_ = _run_schema_journey()
446+
spans, model_output = run_["spans"], run_["output"]["modelOutput"]
447+
blob = json.dumps(spans)
448+
assert "SECRET_STRUCTURED_OUTPUT_MARKER" not in blob
449+
assert model_output not in blob and json.dumps(model_output) not in blob
450+
_assert_no_content(spans)
451+
452+
flow = _one(spans, "structuredFlow")
453+
generate = _one(spans, "generate")
454+
for span in (flow, generate):
455+
assert _status_code(span) == 2, span["name"]
456+
message = span["status"]["message"]
457+
assert message.startswith("INVALID_ARGUMENT: Schema validation failed. Parse Errors:"), message
458+
assert message.endswith(REDACTED_NOTE) and "Provided data:" not in message, message
459+
events = _exception_events(span)
460+
assert len(events) == 1, span["name"]
461+
# Kept as recorded: sdk-trace-base 1.25 Span.recordException uses `error.code` first
462+
# (GenkitError.code is the HTTP status, 400 for INVALID_ARGUMENT).
463+
assert events[0]["exception.type"] == "400"
464+
assert events[0]["exception.message"] == message
465+
assert events[0]["exception.stacktrace"].endswith(REDACTED_NOTE)
466+
assert _attrs(generate)["genkit:isFailureSource"] is True
467+
468+
469+
def test_schema_validation_error_keeps_data_after_opt_in(built_package: Path) -> None:
470+
"""Control for N1: with captureContent the same error message is exported unchanged."""
471+
spans = _run_schema_journey({"CAPTURE_CONTENT": "1"})["spans"]
472+
generate = _one(spans, "generate")
473+
assert "Provided data:" in generate["status"]["message"]
474+
assert "SECRET_STRUCTURED_OUTPUT_MARKER" in generate["status"]["message"]
475+
assert "SECRET_STRUCTURED_OUTPUT_MARKER" in _exception_events(generate)[0]["exception.message"]
476+
assert REDACTED_NOTE not in json.dumps(spans)
477+
478+
416479
def test_beta_agent_session_id_maps_to_session_id(built_package: Path) -> None:
417480
"""Session: the native key at 1.42.0 is genkit:metadata:agent:sessionId on the beta agent span."""
418481
with Receiver() as receiver:

‎typescript/packages/traceai_genkit/src/__tests__/processor.test.ts‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,80 @@ describe("FIGenkitSpanProcessor: forwarding to the Future AGI provider", () => {
110110
});
111111
});
112112

113+
describe("FIGenkitSpanProcessor: schema-validation data in errors", () => {
114+
// @genkit-ai/core 1.42.0 src/schema.ts:78 (ValidationError) and src/error.ts:78 (GenkitError prefix).
115+
// core/src/tracing/instrumentation.ts:156-162 copies e.message to status.message and records the exception.
116+
const DATA_MARKER = "SECRET_STRUCTURED_OUTPUT_MARKER";
117+
const REDACTED = "[data redacted: captureContent is off]";
118+
const head = "INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- /answer: must be string\n\n";
119+
const message =
120+
head +
121+
`Provided data:\n\n${JSON.stringify({ answer: 42, note: DATA_MARKER }, null, 2)}\n\n` +
122+
`Required JSON schema:\n\n${JSON.stringify({ type: "object", properties: { answer: { type: "string" } } }, null, 2)}`;
123+
const stacktrace = `GenkitError: ${message}\n at parseSchema (schema.js:71:11)\n at GenerateResponse.assertValidSchema (response.js:124:7)`;
124+
const validationSpan = (spanId: string) =>
125+
v1Span({
126+
name: "generate",
127+
attributes: { ...generateAttributes(), "genkit:state": "error" },
128+
spanId,
129+
status: { code: 2, message },
130+
events: [
131+
{ name: "custom", time: [1, 0], attributes: { note: "not an exception" } },
132+
{
133+
name: "exception",
134+
time: [1, 5],
135+
attributes: { "exception.type": "GenkitError", "exception.message": message, "exception.stacktrace": stacktrace },
136+
},
137+
],
138+
});
139+
140+
it("cuts everything from 'Provided data:' in status.message and exception events when content is off", async () => {
141+
const { exporter, provider } = fiLikeProvider();
142+
const span = validationSpan("9999999999999991");
143+
const processor = new FIGenkitSpanProcessor({ tracerProvider: provider });
144+
processor.onEnd(span);
145+
await processor.forceFlush();
146+
const [exported] = exporter.getFinishedSpans();
147+
expect(JSON.stringify({ status: exported.status, events: exported.events })).not.toContain(DATA_MARKER);
148+
expect(exported.status).toEqual({ code: 2, message: head + REDACTED });
149+
expect(exported.events.map((e) => e.name)).toEqual(["custom", "exception"]);
150+
expect(exported.events[0]).toEqual(span.events[0]);
151+
const exception = exported.events[1];
152+
expect(exception.time).toEqual([1, 5]);
153+
expect(exception.attributes).toEqual({
154+
"exception.type": "GenkitError",
155+
"exception.message": head + REDACTED,
156+
"exception.stacktrace": `GenkitError: ${head}${REDACTED}`,
157+
});
158+
// Genkit's own span (read by the Dev UI exporter) keeps the full message.
159+
expect(span.status.message).toBe(message);
160+
expect(span.events[1].attributes?.["exception.message"]).toBe(message);
161+
});
162+
163+
it("exports status.message and exception events unchanged with captureContent", async () => {
164+
const { exporter, provider } = fiLikeProvider();
165+
const span = validationSpan("9999999999999992");
166+
const processor = new FIGenkitSpanProcessor({ tracerProvider: provider, captureContent: true });
167+
processor.onEnd(span);
168+
await processor.forceFlush();
169+
const [exported] = exporter.getFinishedSpans();
170+
expect(exported.status).toEqual({ code: 2, message });
171+
expect(exported.events).toEqual(span.events);
172+
expect(JSON.stringify(exported.events)).toContain(DATA_MARKER);
173+
});
174+
175+
it("leaves error messages without the marker unchanged when content is off", async () => {
176+
const { exporter, provider } = fiLikeProvider();
177+
const processor = new FIGenkitSpanProcessor({ tracerProvider: provider });
178+
const events = [{ name: "exception", time: [1, 0] as [number, number], attributes: { "exception.type": "Error", "exception.message": "boom", "exception.stacktrace": "Error: boom\n at x (y.js:1:1)" } }];
179+
processor.onEnd(v1Span({ name: "lookup", attributes: toolAttributes(), spanId: "9999999999999993", status: { code: 2, message: "boom" }, events }));
180+
await processor.forceFlush();
181+
const [exported] = exporter.getFinishedSpans();
182+
expect(exported.status).toEqual({ code: 2, message: "boom" });
183+
expect(exported.events).toEqual(events);
184+
});
185+
});
186+
113187
describe("FIGenkitSpanProcessor: failure isolation", () => {
114188
it("onEnd never throws when the downstream processor throws", () => {
115189
const { provider } = fiLikeProvider();

0 commit comments

Comments
 (0)