Repository navigation
feat(typescript): hide error details with the content hide flags - #271
Merged
Merged
Conversation
Error text is content. Provider error bodies quote the prompt back and carry request/session ids; JavaScript stacks carry local file paths. With content hidden, traceAI-ts still exported them, and LangChain put run.error (message plus the full stack) in the status description. Core (@traceai/fi-core 1.1.0): - TraceConfig.hideErrorDetails / FI_HIDE_ERROR_DETAILS. Unset, it is on when any of hideInputs, hideOutputs, hideInputMessages, hideOutputMessages, hideInputText, hideOutputText is on; an explicit true/false wins. - Enforced once, in FISpan: recordException, setStatus, addEvent and setAttribute(s). When on, ERROR has no description, exception events keep only exception.type/escaped, and error.message / exception.* / *.error.message keys are dropped. - Always: error.type on ERROR spans (provider code, else error class), and an ERROR description that is the exception message only, never a stack trace. - FITracer.startActiveSpan makes the FISpan the active span, so trace.getActiveSpan() goes through the policy. - FI_HIDE_* env vars now apply when no traceConfig is passed (as in Python). LangChain (@traceai/langchain 0.1.3): the status description comes from the error passed to handle*Error, not run.error; the stack stays only on the exception event. OpenAI Agents (@traceai/openai-agents 0.1.1): the status description is the error message; error data moves to the exception event. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Security fix that predates this PR: FITracer.startSpan passed options.attributes straight to the SDK span, so attributes given at span creation skipped FISpan and every masking rule (hide flags, PII redaction, error-detail hiding). About 22 instrumentors put the prompt there, OpenAI included, so with FI_HIDE_INPUTS=true the prompt was still exported in input.value, gen_ai.input.messages and raw.input. This dates from May 2025, so every published @traceai/fi-core release has it. @traceai/fi-core (1.1.0, unreleased; npm latest is 1.0.0): - startSpan and startActiveSpan run creation-time attributes through maskAttributes(), the same masking FISpan.setAttributes uses. The masked attributes are still passed at creation, so samplers see what is exported. An error.type passed at creation is kept. - hideInputs also redacts raw.input and llm.prompt_template.variables; hideOutputs also redacts raw.output. - The error-detail key filter is now the first masking rule, so every attribute path shares it. - Hidden mode filters events with a denylist (Python's strip_error_details): exception.message, exception.stacktrace, error.message and the *.error.message-style keys are dropped, everything else is kept. - exception.type is the same in both modes. FISpan.recordException builds the event itself: a string code that is an identifier, else the error name / class. A numeric code (HTTP 400) is no longer the type. - stripStackTrace keeps a multi-line message whole when the description is a full err.stack, and keeps a relayed Python traceback's final "Type: message" line (chained tracebacks too). - The openai.test.ts trace-config snapshot expected input.value and raw.input unmasked with hideInputs: true. It now expects them redacted. @traceai/openai-agents: span errors are recorded as AgentSpanError, so exception.type / error.type are not missing or _OTHER. @traceai/bedrock 0.1.1: non-Error failures (cross-realm errors, strings, plain objects) get recordException, error.type and a message-only status instead of String(error). @traceai/openai 0.1.14, @traceai/anthropic 0.1.3: version bumps only. Their published versions depend on @traceai/fi-core ^0.1.x, so they keep the bug until they are republished against 1.1.0. Tests: new hidden-mode tests assert the prompt is in no exported attribute, event or status for startSpan, startActiveSpan, the sampler, the real OpenAI SDK and the real LangChain instrumentation (against local HTTP servers), and Bedrock error paths. Co-Authored-By: Claude Opus 5.5 <[email protected]>
dropRepeatedStackHeader sliced and regex-matched the rest of the message for every blank-line separator (quadratic: about 165 s at 900 KB). It now walks the text once with forward-only indexes and compares the message only when its length can fit. The Exception Group regex, which was quadratic on long whitespace runs, is a suffix check, and the frame search walks lines with indexOf instead of splitting. Link attributes (SpanOptions.links, addLink, addLinks) go through maskAttributes(). Adds 1 MB performance tests (CPU time) for each message shape and a link masking test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
nik13
marked this pull request as ready for review
October 21, 2026 23:43
nik13
added a commit
that referenced
this pull request
Oct 22, 2026
The repo now releases dev -> main (#283), and #271/#272 target dev, so this PR moves to dev too. Changing the base does not start the pull_request workflows; this empty commit does. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Merged
7 of 9 tasks
Conflicts in typescript/packages/traceai_openai_agents (version only): dev has the unreleased @traceai/openai-agents 0.2.0 rewrite and this PR bumped the package to 0.1.1. The PR's change now ships as the next free version, 0.2.1 (package.json, src/version.ts, package CHANGELOG entry above dev's 0.2.0, root CHANGELOG heading). instrumentation.ts and its test merged without conflicts; the AgentSpanError / message-only status change applies unchanged on top of dev's rewrite. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The dev merge moved @traceai/openai-agents to 0.2.1; the VERSION test still pinned dev's 0.2.0. Co-Authored-By: Claude Opus 5.5 <[email protected]>
jabrailkhalil
pushed a commit
to jabrailkhalil/traceAI
that referenced
this pull request
Nov 1, 2026
…TracerProvider() InstrumentationBase calls enable() from its constructor, before the subclass stores traceConfig, so the FITracer was always built with traceConfig undefined. fi-core returns its defaults for undefined without reading FI_HIDE_INPUTS / FI_HIDE_OUTPUTS, so neither the option nor the environment variables redacted anything. The FITracer also kept the tracer it got at that point. A provider set with setTracerProvider() afterwards (the order the README shows) was ignored unless enable() ran again, and spans went to the global provider instead. Build the FITracer again once the config is stored and when the module is patched, and always pass an options object so fi-core reads the environment. Three new real-SDK tests cover the README order, traceConfig redaction and FI_HIDE_* redaction; all three fail without this change. raw.input / raw.output are still exported by fi-core 1.0.0 (fixed in future-agi#271); the README says so. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Security fix in this PR (it predates the PR)
Content hiding never applied to the attributes passed when a span is created.
FITracer.startSpanhandedoptions.attributesstraight to the OpenTelemetry span, so they skippedFISpanand every masking rule: the hide flags, PII redaction, and the new error-detail hiding. About two dozen instrumentors pass the prompt there (23 by a grep of theirstartSpancalls),@traceai/openaiincluded. So withFI_HIDE_INPUTS=true, the prompt was still exported ininput.value,gen_ai.input.messagesandraw.input. The review reproduced it end to end: the stored OpenAI rows had the prompt in theinputcolumn and in those attributes.@traceai/fi-corerelease (0.1.0 to 1.0.0) has it.startSpanandstartActiveSpannow run creation-time attributes throughmaskAttributes(), the same maskingFISpan.setAttributesuses. The masked attributes are still passed at creation, so a sampler sees the same values that get exported. Anerror.typepassed at creation is kept.hideInputsnow also redactsraw.inputandllm.prompt_template.variables, andhideOutputsredactsraw.output. They carry the same content asinput.value/output.value: the raw request body, the raw response body, and the values filled into a prompt template.@traceai/fi-corestays at 1.1.0. It is unreleased (npm latest is 1.0.0) and ships this fix together with the feature, so one minor bump covers both.^1.0.0get it when users upgrade.@traceai/openai0.1.13,@traceai/anthropic0.1.2,@traceai/bedrock0.1.0 and@traceai/langchain0.1.2 were published against^0.1.x, so they keep the bug.^1.1.0.workspace:^becomes^1.1.0at publish.Why (the original change)
Error text is content. Provider error bodies quote the prompt back and carry request/session ids, and stack traces carry local file paths. With content hidden, traceAI-ts still exported all of it. The LangChain tracer also put
run.error(the message plus the full stack) in the status description. This is the same leak class as Pydantic AI's GHSA-4x9p-g9wm-8q7f, and we hit it ourselves on TH-8226 and with a Vercel AI Gateway error body. The Python side is #272 (feat/hide-error-details-python). The two branches still merge with no conflicts (git merge-tree), and this branch merges cleanly intodev.What changes
@traceai/fi-core1.0.0 → 1.1.0Always (no flag):
Status stays
ERROR.The span gets
error.type: the provider error code when the error has one, otherwise the error class name, otherwise_OTHER. Provider codes read:code, e.g.model_not_founderror.type, e.g.not_found_errorECONNREFUSEDtypewhen there is no codeAn
error.typethe instrumentation set itself, at creation or later, is kept.exception.typeis the same in both modes.FISpan.recordExceptionbuilds the event itself, with the same keys as the SDK. The type is a stringcodethat is an identifier (as the SDK records it), otherwise the error name / class, as in Python. A numeric code such as HTTP400is no longer the type; the SDK alone would record"400".The status description is the message only.
FISpan.setStatusstrips JS stack frames, and turns"<Name>: <message>"(String(error)) into the message.err.stack, a multi-line message is kept whole.Type: messageline, for chained tracebacks too.exception.stacktrace.New
hideErrorDetails/FI_HIDE_ERROR_DETAILS(tri-state):hideInputs,hideOutputs,hideInputMessages,hideOutputMessages,hideInputText,hideOutputTextis on.true/false(option or env) wins.strip_error_details, applies to span attributes (at creation and later), exception events and other events. It dropsexception.message,exception.stacktrace,error.message, and keys ending in.exception.message,.exception.stacktrace,.error.messageor.error_message.exception.type,exception.escaped,error.typeand custom attributes.Where it is enforced:
mask()/maskAttributes().FISpanuses it forsetAttribute(s), andFITracerfor creation-time attributes.FISpan:recordException,setStatusandaddEvent.FITracer.startActiveSpanmakes theFISpanthe active span, sotrace.getActiveSpan()?.recordException(...)is filtered too.Env vars without a
traceConfig:FI_HIDE_*,FI_PII_REDACTIONandFI_BASE64_IMAGE_MAX_LENGTHare now read even when notraceConfigis passed, as in Python.Instrumentation packages
@traceai/langchain0.1.2 → 0.1.3: the status description and exception come from the error object passed tohandle*Error, notrun.error. When that object was not seen,run.erroris split back into message and stack.@traceai/openai-agents0.1.0 → 0.1.1:"<message>: <data>".datagoes to the exception event.AgentSpanError, soexception.type/error.typeare not missing or_OTHER.@traceai/bedrock0.1.0 → 0.1.1: non-Errorfailures (cross-realm errors, thrown strings, plain objects) getrecordException,error.typeand a message-only status. Before, they gotsetStatus(String(error))("TypeError: msg","[object Object]") with no exception event.@traceai/openai0.1.13 → 0.1.14,@traceai/anthropic0.1.2 → 0.1.3: version bumps only, so they can be republished against fi-core 1.1.0 (see above).Behaviour changes (all in the changelogs)
raw.input,raw.outputandllm.prompt_template.variables.hideErrorDetails: falsekeeps them.FI_HIDE_*env vars set without atraceConfignow take effect.error.typeattribute.FITracer.startActiveSpan,trace.getActiveSpan()is theFISpan, not the SDK span. Code that casts it to the SDKSpan/ReadableSpanbreaks there.Changelogs and docs
docs/configuration/trace-config.mdlists every dropped key and the creation-time masking.docs/typescript/fi-core.mdis updated the same way.Review notes addressed (round 2)
startSpancreation attributes skip maskingmaskAttributes()instartSpan/startActiveSpan, masked attributes still passed for the sampler.raw.input,raw.outputandllm.prompt_template.variablescovered. New tests below, plus a real e2e that scans every stored column.recordExceptionnameAgentSpanErrorstripStackTraceType: msgline(s).recordBedrockError()in all six error paths.exception.typefor a numeric codestartActiveSpancontext noteAlso found:
openai.test.ts's "should respect a trace config and mask attributes accordingly" had an inline snapshot that expectedinput.valueandraw.inputunmasked underhideInputs: true. It had locked the bug in. It now expects them redacted. That suite only runs with ts-jest diagnostics off, because of a pre-existing zod 4 type error.Tests
Node 26.8.1, pnpm 10.30.1. I ran every package's jest one at a time. traceAI has no TS CI.
fi-core: 124 passed (101 before).creationAttributes.test.tscovers the reviewer's exact repro. ForstartSpan,startActiveSpan(with context attributes) and a recording sampler, it asserts that in hidden mode the prompt is in no exported attribute, event, status or link, and that the sampler sees the masked values.hideErrorDetailsalone drops creation-time error keys,raw.output, PII redaction at creation, that the no-flags output is unchanged, that anerror.typeset at creation is kept, and that other span options (kind, startTime, parent) still pass through.errorDetails.test.tsadds the denylist, numeric-code parity, thrown strings, multi-lineerr.stack, and relayed and chained Python tracebacks.traceai_openai: newhideContent.test.tsuses the real OpenAI SDK against a local HTTP server whose 404 body quotes the prompt.hideInputs(and withhideInputs+hideOutputsand an echoing server), the prompt and system prompt are in no exported attribute, event or status.openai.test.ts, which still fails to compile ondev(zod 4). With diagnostics off it passes 18/18 (2 skipped) after the snapshot fix.traceai_langchain: 36 passed. NewhideContent.test.tsuses the realLangChainInstrumentation(patchedCallbackManager) and a realChatOpenAIagainst a local server.hideInputs, an LLM success, an LLM 404 and a failing step leave no prompt anywhere.output.value. Outputs are not inputs.hideInputs+hideOutputs, the chain, success or failure, exports no prompt at all.traceai_bedrock: newerrors.test.ts, 8 passed. It covers cross-realm errors, thrown strings, rejected plain objects and hidden mode, for InvokeModel and Converse. 4 of them fail on the previous code.traceai_openai_agents: 14 passed.tsc --noEmitis clean for fi-core, bedrock, langchain, openai-agents and openai.Real check (local Future AGI, project
th8103-hideerr-ts-r2)fi_verify.pypasses: 3 traces, 21 spans, 12 ERROR. Each run makes these calls through the local CLIProxy:FITracer.startActiveSpanwith the prompt as a start attribute;claude-haiku-4-5);400 unknown provider …,code: model_not_found; the proxy echoes the model name);ChatOpenAIcall to that model;No
traceConfigwas passed, so only env vars drive the instrumentations. The prompt and the model name carry separate markers.What was scanned: a new checker reads every stored copy of the data, not just status and events.
spans: every column (46) of every row of the trace, nested JSON included.traces/tracer_tracerows.spansandtraces.dfaf59e3…error.typemodel_not_found/Error; stack only on the eventFI_HIDE_INPUTS=true90f4719d…output/output.value. That is an output, andFI_HIDE_INPUTSdoes not hide outputs. Everyinputcolumn is__REDACTED__(12 B),raw.input/input.valueredacted, nogen_ai.input.messages, template variables redactedmodel,gen_ai.request.model,gen_ai.request.parameters, LangChainmetadata.ls_model_name, the hourly rollup'smodel), never in error textstatus_message,error.typepresent, noerror.message, exception events{exception.type}onlyFI_HIDE_INPUTS=true+FI_HIDE_OUTPUTS=true61b0446c…The previous head's hidden run (the review's trace
373752f4…, projectth8103-hideerr-ts-review) had the prompt in the OpenAI row'sinputcolumn,gen_ai.input.messagesandraw.input.Audit of instrumentors (
typescript/packages/*)Covered by the wrapper: every instrumentor whose spans come from
FITracer.startSpan/startActiveSpan. They now get masking at creation too.There is no TS LiteLLM instrumentor.
Bypass the wrapper, follow-ups (not changed here): these packages apply no
TraceConfigat all.@traceai/llamaindex:src/instrumentation.ts:83-187uses the raw tracer.@traceai/vercel:FISpanProcessor.onEnd(src/FISpanProcessor.ts:53,:122) post-processes AI SDK spans, which recordexception.*andstatus.messagethemselves.@traceai/mastra: MastraerrorInfopasses throughSpanConverterunchanged.@traceai/mcpcreates no spans.Known gaps, not in this PR
embedding.embeddings.N.embedding.text(the embedded input) is covered by no hide flag. Python has the same gap.@traceai/openaiputsinstructions(a system prompt) ingen_ai.request.parameters. TS has nohideLlmInvocationParametersflag; Python does.TraceConfig.maskdoes not redactraw.input/raw.output/llm.prompt_template.variableseither. Worth the same change in feat(python): hide error details with the content hide flags #272 or a follow-up.startSpan:FITracer.startSpandoes not add session / user / metadata context attributes, as fi-core's first version and Python'sstart_spando (also dropped in 685eaf4). Some instrumentors add them themselves. This is not a hiding issue, so it is left alone.Noticed while testing (pre-existing, not in scope)
FITracerProvider.addSpanProcessorpushes to_registeredSpanProcessors, which OTel SDK 2.x'sBasicTracerProviderno longer reads.register({ batch: true })goes through that path (src/otel.ts:601).register()installs no context manager, so instrumented spans don't nest under a manualstartActiveSpanroot unless the app sets one.Round 3 (2026-10-19)
An independent re-review found
stripStackTrace/sanitizeStatusquadratic in message size (dropRepeatedStackHeadersliced and regex-matched the rest of the text for every"\n\n"). Fixed in 5f73cd9:dropRepeatedStackHeaderwalks the text once with forward-only indexes; the message is compared only when its length fits the tail. Same results as before for multi-line messages, relayed Python tracebacks and frames.Exception Groupregex (quadratic on long whitespace runs; 81 s at 64 KB) is a suffix check; the frame search walks lines withindexOf.SpanOptions.links,addLink,addLinks) are now masked withmaskAttributes().setStatus(default mode) andrecordException; a link masking test.CPU time at 1 MB, after: 0-17 ms per shape. Before (64 KB): repeated header 0.7 s, different paragraphs 1.2 s, whitespace run before a traceback header 81 s.
Tests: fi-core 142 pass (was 124), openai 93, anthropic 53 (3 skipped), bedrock 8, langchain 36, openai_agents 14.
openai.test.tsstill fails to compile (zod 4, pre-existing).🤖 Generated with Claude Code
Stored-row verification (2026-10-19, head 5f73cd9)
Ran a real e2e against a local Future AGI stack. It has 4 failing spans:
ChatOpenAIfailure;I then scanned every stored ClickHouse
spanscolumn of every row. fi_verify passed in all three modes.error.type=Error/model_not_foundFI_HIDE_INPUTS=trueerror.typekeptFI_HIDE_INPUTS+FI_HIDE_OUTPUTSerror.typekeptThe model name appears only in model fields, never in error text.
Merged
dev(2026-10-21,0a4653e)devmerged in with no conflicts (merge commit0a4653e). It brings only Python and CI changes;typescript/,docs/and the rootCHANGELOG.mdare byte-identical to5f73cd9. Re-run on the merge (Node 26.8.1, pnpm 10.30.1, frozen lockfile):openai.test.tsstill fails to compile under ts-jest diagnostics (zod 4, pre-existing). Correction to the Tests section above: with diagnostics off it gives 17 passed, 2 skipped and 1 failed, not 18/18. The failure iscreates a span for chat completions parse: under zod 4,zodResponseFormatemits an empty JSON schema, so the inline snapshot does not match. This PR does not touch that test or the lockfile, so it fails the same way ondev. The snapshot this PR fixed (should respect a trace config and mask attributes accordingly) passes.Nothing here is on npm yet (
@traceai/fi-corelatest 1.0.0, openai 0.1.13, anthropic 0.1.2, bedrock 0.1.0, langchain 0.1.2, openai-agents 0.1.0), so the bumped versions are free.