Skip to content

feat(typescript): hide error details with the content hide flags - #271

Merged
nik13 merged 6 commits into
devfrom
feat/hide-error-details-ts
Oct 26, 2026
Merged

nik13 merged 6 commits into
devfrom
feat/hide-error-details-ts

Conversation

@nik13

@nik13 nik13 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Security fix in this PR (it predates the PR)

Content hiding never applied to the attributes passed when a span is created. FITracer.startSpan handed options.attributes straight to the OpenTelemetry span, so they skipped FISpan and every masking rule: the hide flags, PII redaction, and the new error-detail hiding. About two dozen instrumentors pass the prompt there (23 by a grep of their startSpan calls), @traceai/openai included. So with FI_HIDE_INPUTS=true, the prompt was still exported in input.value, gen_ai.input.messages and raw.input. The review reproduced it end to end: the stored OpenAI rows had the prompt in the input column and in those attributes.

  • Since when: commit 685eaf4 (2025-05-15), so every published @traceai/fi-core release (0.1.0 to 1.0.0) has it.
  • Fix: startSpan and startActiveSpan now run creation-time attributes through maskAttributes(), the same masking FISpan.setAttributes uses. The masked attributes are still passed at creation, so a sampler sees the same values that get exported. An error.type passed at creation is kept.
  • Gaps closed alongside it: hideInputs now also redacts raw.input and llm.prompt_template.variables, and hideOutputs redacts raw.output. They carry the same content as input.value / output.value: the raw request body, the raw response body, and the values filled into a prompt template.
  • Versions: @traceai/fi-core stays at 1.1.0. It is unreleased (npm latest is 1.0.0) and ships this fix together with the feature, so one minor bump covers both.
  • Release: the fix only reaches an instrumentation package that resolves fi-core 1.1.0.
    • Packages published with ^1.0.0 get it when users upgrade.
    • @traceai/openai 0.1.13, @traceai/anthropic 0.1.2, @traceai/bedrock 0.1.0 and @traceai/langchain 0.1.2 were published against ^0.1.x, so they keep the bug.
    • This PR bumps them to 0.1.14 / 0.1.3 / 0.1.1 / 0.1.3 so they can be republished against ^1.1.0. workspace:^ becomes ^1.1.0 at publish.
    • Publish fi-core first.

Why (the original change)

Error text is content. Provider error bodies quote the prompt back and carry request/session ids, and stack traces carry local file paths. With content hidden, traceAI-ts still exported all of it. The LangChain tracer also put run.error (the message plus the full stack) in the status description. This is the same leak class as Pydantic AI's GHSA-4x9p-g9wm-8q7f, and we hit it ourselves on TH-8226 and with a Vercel AI Gateway error body. The Python side is #272 (feat/hide-error-details-python). The two branches still merge with no conflicts (git merge-tree), and this branch merges cleanly into dev.

What changes

@traceai/fi-core 1.0.0 → 1.1.0

Always (no flag):

  • Status stays ERROR.

  • The span gets error.type: the provider error code when the error has one, otherwise the error class name, otherwise _OTHER. Provider codes read:

    • OpenAI-compatible code, e.g. model_not_found
    • Anthropic body error.type, e.g. not_found_error
    • Node ECONNREFUSED
    • OpenAI type when there is no code

    An error.type the instrumentation set itself, at creation or later, is kept.

  • exception.type is the same in both modes. FISpan.recordException builds the event itself, with the same keys as the SDK. The type is a string code that is an identifier (as the SDK records it), otherwise the error name / class, as in Python. A numeric code such as HTTP 400 is no longer the type; the SDK alone would record "400".

  • The status description is the message only.

    • FISpan.setStatus strips JS stack frames, and turns "<Name>: <message>" (String(error)) into the message.
    • When the description is a full err.stack, a multi-line message is kept whole.
    • A relayed Python traceback loses its frames but keeps its final Type: message line, for chained tracebacks too.
    • The stack stays only in exception.stacktrace.

New hideErrorDetails / FI_HIDE_ERROR_DETAILS (tri-state):

  • Unset: on when any of hideInputs, hideOutputs, hideInputMessages, hideOutputMessages, hideInputText, hideOutputText is on.
  • An explicit true / false (option or env) wins.
  • When on:
    • The ERROR status has no description.
    • A denylist, the same as Python's strip_error_details, applies to span attributes (at creation and later), exception events and other events. It drops exception.message, exception.stacktrace, error.message, and keys ending in .exception.message, .exception.stacktrace, .error.message or .error_message.
    • Everything else is kept: exception.type, exception.escaped, error.type and custom attributes.

Where it is enforced:

  • Masking (content and error detail) is in one function, mask() / maskAttributes(). FISpan uses it for setAttribute(s), and FITracer for creation-time attributes.
  • Events and status go through FISpan: recordException, setStatus and addEvent.
  • FITracer.startActiveSpan makes the FISpan the active span, so trace.getActiveSpan()?.recordException(...) is filtered too.

Env vars without a traceConfig: FI_HIDE_*, FI_PII_REDACTION and FI_BASE64_IMAGE_MAX_LENGTH are now read even when no traceConfig is passed, as in Python.

Instrumentation packages

  • @traceai/langchain 0.1.2 → 0.1.3: the status description and exception come from the error object passed to handle*Error, not run.error. When that object was not seen, run.error is split back into message and stack.
  • @traceai/openai-agents 0.1.0 → 0.1.1:
    • The status is the message only. It was "<message>: <data>".
    • data goes to the exception event.
    • The exception is named AgentSpanError, so exception.type / error.type are not missing or _OTHER.
  • @traceai/bedrock 0.1.0 → 0.1.1: non-Error failures (cross-realm errors, thrown strings, plain objects) get recordException, error.type and a message-only status. Before, they got setStatus(String(error)) ("TypeError: msg", "[object Object]") with no exception event.
  • @traceai/openai 0.1.13 → 0.1.14, @traceai/anthropic 0.1.2 → 0.1.3: version bumps only, so they can be republished against fi-core 1.1.0 (see above).

Behaviour changes (all in the changelogs)

  • With a hide flag on, creation-time content is now redacted, including raw.input, raw.output and llm.prompt_template.variables.
  • Users of any content-hiding flag now also lose error messages, stack traces and status descriptions. hideErrorDetails: false keeps them.
  • FI_HIDE_* env vars set without a traceConfig now take effect.
  • ERROR spans get a new error.type attribute.
  • Inside FITracer.startActiveSpan, trace.getActiveSpan() is the FISpan, not the SDK span. Code that casts it to the SDK Span / ReadableSpan breaks there.

Changelogs and docs

  • Changelogs: root, fi-core, openai, anthropic, bedrock (new), langchain and openai-agents. The root changelog opens with the security note.
  • Docs: docs/configuration/trace-config.md lists every dropped key and the creation-time masking. docs/typescript/fi-core.md is updated the same way.

Review notes addressed (round 2)

Note Change
Blocking: startSpan creation attributes skip masking maskAttributes() in startSpan / startActiveSpan, masked attributes still passed for the sampler. raw.input, raw.output and llm.prompt_template.variables covered. New tests below, plus a real e2e that scans every stored column.
(a) denylist for exception events Done, same keys as Python. A test shows custom attributes are kept and the message and stack are dropped.
(b) agents recordException name AgentSpanError
(c) stripStackTrace Multi-line messages are kept whole. A relayed (and chained) Python traceback keeps its Type: msg line(s).
(d) Bedrock non-Error branches recordBedrockError() in all six error paths.
(e) exception.type for a numeric code Error name / class in both modes.
(f) startActiveSpan context note In the fi-core and root changelogs.
(g) docs list all dropped keys Done.
(h) call out the fix, version This section at the top, the root changelog note, and the fi-core changelog's first section. Versions: see above.

Also found: openai.test.ts's "should respect a trace config and mask attributes accordingly" had an inline snapshot that expected input.value and raw.input unmasked under hideInputs: true. It had locked the bug in. It now expects them redacted. That suite only runs with ts-jest diagnostics off, because of a pre-existing zod 4 type error.

Tests

Node 26.8.1, pnpm 10.30.1. I ran every package's jest one at a time. traceAI has no TS CI.

  • fi-core: 124 passed (101 before).
    • New creationAttributes.test.ts covers the reviewer's exact repro. For startSpan, startActiveSpan (with context attributes) and a recording sampler, it asserts that in hidden mode the prompt is in no exported attribute, event, status or link, and that the sampler sees the masked values.
    • It also checks that hideErrorDetails alone drops creation-time error keys, raw.output, PII redaction at creation, that the no-flags output is unchanged, that an error.type set at creation is kept, and that other span options (kind, startTime, parent) still pass through.
    • 10 of its 13 tests fail on the previous head.
    • errorDetails.test.ts adds the denylist, numeric-code parity, thrown strings, multi-line err.stack, and relayed and chained Python tracebacks.
  • traceai_openai: new hideContent.test.ts uses the real OpenAI SDK against a local HTTP server whose 404 body quotes the prompt.
    • It covers chat success, chat 404, a chat stream, legacy completion and legacy completion 404.
    • With hideInputs (and with hideInputs + hideOutputs and an echoing server), the prompt and system prompt are in no exported attribute, event or status.
    • A control test without flags shows the check does see the prompt.
    • It fails on the previous fi-core.
    • The package's suites pass (93 tests, the new ones included), except openai.test.ts, which still fails to compile on dev (zod 4). With diagnostics off it passes 18/18 (2 skipped) after the snapshot fix.
  • traceai_langchain: 36 passed. New hideContent.test.ts uses the real LangChainInstrumentation (patched CallbackManager) and a real ChatOpenAI against a local server.
    • With hideInputs, an LLM success, an LLM 404 and a failing step leave no prompt anywhere.
    • With a prompt-template chain, the template variables are redacted, and the only attribute still holding the prompt is the template's output.value. Outputs are not inputs.
    • With hideInputs + hideOutputs, the chain, success or failure, exports no prompt at all.
  • traceai_bedrock: new errors.test.ts, 8 passed. It covers cross-realm errors, thrown strings, rejected plain objects and hidden mode, for InvokeModel and Converse. 4 of them fail on the previous code.
  • traceai_openai_agents: 14 passed.
  • All other packages pass. llamaindex, mcp and vercel have no tests. tsc --noEmit is clean for fi-core, bedrock, langchain, openai-agents and openai.

Real check (local Future AGI, project th8103-hideerr-ts-r2)

fi_verify.py passes: 3 traces, 21 spans, 12 ERROR. Each run makes these calls through the local CLIProxy:

  • a root span created by FITracer.startActiveSpan with the prompt as a start attribute;
  • a successful OpenAI SDK call (claude-haiku-4-5);
  • an OpenAI SDK call to a bad model (400 unknown provider …, code: model_not_found; the proxy echoes the model name);
  • a ChatOpenAI call to that model;
  • a LangChain chain (prompt template → a step that throws with the prompt in its message).

No traceConfig was passed, so only env vars drive the instrumentations. The prompt and the model name carry separate markers.

What was scanned: a new checker reads every stored copy of the data, not just status and events.

  • ClickHouse spans: every column (46) of every row of the trace, nested JSON included.
  • ClickHouse traces: the traces / tracer_trace rows.
  • ClickHouse other tables: every column of every other small table (rollups, sessions).
  • Other projects: a whole-table search of spans and traces.
  • Postgres: every row of all 313 public tables.
  • Exporter: every attribute, event, status and link it was handed.
Run Trace Prompt marker found Model marker found ERROR spans
default dfaf59e3… in inputs, statuses and events, as expected in model fields and error text, as expected message-only description; error.type model_not_found / Error; stack only on the event
FI_HIDE_INPUTS=true 90f4719d… only in the prompt template span's output / output.value. That is an output, and FI_HIDE_INPUTS does not hide outputs. Every input column is __REDACTED__ (12 B), raw.input / input.value redacted, no gen_ai.input.messages, template variables redacted only in model fields (model, gen_ai.request.model, gen_ai.request.parameters, LangChain metadata.ls_model_name, the hourly rollup's model), never in error text 4/4: empty status_message, error.type present, no error.message, exception events {exception.type} only
FI_HIDE_INPUTS=true + FI_HIDE_OUTPUTS=true 61b0446c… nowhere, in any column of any row or store model fields only same as above

The previous head's hidden run (the review's trace 373752f4…, project th8103-hideerr-ts-review) had the prompt in the OpenAI row's input column, gen_ai.input.messages and raw.input.

Audit of instrumentors (typescript/packages/*)

Covered by the wrapper: every instrumentor whose spans come from FITracer.startSpan / startActiveSpan. They now get masking at creation too.

  • OpenAI, Anthropic, Bedrock, Google GenAI, LangChain, OpenAI Agents
  • Cohere, Mistral, Groq, Together, Fireworks, DeepSeek, Cerebras, xAI, vLLM, Ollama, HuggingFace, Portkey, Vertex AI, Google ADK
  • BeeAI, Strands, Instructor, Guardrails, LiveKit, Pipecat
  • the vector DBs: Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB, MongoDB, pgvector, Redis

There is no TS LiteLLM instrumentor.

Bypass the wrapper, follow-ups (not changed here): these packages apply no TraceConfig at all.

  • @traceai/llamaindex: src/instrumentation.ts:83-187 uses the raw tracer.
  • @traceai/vercel: FISpanProcessor.onEnd (src/FISpanProcessor.ts:53, :122) post-processes AI SDK spans, which record exception.* and status.message themselves.
  • @traceai/mastra: Mastra errorInfo passes through SpanConverter unchanged.
  • @traceai/mcp creates no spans.

Known gaps, not in this PR

  • Embeddings: embedding.embeddings.N.embedding.text (the embedded input) is covered by no hide flag. Python has the same gap.
  • Responses API: @traceai/openai puts instructions (a system prompt) in gen_ai.request.parameters. TS has no hideLlmInvocationParameters flag; Python does.
  • Python parity: Python's TraceConfig.mask does not redact raw.input / raw.output / llm.prompt_template.variables either. Worth the same change in feat(python): hide error details with the content hide flags #272 or a follow-up.
  • Context attributes in startSpan: FITracer.startSpan does not add session / user / metadata context attributes, as fi-core's first version and Python's start_span do (also dropped in 685eaf4). Some instrumentors add them themselves. This is not a hiding issue, so it is left alone.

Noticed while testing (pre-existing, not in scope)

  • FITracerProvider.addSpanProcessor pushes to _registeredSpanProcessors, which OTel SDK 2.x's BasicTracerProvider no longer reads. register({ batch: true }) goes through that path (src/otel.ts:601).
  • register() installs no context manager, so instrumented spans don't nest under a manual startActiveSpan root unless the app sets one.

Round 3 (2026-10-19)

An independent re-review found stripStackTrace / sanitizeStatus quadratic in message size (dropRepeatedStackHeader sliced and regex-matched the rest of the text for every "\n\n"). Fixed in 5f73cd9:

  • dropRepeatedStackHeader walks the text once with forward-only indexes; the message is compared only when its length fits the tail. Same results as before for multi-line messages, relayed Python tracebacks and frames.
  • The Exception Group regex (quadratic on long whitespace runs; 81 s at 64 KB) is a suffix check; the frame search walks lines with indexOf.
  • Link attributes (SpanOptions.links, addLink, addLinks) are now masked with maskAttributes().
  • New tests: 1 MB bounds (CPU time, under 200 ms) for repeated header paragraphs, blank lines, long line, many colons, many tracebacks, long indented bodies, and a 1 MB message through setStatus (default mode) and recordException; a link masking test.

CPU time at 1 MB, after: 0-17 ms per shape. Before (64 KB): repeated header 0.7 s, different paragraphs 1.2 s, whitespace run before a traceback header 81 s.

Tests: fi-core 142 pass (was 124), openai 93, anthropic 53 (3 skipped), bedrock 8, langchain 36, openai_agents 14. openai.test.ts still fails to compile (zod 4, pre-existing).

🤖 Generated with Claude Code

Stored-row verification (2026-10-19, head 5f73cd9)

Ran a real e2e against a local Future AGI stack. It has 4 failing spans:

  • an OpenAI bad-model call through a proxy;
  • a LangChain ChatOpenAI failure;
  • a LangChain chain whose step throws an error quoting the prompt.

I then scanned every stored ClickHouse spans column of every row. fi_verify passed in all three modes.

Mode Error spans (4) Prompt marker in stored rows Traceback / local path
default message-only descriptions; error.type = Error / model_not_found in content fields, as expected none
FI_HIDE_INPUTS=true empty descriptions, error.type kept only in the prompt-template span's output (an output, not hidden by inputs-only) none
FI_HIDE_INPUTS + FI_HIDE_OUTPUTS empty descriptions, error.type kept 0 none

The model name appears only in model fields, never in error text.

Merged dev (2026-10-21, 0a4653e)

dev merged in with no conflicts (merge commit 0a4653e). It brings only Python and CI changes; typescript/, docs/ and the root CHANGELOG.md are byte-identical to 5f73cd9. Re-run on the merge (Node 26.8.1, pnpm 10.30.1, frozen lockfile):

  • fi-core 142 passed; openai 93 passed; anthropic 53 passed, 3 skipped; bedrock 8; langchain 36; openai-agents 14.
  • openai.test.ts still fails to compile under ts-jest diagnostics (zod 4, pre-existing). Correction to the Tests section above: with diagnostics off it gives 17 passed, 2 skipped and 1 failed, not 18/18. The failure is creates a span for chat completions parse: under zod 4, zodResponseFormat emits an empty JSON schema, so the inline snapshot does not match. This PR does not touch that test or the lockfile, so it fails the same way on dev. The snapshot this PR fixed (should respect a trace config and mask attributes accordingly) passes.

Nothing here is on npm yet (@traceai/fi-core latest 1.0.0, openai 0.1.13, anthropic 0.1.2, bedrock 0.1.0, langchain 0.1.2, openai-agents 0.1.0), so the bumped versions are free.

nik13 and others added 4 commits October 18, 2026 20:34
Error text is content. Provider error bodies quote the prompt back and carry
request/session ids; JavaScript stacks carry local file paths. With content
hidden, traceAI-ts still exported them, and LangChain put run.error (message
plus the full stack) in the status description.

Core (@traceai/fi-core 1.1.0):
- TraceConfig.hideErrorDetails / FI_HIDE_ERROR_DETAILS. Unset, it is on when
  any of hideInputs, hideOutputs, hideInputMessages, hideOutputMessages,
  hideInputText, hideOutputText is on; an explicit true/false wins.
- Enforced once, in FISpan: recordException, setStatus, addEvent and
  setAttribute(s). When on, ERROR has no description, exception events keep
  only exception.type/escaped, and error.message / exception.* /
  *.error.message keys are dropped.
- Always: error.type on ERROR spans (provider code, else error class), and an
  ERROR description that is the exception message only, never a stack trace.
- FITracer.startActiveSpan makes the FISpan the active span, so
  trace.getActiveSpan() goes through the policy.
- FI_HIDE_* env vars now apply when no traceConfig is passed (as in Python).

LangChain (@traceai/langchain 0.1.3): the status description comes from the
error passed to handle*Error, not run.error; the stack stays only on the
exception event.

OpenAI Agents (@traceai/openai-agents 0.1.1): the status description is the
error message; error data moves to the exception event.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Security fix that predates this PR: FITracer.startSpan passed
options.attributes straight to the SDK span, so attributes given at span
creation skipped FISpan and every masking rule (hide flags, PII redaction,
error-detail hiding). About 22 instrumentors put the prompt there, OpenAI
included, so with FI_HIDE_INPUTS=true the prompt was still exported in
input.value, gen_ai.input.messages and raw.input. This dates from May 2025,
so every published @traceai/fi-core release has it.

@traceai/fi-core (1.1.0, unreleased; npm latest is 1.0.0):
- startSpan and startActiveSpan run creation-time attributes through
  maskAttributes(), the same masking FISpan.setAttributes uses. The masked
  attributes are still passed at creation, so samplers see what is exported.
  An error.type passed at creation is kept.
- hideInputs also redacts raw.input and llm.prompt_template.variables;
  hideOutputs also redacts raw.output.
- The error-detail key filter is now the first masking rule, so every
  attribute path shares it.
- Hidden mode filters events with a denylist (Python's strip_error_details):
  exception.message, exception.stacktrace, error.message and the
  *.error.message-style keys are dropped, everything else is kept.
- exception.type is the same in both modes. FISpan.recordException builds
  the event itself: a string code that is an identifier, else the error
  name / class. A numeric code (HTTP 400) is no longer the type.
- stripStackTrace keeps a multi-line message whole when the description is
  a full err.stack, and keeps a relayed Python traceback's final
  "Type: message" line (chained tracebacks too).
- The openai.test.ts trace-config snapshot expected input.value and
  raw.input unmasked with hideInputs: true. It now expects them redacted.

@traceai/openai-agents: span errors are recorded as AgentSpanError, so
exception.type / error.type are not missing or _OTHER.

@traceai/bedrock 0.1.1: non-Error failures (cross-realm errors, strings,
plain objects) get recordException, error.type and a message-only status
instead of String(error).

@traceai/openai 0.1.14, @traceai/anthropic 0.1.3: version bumps only. Their
published versions depend on @traceai/fi-core ^0.1.x, so they keep the bug
until they are republished against 1.1.0.

Tests: new hidden-mode tests assert the prompt is in no exported attribute,
event or status for startSpan, startActiveSpan, the sampler, the real
OpenAI SDK and the real LangChain instrumentation (against local HTTP
servers), and Bedrock error paths.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
dropRepeatedStackHeader sliced and regex-matched the rest of the message for every blank-line
separator (quadratic: about 165 s at 900 KB). It now walks the text once with forward-only
indexes and compares the message only when its length can fit. The Exception Group regex,
which was quadratic on long whitespace runs, is a suffix check, and the frame search walks
lines with indexOf instead of splitting.

Link attributes (SpanOptions.links, addLink, addLinks) go through maskAttributes().

Adds 1 MB performance tests (CPU time) for each message shape and a link masking test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@nik13
nik13 marked this pull request as ready for review October 21, 2026 23:43
nik13 added a commit that referenced this pull request Oct 22, 2026
The repo now releases dev -> main (#283), and #271/#272 target dev, so
this PR moves to dev too. Changing the base does not start the
pull_request workflows; this empty commit does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nik13 and others added 2 commits October 26, 2026 01:42
Conflicts in typescript/packages/traceai_openai_agents (version only):
dev has the unreleased @traceai/openai-agents 0.2.0 rewrite and this PR
bumped the package to 0.1.1. The PR's change now ships as the next free
version, 0.2.1 (package.json, src/version.ts, package CHANGELOG entry
above dev's 0.2.0, root CHANGELOG heading). instrumentation.ts and its
test merged without conflicts; the AgentSpanError / message-only status
change applies unchanged on top of dev's rewrite.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The dev merge moved @traceai/openai-agents to 0.2.1; the VERSION test
still pinned dev's 0.2.0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@nik13
nik13 merged commit e90ffe7 into dev Oct 26, 2026
4 checks passed
@nik13
nik13 deleted the feat/hide-error-details-ts branch October 26, 2026 02:41
jabrailkhalil pushed a commit to jabrailkhalil/traceAI that referenced this pull request Nov 1, 2026
…TracerProvider()

InstrumentationBase calls enable() from its constructor, before the
subclass stores traceConfig, so the FITracer was always built with
traceConfig undefined. fi-core returns its defaults for undefined
without reading FI_HIDE_INPUTS / FI_HIDE_OUTPUTS, so neither the option
nor the environment variables redacted anything.

The FITracer also kept the tracer it got at that point. A provider set
with setTracerProvider() afterwards (the order the README shows) was
ignored unless enable() ran again, and spans went to the global
provider instead.

Build the FITracer again once the config is stored and when the module
is patched, and always pass an options object so fi-core reads the
environment. Three new real-SDK tests cover the README order,
traceConfig redaction and FI_HIDE_* redaction; all three fail without
this change. raw.input / raw.output are still exported by fi-core 1.0.0
(fixed in future-agi#271); the README says so.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant