feat(python): hide error details with the content hide flags - #272

What does this PR do?

Treats error text as content on the Python side. When content is hidden, failed spans keep status ERROR, error.type and exception.type, and drop the status description, exception.message, exception.stacktrace, error.message and the provider error response. When content is shown, the ERROR description is the exception message only, never a traceback.

Behaviour change for users of the hide flags: setting any of hide_inputs, hide_outputs, hide_input_messages, hide_output_messages, hide_input_text or hide_output_text (or their FI_HIDE_* env vars) now also hides error details. Set hide_error_details=False / FI_HIDE_ERROR_DETAILS=false to keep the old behaviour.

Why?

Decision (2026-10-18): do what the other tracing SDKs do. OpenInference, OpenLLMetry, OpenLIT, OTel util-genai, Langfuse, LangSmith, Braintrust and Datadog all keep error text when content capture is off. Pydantic AI was the exception and got GHSA-4x9p-g9wm-8q7f (CVE-2026-107291) for it. Its fix keeps only exception.type on the exception event and sets ERROR with no description when include_content=False. The OTel spec also says the status description SHOULD NOT carry sensitive details, and that exception.message may contain sensitive information.

Leaks we hit with content hidden:

• (a) A LangChain failure stored repr(error) plus the full Python traceback (including a local /private/tmp path) in status_message (TH-8226 AgentCore recipe). traceAI got this from OpenInference: status.description = run.error.
• (b) A Vercel AI Gateway error body with provider request ids and a gateway session id landed in the status message.
• (c) Provider error bodies can quote the user's prompt back.

How

Core, fi-instrumentation-otel 1.1.0 → 1.2.0

• TraceConfig.hide_error_details / FI_HIDE_ERROR_DETAILS has three states. If neither the argument nor the env var is set, it is derived: on when any of the six content hide flags above is on. An explicit true/false, from the constructor or the env var, wins. The raw setting lives in a private field (_hide_error_details) and hide_error_details is a property that resolves it, so dataclasses.replace(TraceConfig(), hide_inputs=True).hide_error_details is True, while an explicit value survives replace(). The constructor argument, the attribute and the env var are unchanged.
• The rule is enforced in one place, FiSpan (_spans.py), through record_exception, add_event("exception"), set_status, set_attribute(s) / TraceConfig.mask, and __enter__/__exit__. That also covers OpenTelemetry's own use_span / start_as_current_span exception handling (which builds "Type: msg" and calls record_exception) and with tracer.start_span(...) as span:. That block used to yield the raw SDK span; it now yields the FiSpan.
• When error details are hidden: ERROR has no description. Exception events keep only exception.type and exception.escaped. error.message, exception.message, exception.stacktrace and any key ending in .exception.message, .exception.stacktrace, .error.message or .error_message are dropped (for example autogen.error.message and autogen.tool.error_message).
• Always, whatever the flag:

  • error.type is set only on a span that ends with status ERROR (checked in end()), from the exception being handled when the status was set, or else the last recorded one. An exception recorded on a span that still ends OK, such as a recovered retry, does not set it (OTel semconv: error.type marks a failed operation).

  • The provider error code is read only from exceptions of known provider SDKs, or subclasses of them (the class MRO is checked, so langchain-openai's OpenAIInvalidRequestError, a subclass of openai.BadRequestError, still maps):

    SDK	read from	example
    openai, groq, anthropic	code, body["error"]["code"/"type"], type	model_not_found, invalid_request_error, not_found_error
    botocore, boto3, aiobotocore	response["Error"]["Code"]	ValidationException
    google-genai	status	NOT_FOUND
    google-api-core	grpc_status_code.name	NOT_FOUND
    mistralai	JSON body type/code	invalid_model
    httpx / httpx2 HTTPStatusError	response.status_code	"404"

    Everything else gets the fully qualified exception class. A generic .code / .status / .type lookup is not used, because other libraries put short codes there too (SQLAlchemy e3q8, Django ValidationError invalid). cohere errors carry only a status code and a free-text body, so they get the class name. Free text is never used as a code. An instrumentor's own error.type is not overwritten.

  • The ERROR description is reduced to str(exc): "Type: msg" becomes msg, also when one such line is joined per exception (llama-index _callback.py), and anything from Traceback (most recent call last) onwards is cut.

  • FiSpan keeps only a summary (class names and message) of each recorded exception, and clears it in end(). It never holds the exception, so traceback frames and their locals are not kept alive.

• fi_instrumentation/instrumentation/version.py now says 1.2.0, like pyproject.toml (it still said 0.1.19).
• New _errors.py holds the helpers. Docs updated: docs/configuration/trace-config.md, environment-variables.md and python/README.md.

LangChain, traceAI-langchain 0.2.0 → 0.2.1

• The status description is built from the error object passed to on_*_error, not from run.error (repr plus the traceback). The stacktrace stays only in exception.stacktrace on the exception event. Non-Exception errors record their own traceback, not traceback.format_exc().
• When error details are hidden, a failed run records no output attributes. The end-to-end run caught this one: langchain-core fills a failed chat-model run's outputs with the provider error response (body, response headers, request ids, AI-gateway metadata). With FI_HIDE_INPUTS=true, the ChatOpenAI span still stored that response in output.value.

AutoGen, traceAI-autogen 0.2.0 → 0.2.1

• The instrumentor built an FITracer but then created every agent, team and tool span with a separate raw tracer, so TraceConfig never applied to them. These spans now use the FITracer.

Instrumentor audit

Every instrumentor that builds its spans through FITracer is covered by the FiSpan enforcement, with no per-instrumentor change: anthropic, bedrock, cerebras, cohere, crewai, deepseek, dspy, fireworks, google-adk, google-genai, groq, guardrails, haystack, huggingface, instructor, langchain, litellm, llama_index, mistralai, ollama, openai, openai-agents, portkey, smolagents, together, vertexai, vllm and xai. I checked the named ones by reading their code:

• OpenAI: _WithSpan / _stream.py.
• Anthropic: _wrappers.py sets str(exception) from inside the except.
• LiteLLM: start_as_current_span plus record_exception.
• Bedrock: _wrappers.py, _response_accumulator.py, utils._finish, and the invoke_model path that ends the span before re-raising. error.type still lands: set_status runs inside the except, and the span ends through FiSpan.end() with status ERROR.
• Google GenAI: _wrappers.py / _stream.py.

None of these hold raw spans or format tracebacks themselves. The only traceback.format_* call left in python/frameworks is LangChain's exception event.

Fixed here: autogen (raw tracer, above) and langchain (run.error and failed-run outputs).

Follow-ups (not in this PR):

1. Instrumentors that don't use FITracer. They take no TraceConfig, so neither the content hide flags nor hide_error_details apply to them. They set str(e) / f"{type(e).__name__}: {e}" status descriptions and call record_exception on raw spans, which keeps the message and the stacktrace:

   • a2a/traceai_a2a/_a2a_client.py, a2a/traceai_a2a/_a2a_server.py
   • beeai/traceai_beeai/_middleware.py
   • strands/traceai_strands/_callback.py
   • claude-agent-sdk/traceai_claude_agent_sdk/{_hooks,_client_wrapper,_subagent_tracker}.py, which also set claude_agent.error.message and claude_agent.tool.error_message
   • pydantic-ai/traceai_pydantic_ai/{_instrumentor,_agent_wrapper}.py, which also set pydantic_ai.error.message and pydantic_ai.tool.error_message
   • vector stores: chromadb, lancedb, milvus, mongodb-vector, pgvector, pinecone, qdrant, redis-vector and weaviate (_wrappers.py in each)
   • agno and pipecat export the framework's own OTel spans as they are; mcp only propagates context.
   • livekit/traceai_livekit rewrites livekit-agents' own OTel spans at the exporter and takes no TraceConfig, so error text and content on those spans pass through untouched.

   Fix: build these spans through FITracer, or scrub at export time in register().

2. autogen content keys. autogen.input.last_message, autogen.output.content, autogen.task.content and autogen.tool.args are custom keys, so hide_inputs / hide_outputs don't mask them.

3. openai-agents. A failed Agents-SDK span gets the description "{message}: {data}" from the SDK's SpanError. There is no Python exception, so it gets no error.type. Hidden mode drops the description; default mode keeps data.

4. LiteLLM streaming, pre-existing and unrelated to hiding. On the sync path, an error before the first chunk records the exception but never sets ERROR or ends the span. The async path calls span_cm.__exit__(e, type(e), tb) with the arguments swapped.

How was it tested?

• Unit tests:

  • python/tests/test_error_details.py (116 tests):
    • derivation: each of the six flags, as an argument and as an env var, implies hide_error_details; non-content flags don't; an explicit or env false overrides; the constructor beats the env; an invalid env value falls back to the derived value
    • dataclasses.replace(): derives it again for each flag, keeps an explicit or env value, takes an explicit hide_error_details; positional arguments still work
    • mask
    • error.type extraction: each provider SDK above (including subclasses and the class-name fallbacks), and SQLAlchemy / Django / other libraries with short .code / .status values getting the class name
    • error.type only on failed spans: recovered retry ending OK, no status, exception event without ERROR, ERROR then OK, retry then failure (the failing exception wins), status set outside except (LangChain), status before record_exception, hidden mode
    • the exception and its frames are not held (weakref is dead before end())
    • description clean-up, including one "Type: msg" line per exception
    • record_exception, set_status (in both forms), add_event("exception") and set_attribute
    • use_span / start_as_current_span auto-recording, the span used as a context manager, the decorator path, and suppressed spans
  • python/frameworks/langchain/tests/test_error_status.py (11 tests): message-only description on chain and LLM runs, hidden mode (derived and explicit), explicit false, the run.error fallback, the bare-span check on _update_span, and failed-run outputs (kept by default, dropped when hidden). I checked that the new output tests fail without the fix. Its fake provider error now claims the openai module, since codes are read only from provider SDKs.
  • 23 of the new core tests fail on the previous commit (3f43aa2), as expected: the replace(), provider-only, failed-span-only, frames-held and multi-line cases.

• Suites:

  • Core: pytest tests --ignore-glob='tests/test_framework_*' --ignore=tests/test_otel.py gives 256 passed, 1 skipped and 12 failed. The same 12 fail on origin/dev (test_batch_span_processor_tuning ×3, test_context_attributes ×8, test_settings ×1), and tests/test_otel.py fails to import on origin/dev too.
  • LangChain: 58 passed, 1 skipped.
  • AutoGen: 129 passed, 12 failed, the same 12 as before this change (all mocked-OTel tests).
  • test_framework_{openai,langchain}.py, test_tracers.py, test_config.py, test_error_details.py: 206 passed, 4 failed; the same 4 fail on 3f43aa2.
  • test_error_details.py, test_config.py and test_tracers.py also pass on the minimum OpenTelemetry 1.29.

• ruff check reports no new rule categories on the changed files, only more of the Optional[...] style hints (UP045) that the Python 3.9 floor requires. mypy passes on _errors.py, _spans.py and config.py.

• Real SDK exceptions → error.type: OpenAI through CLIProxy (model_not_found), Anthropic through CLIProxy (invalid_request_error, which is what CLIProxy returns for an unknown Claude model) and against an Anthropic-shaped 404 (not_found_error), the real Gemini API with an unknown model (NOT_FOUND), groq / mistralai / cohere against a local server returning their documented error bodies (model_not_found / invalid_model / class name), botocore bedrock-runtime via Stubber (ValidationException), google.api_core.exceptions.NotFound (NOT_FOUND), httpx raise_for_status() ("503"), and real sqlalchemy.exc.OperationalError (.code == "e3q8") and Django ValidationError(code="invalid"), which both get the class name.

• End-to-end against the local Future AGI (project th8103-hideerr-python). Each run has one root span with three real failures:

  1. the OpenAI SDK calling a bad model through the local CLIProxy, which returns 400 model_not_found
  2. a LangChain ChatOpenAI chain calling the same bad model
  3. an FITracer.start_as_current_span block that raises with a local /private/tmp path in the message, recorded by use_span

  Stored ClickHouse rows:

  run	LangChain chain status_message	use_span status_message	error.type	traceback in row	/private/tmp in row	error body / response headers in row
  origin/dev, FI_HIDE_INPUTS=true	OpenAIInvalidRequestError("Error code: 400 ...")Traceback (most recent call last): ... (5,516 chars)	FileNotFoundError: [Errno 2] ... '/private/tmp/...' (prompt)	none	yes	yes	yes (ChatOpenAI output.value had the body and the response headers)
  this PR, default	Error code: 400 - {'error': {... 'code': 'model_not_found' ...}} (message only)	[Errno 2] No such file or directory: '/private/tmp/...' (message only)	model_not_found / FileNotFoundError	no	message only, as intended	kept (default unchanged)
  this PR, FI_HIDE_INPUTS=true	empty	empty	model_not_found / FileNotFoundError	no	no	no

  All 10 failed spans were stored with status ERROR in both runs. The local backend keeps only event names and timestamps in span_events, so I checked the exception-event attributes on the spans exported in the same runs. Default mode had exception.type/message/stacktrace/escaped. Hidden mode had exception.type/escaped only.

• End-to-end after the review follow-ups (head c845102, project th8103-hideerr-python-r2): the three failures above plus four new cases. Stored ClickHouse rows:

  span	default status_message	hidden status_message	error.type (both runs)
  OpenAI ChatCompletion (×2), LangChain chain, ChatOpenAI	message only	empty	model_not_found
  use_span with a /private/tmp path	message only	empty	FileNotFoundError
  recovered retry: TimeoutError recorded, span ends OK	(status OK)	(status OK)	none
  real Gemini call to an unknown model in use_span	404 NOT_FOUND. {...}	empty	NOT_FOUND
  real sqlalchemy.exc.OperationalError (.code = e3q8)	message only	empty	sqlalchemy.exc.OperationalError
  two exceptions recorded, llama-index-style joined status	first failure: ...\n'missing_key' (no type prefixes)	empty	KeyError

  A scan of every stored column (status_message, input, output, attrs_string, span_events, attributes_extra, tags) of the hidden run's 10 rows found no prompt text, path, traceback, SQL, provider message or response_metadata. The default run has no traceback in any column.

Notes for reviewers

• traceAI-langchain and traceAI-autogen keep their fi-instrumentation-otel pins. The LangChain change reads hide_error_details with getattr, so it also works with core 1.1.x, where hiding just isn't available. Publish core 1.2.0 first, or together with them.
• FiSpan.set_status takes the exception from sys.exc_info() when it is called inside an except block (use_span and most instrumentors do this), or else from the last record_exception. Order doesn't matter. error.type itself is written in FiSpan.end(), and only when the span's final status (span.status) is ERROR.
• TraceConfig.__init__ is now written by hand: the public argument hide_error_details fills the private field, and replace() passes the private field by name. dataclasses.fields(TraceConfig) and the repr show _hide_error_details (the raw value). mypy's dataclass plugin types dataclasses.replace() by field name, so replace(config, hide_error_details=...) works at runtime but mypy flags it; the constructor and the attribute type-check as before.
• The TypeScript side is a separate PR (feat/hide-error-details-ts).

🤖 Generated with Claude Code

Round 3 (2026-10-19)

• Performance and privacy fix. clean_error_description built a regex containing each exception message and ran it on every ERROR set_status in default mode (10 KB: 8 ms, 100 KB: 0.83 s, 1 MB: 6.4 s; a LangChain parser error carrying 100 KB went from 9 ms to 0.58 s in chain.invoke), and every distinct message stayed in the global re cache after the span ended. It now strips the known <Type>: line prefixes with str.find / startswith; message text never goes into a regex (100 KB: 0.0001 s). Tests: 1 MB message under 0.1 s, many 100 KB messages, and the re cache does not grow over 100 distinct messages.
• Content-hiding fix. hide_inputs now also redacts raw.input and the prompt template variables (llm.prompt_template.variables, gen_ai.prompt.template.variables); hide_outputs also redacts raw.output. Same keys as @traceai/fi-core 1.1.0 (traceAI#271). Before, a span carrying these kept the prompt in the stored row with the flag on. Unit tests per key; CHANGELOG entry.
• litellm claim corrected. The table above and the code comment said litellm carries code/body. Real litellm 1.106 exceptions have code=None, body=None, type=None, so a litellm 404 gets litellm.exceptions.NotFoundError (class name). litellm is no longer listed as a provider-code reader in the docs; no special-casing added.
• error.type fallback. An ERROR status with nothing to name gets _OTHER (OTel semconv). The fallback no longer uses a stale recovered exception: order is the in-flight exception (sys.exc_info() at set_status), then an exception recorded with escaped=True on this span, then the newest recorded exception the status description mentions (the LangChain pattern: record in a callback, set the status later), else _OTHER. Recovered retries still leave the span OK without error.type; use_span still names the exception. Tests added for each case.

Stored-row verification (2026-10-19, head 78c56ae)

Ran a real e2e against a local Future AGI stack: a LangChain parser error with a ~230 KB message quoting the prompt, an OpenAI bad-model call through a proxy, a span carrying raw.input/raw.output/template variables, and a use_span failure on a path. I then scanned every stored ClickHouse spans column of every row. fi_verify passed in all three modes.

chain.invoke CPU time was 0.05–0.07 s with the 230 KB message. Wall time was dominated by slow exports on a heavily loaded machine.

Merged dev (2026-10-21, 99d1a55)

The branch was 41 commits behind dev and CI had never run on it (the Python tests workflow reached dev after the last push). dev merged in with no conflicts (merge commit 99d1a55). Local checks on the merge, Python 3.11, Poetry 2.1.2 with the locks CI uses:

• fi_instrumentation (CI's command, pytest tests --ignore-glob='tests/test_framework_*.py'): 337 passed.
• langchain against this core: own tests 58 passed, 1 skipped; tests/test_framework_langchain.py 32 passed, 2 failed. Those 2 are the ones .github/python-tests-known-broken.txt lists (dependency count, a set_attributes call).
• autogen against this core: 12 own + 5 core failures, exactly the 17 the known-broken list names. 129 own tests pass.

Version overlap with #281 (base main, bumps the same package to 1.1.1): the two PRs conflict in python/pyproject.toml and python/CHANGELOG.md. Whichever merges second sets the version in all three files (pyproject.toml, CHANGELOG.md, fi_instrumentation/instrumentation/version.py): 1.2.0 if this PR is second, 1.2.1 for #281 if it is second. Nothing here is on PyPI yet (fi-instrumentation-otel 1.1.0, traceAI-langchain 0.2.0, traceAI-autogen 0.2.0 are the latest).