Skip to content

Commit 5b9a6b5

Browse files
nik13claude
andcommitted
fix(ag2): follow TraceConfig privacy default; never export paths, stack traces or raw error text (TH-8236)
Content is captured by default and hidden with FI_HIDE_* / TraceConfig like every traceAI instrumentor; capture_content now defaults to True (AG2's own default). Tracebacks and absolute local paths are removed from content attributes, ERROR status descriptions carry the exception class only, exception events keep exception.type only. Verified on a real run against local Future AGI. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 381d7b1 commit 5b9a6b5

7 files changed

Lines changed: 261 additions & 43 deletions

File tree

‎python/frameworks/ag2/CHANGELOG.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,16 @@
22

33
## [0.1.0] - unreleased
44

5+
- Privacy follows the traceAI `TraceConfig` convention (re-verification 2026-10-16):
6+
content is captured by default and hidden with `FI_HIDE_*` / `TraceConfig`;
7+
`capture_content` now defaults to `True` (AG2's own default) instead of a
8+
second, package-specific switch. In both modes local paths, stack traces
9+
and raw exception text are no longer exported: tracebacks in tool messages
10+
are cut, absolute paths become `[path]`, an ERROR status carries only the
11+
exception class name, and exception events keep `exception.type`.
12+
513
- New package for AG2 1.x (PyPI `ag2`, import `ag2`). `setup()` attaches AG2's
6-
own `TelemetryMiddleware` with `capture_content=False` and registers the
14+
own `TelemetryMiddleware` and registers the
715
Future AGI exporter; `AG2SpanProcessor` sets `gen_ai.span.kind` from
816
`gen_ai.operation.name` and aliases AG2's cache/thinking usage keys
917
(originals kept). Does not wrap `Agent` and never imports `autogen`.

‎python/frameworks/ag2/README.md‎

Lines changed: 31 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ This package does not wrap `Agent`. It:
2525

2626
1. registers the Future AGI exporter through `fi_instrumentation.register()`
2727
(or uses the provider you pass),
28-
2. attaches `TelemetryMiddleware` to your agents with `capture_content=False`
29-
through AG2's public `Agent.add_middleware`,
28+
2. attaches `TelemetryMiddleware` to your agents through AG2's public
29+
`Agent.add_middleware` (content is captured by default, like every traceAI
30+
instrumentor; hide it with `FI_HIDE_*` / `TraceConfig`),
3031
3. installs `AG2SpanProcessor`, which, on spans from AG2's
3132
`opentelemetry.instrumentation.ag2` scope only:
3233
- sets `gen_ai.span.kind` from `gen_ai.operation.name`
@@ -46,8 +47,8 @@ This package does not wrap `Agent`. It:
4647
span that repeats a chat span's usage (see below), they move to
4748
`ag2.usage.*` with the token counts.
4849

49-
- applies `TraceConfig` (`FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS`, ...) as a
50-
second content gate.
50+
- applies `TraceConfig` (`FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS`, ...), and
51+
removes local paths, stack traces and raw error text (see Privacy).
5152

5253
## Set up
5354

@@ -70,7 +71,7 @@ from traceai_ag2 import setup
7071
trace_provider = register(project_type=ProjectType.OBSERVE, project_name="my-chatbot")
7172

7273
agent = Agent("assistant", "You are helpful.", config=OpenAIConfig(model="gpt-4o-mini"))
73-
setup(agent, tracer_provider=trace_provider) # capture_content=False
74+
setup(agent, tracer_provider=trace_provider) # content captured; FI_HIDE_* hides it
7475

7576
reply = await agent.ask("Hello") # inside an async function
7677
trace_provider.force_flush() # short scripts: flush before exit
@@ -130,17 +131,29 @@ An offline example with a scripted model (no model API key) is in
130131

131132
## Privacy
132133

133-
Content is off unless you opt in. Upstream `TelemetryMiddleware` defaults
134-
`capture_content=True`; `setup` and `create_telemetry_middleware` pass `False`.
135-
With it off, spans carry no prompts, completions, tool arguments, tool results
136-
or human-input text.
134+
This package follows the traceAI `TraceConfig` convention: prompts, replies,
135+
tool arguments and tool results are captured by default, and hidden with the
136+
standard variables (or `TraceConfig(...)`):
137137

138-
```python
139-
setup(agent, tracer_provider=trace_provider, capture_content=True) # opt in
138+
```bash
139+
export FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true # or FI_HIDE_INPUT_MESSAGES, FI_HIDE_OUTPUT_MESSAGES, FI_HIDE_INPUT_TEXT, ...
140140
```
141141

142-
With content on, `TraceConfig` (or the matching `FI_HIDE_*` variables) still
143-
removes it in the span processor. AG2 records message content as one JSON
142+
There is no second switch. `capture_content` is AG2's own `TelemetryMiddleware`
143+
argument and defaults to `True`, as upstream; `setup(..., capture_content=False)`
144+
stops AG2 itself from recording content, but the supported way to hide it is
145+
`FI_HIDE_*`.
146+
147+
Never exported, in either mode: local file paths, user names, stack traces and
148+
raw exception text. Tracebacks that AG2 puts into a tool message (a failing
149+
tool's error is sent back to the model and recorded in the next chat span's
150+
input) are cut to `[stack trace removed]`, absolute local paths become
151+
`[path]`, a failing span's status description is the exception class name
152+
(for example `RuntimeError`) instead of `str(exc)`, and exception events keep
153+
`exception.type` only.
154+
155+
`TraceConfig` (or the matching `FI_HIDE_*` variables) removes it in the span
156+
processor. AG2 records message content as one JSON
144157
string per span (`gen_ai.input.messages`, `gen_ai.output.messages`), not as
145158
the per-message keys that `TraceConfig.mask`'s text and image rules match, so
146159
the processor drops whole attributes:
@@ -168,8 +181,8 @@ not copied to `input.value` / `output.value` either.
168181
| Tokens | `gen_ai.usage.input_tokens` / `output_tokens` pass through, except on `anthropic` spans, where cache read and write tokens are added to `input_tokens` (and `total_tokens` recomputed when present; see the note below). Cache tokens are aliased to `gen_ai.usage.cache_read_tokens` / `cache_write_tokens` and thinking tokens to `gen_ai.usage.reasoning.output_tokens`, originals kept | When the usage field is zero/absent. Cost is not emitted |
169182
| Session | Not emitted by AG2. `session.id` from traceAI's `using_session` / `using_attributes` context, which wins over a static value; or set it yourself: `setup(..., span_attributes={"session.id": "..."})` | When neither is set |
170183
| User, metadata, tags | `user.id`, `metadata`, `tag.tags` (and the other traceAI context keys) from `using_user` / `using_metadata` / `using_tags` / `using_attributes`; a key AG2 or `span_attributes` already set is not overridden | When the context does not set them |
171-
| Tools | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`; arguments/result only with `capture_content=True` | |
172-
| Input / output | `input.value` / `output.value` (what Observe shows as a span's input and output), copied from the captured chat messages, tool arguments and result, or human input. A single chat completion goes on `output.value` as its message object, which Observe's chat view shows as the assistant reply. `invoke_agent` gets the first chat input and the last chat output of its turn, so the trace list shows the question and the answer | Without `capture_content=True`, when a `TraceConfig` flag hides the source, and always on `record_usage` spans (AG2 records no content there) |
184+
| Tools | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`; arguments/result unless hidden by `FI_HIDE_*` | |
185+
| Input / output | `input.value` / `output.value` (what Observe shows as a span's input and output), copied from the captured chat messages, tool arguments and result, or human input. A single chat completion goes on `output.value` as its message object, which Observe's chat view shows as the assistant reply. `invoke_agent` gets the first chat input and the last chat output of its turn, so the trace list shows the question and the answer | When a `TraceConfig` flag hides the source, and always on `record_usage` spans (AG2 records no content there) |
173186
| Errors | OTel status `ERROR` on failing tool / model / agent spans | |
174187

175188
Notes:
@@ -254,7 +267,8 @@ Notes:
254267
concurrently (`TracerProvider(active_span_processor=ConcurrentMultiSpanProcessor())`;
255268
`setup` logs a warning), so the exporter can read a span before
256269
`AG2SpanProcessor` normalizes it. Use the default synchronous processor.
257-
- Content appeared without opting in: you constructed `TelemetryMiddleware`
258-
yourself. Pass `capture_content=False`, or use `create_telemetry_middleware`.
270+
- Content is missing: a `FI_HIDE_*` variable or `TraceConfig` hide flag is set.
271+
- Content you wanted hidden still appears: you constructed `TelemetryMiddleware`
272+
yourself on a provider without `AG2SpanProcessor`; use `setup()`.
259273
- This package never imports `autogen`. If your code imports `autogen`, you
260274
are on AG2 Classic or Microsoft AutoGen; use that integration instead.

‎python/frameworks/ag2/tests/test_contract_harness.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@
2828
MODEL,
2929
PROVIDER,
3030
TOOL_CALL_ID,
31+
TOOL_RESULT,
32+
USER_PROMPT,
3133
ask,
3234
assert_no_content,
3335
expected_one_tool_kinds,
@@ -65,7 +67,7 @@ def test_one_tool_run_reaches_collector_path_with_normalized_gen_ai_spans(fi_env
6567
)
6668
try:
6769
agent = weather_agent()
68-
setup(agent, tracer_provider=trace_provider) # capture_content defaults off
70+
setup(agent, tracer_provider=trace_provider) # content captured by default (TraceConfig)
6971
reply = ask(agent)
7072
assert reply.body
7173
assert trace_provider.force_flush(timeout_millis=10_000)
@@ -116,10 +118,12 @@ def test_one_tool_run_reaches_collector_path_with_normalized_gen_ai_spans(fi_env
116118
)
117119
assert promoted_input == 11, "trace-level input tokens must count the model call once"
118120

119-
# session: AG2 emits none; content: capture is off
121+
# session: AG2 emits none; content: captured by default (TraceConfig)
120122
for name, attributes in by_name.items():
121123
assert "session.id" not in attributes, name
122-
assert_no_content(attributes, name)
124+
assert USER_PROMPT in chat["gen_ai.input.messages"]
125+
assert by_name["execute_tool get_weather"]["gen_ai.tool.call.result"] == TOOL_RESULT
126+
assert by_name["execute_tool get_weather"]["output.value"] == TOOL_RESULT
123127

124128
for span in spans:
125129
assert span.get("status", {}).get("code") not in _STATUS_ERROR, span["name"]
@@ -135,6 +139,8 @@ def test_short_script_flushes_spans_before_exit():
135139
"FI_API_KEY": "placeholder-api-key",
136140
"FI_SECRET_KEY": "placeholder-secret-key",
137141
"FI_PROJECT_NAME": "ag2-contract-script",
142+
"FI_HIDE_INPUTS": "true",
143+
"FI_HIDE_OUTPUTS": "true",
138144
"PYTHONPATH": os.pathsep.join(
139145
[str(_PACKAGE_DIR), str(_PYTHON_DIR), env.get("PYTHONPATH", "")]
140146
),

‎python/frameworks/ag2/tests/test_setup.py‎

Lines changed: 113 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -61,34 +61,69 @@ def test_scope_and_strings_match_installed_ag2():
6161
for key in traceai_ag2.USAGE_KEY_ALIASES:
6262
assert f'"{key}"' in source
6363
params = inspect.signature(telemetry.TelemetryMiddleware.__init__).parameters
64-
assert params["capture_content"].default is True # upstream default the helper overrides
64+
assert params["capture_content"].default is True # upstream default; the helper keeps it
6565
# README: hide_llm_invocation_parameters has nothing to act on because AG2
6666
# sets no invocation-parameter attribute.
6767
assert "gen_ai.request.parameters" not in source + inspect.getsource(_telemetry_consts)
6868

6969

70-
# --- AC-05 privacy default ---------------------------------------------------
70+
# --- privacy: traceAI TraceConfig convention (content on by default) -------
7171

7272

73-
def test_setup_defaults_capture_content_off_and_spans_have_no_content():
73+
def test_setup_defaults_capture_content_on_and_spans_carry_content():
7474
provider, exporter = memory_provider()
7575
agent = weather_agent()
76-
setup(agent, tracer_provider=provider)
76+
setup(agent, tracer_provider=provider, config=TraceConfig())
7777

7878
(mw,) = _telemetry_middlewares(agent)
79-
assert mw._capture_content is False
79+
assert mw._capture_content is True
8080

8181
reply = ask(agent)
8282
assert reply.body == FINAL_ANSWER
8383
spans = exporter.get_finished_spans()
84+
(tool,) = find(spans, "execute_tool")
85+
assert attrs(tool)["gen_ai.tool.call.result"] == TOOL_RESULT
86+
(final_chat,) = find(spans, f"chat {MODEL}")
87+
assert USER_PROMPT in attrs(final_chat)["gen_ai.input.messages"]
88+
89+
90+
def test_default_content_is_hidden_by_trace_config_alone():
91+
"""No second switch: TraceConfig hide flags remove content from a default setup()."""
92+
provider, exporter = memory_provider()
93+
agent = weather_agent()
94+
setup(agent, tracer_provider=provider, config=TraceConfig(hide_inputs=True, hide_outputs=True))
95+
ask(agent)
96+
spans = exporter.get_finished_spans()
97+
assert spans
98+
for span in spans:
99+
assert_no_content(attrs(span), span.name)
100+
101+
102+
def test_default_content_is_hidden_by_fi_hide_env(monkeypatch):
103+
monkeypatch.setenv("FI_HIDE_INPUTS", "true")
104+
monkeypatch.setenv("FI_HIDE_OUTPUTS", "true")
105+
provider, exporter = memory_provider()
106+
agent = weather_agent()
107+
setup(agent, tracer_provider=provider) # config=None -> TraceConfig() reads FI_HIDE_*
108+
ask(agent)
109+
spans = exporter.get_finished_spans()
84110
assert spans
85111
for span in spans:
86112
assert_no_content(attrs(span), span.name)
87113

88114

89-
def test_create_telemetry_middleware_defaults_off():
115+
def test_create_telemetry_middleware_defaults_on():
90116
mw = create_telemetry_middleware()
91-
assert mw._capture_content is False
117+
assert mw._capture_content is True
118+
119+
120+
def test_capture_content_false_passthrough_has_no_content():
121+
provider, exporter = memory_provider()
122+
agent = weather_agent()
123+
setup(agent, tracer_provider=provider, capture_content=False)
124+
ask(agent)
125+
for span in exporter.get_finished_spans():
126+
assert_no_content(attrs(span), span.name)
92127

93128

94129
def test_capture_content_opt_in_forwards_true():
@@ -599,7 +634,7 @@ def test_j4_ag2_halt_with_per_call_policies_gives_a_chat_span_without_reason():
599634
_, _, emit_halt = _short_circuit_middlewares()
600635
provider, exporter = memory_provider()
601636
agent = Agent("halted_bot", config=TestConfig("unreachable"), middleware=[emit_halt])
602-
setup(agent, tracer_provider=provider)
637+
setup(agent, tracer_provider=provider, config=TraceConfig(hide_inputs=True, hide_outputs=True))
603638
assert ask(agent, plugins=[Plugin().add_policy(AlertPolicy())]).body == "HALTED: budget exceeded"
604639
spans = exporter.get_finished_spans()
605640
assert [s.name for s in spans] == ["chat", "invoke_agent halted_bot"]
@@ -708,7 +743,7 @@ def test_manual_middleware_with_setup_installing_only_the_processor():
708743
from ag2 import Agent
709744

710745
provider, exporter = memory_provider()
711-
assert setup(tracer_provider=provider) is provider
746+
assert setup(tracer_provider=provider, config=TraceConfig(hide_inputs=True, hide_outputs=True)) is provider
712747
mw = create_telemetry_middleware(tracer_provider=provider, agent_name="late_bot")
713748
agent = Agent("late_bot", config=weather_config(), tools=[get_weather], middleware=[mw])
714749
ask(agent)
@@ -745,3 +780,72 @@ def test_max_tool_result_chars_forwarded_only_when_given():
745780
upstream_default = inspect.signature(TelemetryMiddleware.__init__).parameters["max_tool_result_chars"].default
746781
assert default._max_tool_result_chars == upstream_default
747782
assert create_telemetry_middleware(max_tool_result_chars=16)._max_tool_result_chars == 16
783+
784+
785+
# --- local paths, stack traces and raw error text are never exported --------
786+
787+
788+
def _failing_agent():
789+
from ag2 import Agent
790+
from ag2.events import ModelMessage, ModelResponse, ToolCallEvent
791+
from ag2.testing import TestConfig
792+
from ag2.usage import Usage
793+
794+
def broken(city: str) -> str:
795+
"""Always fails."""
796+
raise RuntimeError("boom-MARK /Users/someone/secret.txt")
797+
798+
return Agent(
799+
"broken_bot",
800+
config=TestConfig(
801+
ToolCallEvent("broken", arguments='{"city": "Paris"}', id="call_b1"),
802+
ModelResponse(ModelMessage("failed"), usage=Usage(prompt_tokens=1, completion_tokens=1), model=MODEL),
803+
),
804+
tools=[broken],
805+
)
806+
807+
808+
@pytest.mark.parametrize("hide", [False, True])
809+
def test_error_text_paths_and_stack_traces_are_not_exported(hide):
810+
provider, exporter = memory_provider()
811+
agent = _failing_agent()
812+
config = TraceConfig(hide_inputs=hide, hide_outputs=hide)
813+
setup(agent, tracer_provider=provider, config=config)
814+
try:
815+
ask(agent)
816+
except RuntimeError: # the scripted model does not catch the tool error
817+
pass
818+
spans = exporter.get_finished_spans()
819+
(tool,) = find(spans, "execute_tool")
820+
assert tool.status.status_code == StatusCode.ERROR
821+
assert tool.status.description == "RuntimeError" # class name only
822+
for span in spans:
823+
for event in span.events:
824+
assert set(event.attributes or {}) <= {"exception.type"}, event.attributes
825+
blob = repr(dict(span.attributes)) + repr(span.status.description) + repr([e.attributes for e in span.events])
826+
for leak in ("/Users/", "secret.txt", "Traceback", "site-packages", "boom-MARK"):
827+
assert leak not in blob, (span.name, leak)
828+
829+
830+
def test_scrub_text_cuts_tracebacks_and_local_paths_but_keeps_urls():
831+
from traceai_ag2._processor import scrub_text
832+
833+
assert "Users" not in scrub_text('File "/Users/me/app.py", line 3')
834+
assert "home" not in scrub_text("open /home/bob/a.txt failed")
835+
assert "bob" not in scrub_text(r"C:\Users\bob\a.py")
836+
assert scrub_text("x Traceback (most recent call last):\n File \"/a/b.py\"\nValueError: q") == "x [stack trace removed]"
837+
assert scrub_text("see https://example.com/home/page and /not/a/known/root") == "see https://example.com/home/page and /not/a/known/root"
838+
839+
840+
def test_traceback_in_a_tool_message_is_cut_from_chat_input_and_input_value():
841+
import json
842+
843+
from traceai_ag2 import normalize_attributes
844+
845+
tb = 'Traceback (most recent call last):\n File "/Users/me/venv/x.py", line 1\nRuntimeError: boom'
846+
messages = json.dumps([{"role": "user", "content": "hi"}, {"role": "tool", "content": tb}])
847+
out = normalize_attributes({"gen_ai.operation.name": "chat", "gen_ai.input.messages": messages}, TraceConfig())
848+
for key in ("gen_ai.input.messages", "input.value"):
849+
assert "Traceback" not in out[key] and "/Users/" not in out[key], key
850+
assert "[stack trace removed]" in out[key]
851+
assert json.loads(out[key])[0]["content"] == "hi"

‎python/frameworks/ag2/traceai_ag2/__init__.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
AG2 1.x already emits OpenTelemetry GenAI spans from its own
44
``TelemetryMiddleware``. This package does not wrap ``Agent``. It:
55
6-
* attaches ``TelemetryMiddleware`` with ``capture_content=False``
6+
* attaches ``TelemetryMiddleware`` (content captured by default, hidden with
7+
``TraceConfig`` / ``FI_HIDE_*``)
78
(``setup`` / ``create_telemetry_middleware``),
89
* registers the Future AGI exporter through ``fi_instrumentation.register()``
910
when you do not pass a provider,

0 commit comments

Comments
 (0)