Skip to content

Commit 54a587c

Browse files
committed
test(ag2-classic): cover the LLM exception path
A model call that raises was untested. The loopback fake model now answers HTTP 400 to a request carrying LLM_FAIL_TRIGGER (OpenAI raises BadRequestError, no retry), and failing_llm_chat runs a chat whose only model call is rejected. - test_setup.py: the LLM span keeps status ERROR, carries error.type=BadRequestError and gen_ai.span.kind=LLM, has no usage or cost, and no content leaks on the error path. - Contract test: the same through register()'s exporting processor (which turns only UNSET into OK) and the shared Receiver; the token-sum check counts successful LLM spans only. Upstream sets error.type and re-raises (llm_wrapper.py:97-101) and the SDK sets ERROR, so these pass on the pre-fix code by design; README Errors row documents the behaviour. approved: Nikhil 2026-10-03 blanket Refs: TH-8237
1 parent cb44a2f commit 54a587c

6 files changed

Lines changed: 102 additions & 10 deletions

File tree

‎python/frameworks/ag2-classic/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ covered by mapping unit tests only.
114114
| Session | `session.id` from upstream `gen_ai.conversation.id` (`ChatResult.chat_id`) on the outermost `conversation` span. Inside `using_session(...)` or `using_attributes(session_id=...)` (from `fi_instrumentation`), that id is set on every AG2 span instead and wins over the chat id; `gen_ai.conversation.id` keeps the chat id. | Nested chats (for example the internal chat behind group-chat speaker selection, or an inner chat a tool starts) keep `gen_ai.conversation.id` but get no `session.id`. Child spans end before the chat id exists, so only the conversation span carries it. A `GroupChatManager` run emits no own conversation span at 0.14.x. `initiate_chats` gives one session per chat: each chat is its own outermost conversation, and the parent `initiate_chats` span (`multi_conversation`) has no `session.id`, so one trace holds several sessions. For one session over the whole call, wrap it in `with using_session("..."):`. |
115115
| User | `user.id` (and `metadata`, `tag.tags`) from `using_user(...)` / `using_attributes(...)`, set on every AG2 span started inside it. | Not emitted upstream; absent without that context. |
116116
| Tools | `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`. | Arguments and result (`gen_ai.tool.call.arguments` / `.result`) only with `capture_content=True`. |
117-
| Errors | Exceptions keep OTel's ERROR status. A failing tool is caught inside `ConversableAgent.execute_function`, so upstream only sets `error.type=ExecutionError`; this package sets status ERROR from `error.type`. A non-zero code exit (`error.type=CodeExecutionError`) also becomes ERROR. | |
117+
| Errors | Exceptions keep OTel's ERROR status. A model call that raises keeps ERROR, and upstream sets `error.type` to the exception class (for example `BadRequestError`, `llm_wrapper.py:97-101`); the span has no usage or cost. A failing tool is caught inside `ConversableAgent.execute_function`, so upstream only sets `error.type=ExecutionError`; this package sets status ERROR from `error.type`. A non-zero code exit (`error.type=CodeExecutionError`) also becomes ERROR. | |
118118
| Retrieval | Not emitted upstream. | Always. |
119119

120120
## Privacy

‎python/frameworks/ag2-classic/tests/_contract_app.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ def main(argv: list) -> int:
2525
from opentelemetry import trace as trace_api
2626

2727
from _fake_openai import FakeOpenAI
28-
from _scenarios import TOOL_SECRET_CITY, broken_tool_chat, group_chat, two_agent_tool_chat
28+
from _scenarios import TOOL_SECRET_CITY, broken_tool_chat, failing_llm_chat, group_chat, two_agent_tool_chat
2929
from traceai_ag2_classic import setup
3030

3131
global_before = type(trace_api.get_tracer_provider()).__name__
@@ -43,7 +43,10 @@ def main(argv: list) -> int:
4343
broken_tool_chat(tracing, fake)
4444
group = group_chat(tracing, fake)
4545
out["group_chat_id"] = str(group.chat_id)
46-
out["llm_requests"] = len(fake.requests)
46+
# The model rejects this one (HTTP 400): an LLM span that ends in an exception.
47+
out["llm_error"] = failing_llm_chat(tracing, fake)
48+
out["llm_requests"] = len(fake.requests) # successful model calls only
49+
out["llm_failed_requests"] = len(fake.failed_requests)
4750

4851
flushed = provider.force_flush(timeout_millis=10000)
4952
out["flushed"] = bool(flushed)

‎python/frameworks/ag2-classic/tests/_fake_openai.py‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,15 @@
44
answers ``POST /v1/chat/completions`` with deterministic replies chosen from the
55
request body:
66
7+
* a request whose messages contain ``LLM_FAIL_TRIGGER`` gets HTTP 400 (the
8+
OpenAI client raises ``BadRequestError`` and does not retry); it is recorded
9+
in ``failed_requests``, not ``requests``;
710
* a group-chat speaker-selection prompt gets ``speaker_reply`` (an agent name);
811
* a request whose last message is a tool result gets ``final_reply``;
912
* a request that offers tools gets one tool call to the first offered tool;
1013
* anything else gets ``text_reply``.
1114
12-
Every reply carries ``usage`` so token attributes can be asserted.
15+
Every successful reply carries ``usage`` so token attributes can be asserted.
1316
"""
1417

1518
from __future__ import annotations
@@ -23,6 +26,7 @@
2326
PROMPT_TOKENS = 11
2427
COMPLETION_TOKENS = 7
2528
FAKE_MODEL = "gpt-4o-mini"
29+
LLM_FAIL_TRIGGER = "FAKE-LLM-REJECT"
2630

2731

2832
class FakeOpenAI:
@@ -39,6 +43,7 @@ def __init__(
3943
self.speaker_reply = speaker_reply
4044
self.tool_arguments = tool_arguments if tool_arguments is not None else {"city": "Paris"}
4145
self.requests: List[Dict[str, Any]] = []
46+
self.failed_requests: List[Dict[str, Any]] = []
4247
self._lock = threading.Lock()
4348
self._counter = 0
4449
owner = self
@@ -50,9 +55,21 @@ def do_POST(self) -> None: # noqa: N802
5055
return
5156
length = int(self.headers.get("Content-Length", "0"))
5257
body = json.loads(self.rfile.read(length).decode("utf-8") or "{}")
53-
payload = owner._reply(body)
58+
status = HTTPStatus.OK
59+
if owner._rejects(body):
60+
status = HTTPStatus.BAD_REQUEST
61+
payload = {
62+
"error": {
63+
"message": "fake model rejected the request",
64+
"type": "invalid_request_error",
65+
"param": None,
66+
"code": "fake_rejected",
67+
}
68+
}
69+
else:
70+
payload = owner._reply(body)
5471
data = json.dumps(payload).encode("utf-8")
55-
self.send_response(HTTPStatus.OK)
72+
self.send_response(status)
5673
self.send_header("Content-Type", "application/json")
5774
self.send_header("Content-Length", str(len(data)))
5875
self.end_headers()
@@ -91,6 +108,14 @@ def llm_config(self, **extra: Any) -> Dict[str, Any]:
91108
entry.update(extra)
92109
return {"config_list": [entry], "cache_seed": None}
93110

111+
def _rejects(self, body: Dict[str, Any]) -> bool:
112+
messages = body.get("messages") or []
113+
if not any(LLM_FAIL_TRIGGER in str(m.get("content") or "") for m in messages if isinstance(m, dict)):
114+
return False
115+
with self._lock:
116+
self.failed_requests.append(body)
117+
return True
118+
94119
def _reply(self, body: Dict[str, Any]) -> Dict[str, Any]:
95120
with self._lock:
96121
self.requests.append(body)

‎python/frameworks/ag2-classic/tests/_scenarios.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,29 @@ def broken_tool_chat(tracing: Any, fake: Any) -> Any:
7373
return user.initiate_chat(assistant, message="call the broken tool", max_turns=2, silent=True)
7474

7575

76+
def failing_llm_chat(tracing: Any, fake: Any) -> str:
77+
"""The model endpoint rejects the request (HTTP 400); the error reaches the caller.
78+
79+
The prompt also carries ``SECRET_PROMPT`` so the error path is checked for
80+
content leaks. Returns the exception class name the caller saw.
81+
"""
82+
from autogen import ConversableAgent
83+
84+
from _fake_openai import LLM_FAIL_TRIGGER
85+
86+
assistant = ConversableAgent("assistant_f", llm_config=fake.llm_config(), human_input_mode="NEVER")
87+
user = ConversableAgent("user_f", llm_config=False, human_input_mode="NEVER", max_consecutive_auto_reply=1)
88+
tracing.instrument_agent(assistant)
89+
tracing.instrument_agent(user)
90+
try:
91+
user.initiate_chat(
92+
assistant, message="{0} {1}".format(LLM_FAIL_TRIGGER, SECRET_PROMPT), max_turns=1, silent=True
93+
)
94+
except Exception as exc: # the caller sees the model error
95+
return type(exc).__name__
96+
raise AssertionError("the fake model was expected to reject the request")
97+
98+
7699
def group_chat(tracing: Any, fake: Any, *, message: str = "Write a haiku about tracing") -> Any:
77100
"""writer -> GroupChatManager with LLM ("auto") speaker selection."""
78101
from autogen import ConversableAgent

‎python/frameworks/ag2-classic/tests/test_contract_harness.py‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,11 @@ def test_ag2_classic_contract_content_off_round_trip():
135135
assert attrs["ag2.span.type"] == span_type, name
136136
assert attrs["gen_ai.span.kind"] == kind, name
137137

138-
# Model, provider, usage and cost on LLM spans.
138+
# Model, provider, usage and cost on LLM spans (the one rejected call is
139+
# asserted below).
139140
for span in names["chat gpt-4o-mini"]:
141+
if span.get("status", {}).get("code") == "STATUS_CODE_ERROR":
142+
continue
140143
attrs = _flatten(span)
141144
assert attrs["gen_ai.operation.name"] == "chat"
142145
assert attrs["gen_ai.provider.name"] == "openai"
@@ -171,6 +174,23 @@ def test_ag2_classic_contract_content_off_round_trip():
171174
assert broken["status"].get("code") == "STATUS_CODE_ERROR"
172175
assert names["execute_tool get_weather"][0].get("status", {}).get("code") != "STATUS_CODE_ERROR"
173176

177+
# A model call that raises: upstream sets error.type and re-raises
178+
# (llm_wrapper.py:97-101). Status stays ERROR through Future AGI's
179+
# exporting processor, which turns only UNSET into OK.
180+
assert app["llm_error"] == "BadRequestError"
181+
assert app["llm_failed_requests"] == 1
182+
llm_spans = [s for s in spans if _flatten(s).get("gen_ai.span.kind") == "LLM"]
183+
failed_llm = [s for s in llm_spans if s.get("status", {}).get("code") == "STATUS_CODE_ERROR"]
184+
assert len(failed_llm) == 1
185+
failed_attrs = _flatten(failed_llm[0])
186+
assert failed_attrs["error.type"] == "BadRequestError"
187+
assert failed_attrs["ag2.span.type"] == "llm"
188+
for key in ("gen_ai.usage.input_tokens", "gen_ai.usage.total_tokens", "gen_ai.cost.total"):
189+
assert key not in failed_attrs, key
190+
ok_llm = [s for s in llm_spans if s not in failed_llm]
191+
for span in ok_llm:
192+
assert span.get("status", {}).get("code") != "STATUS_CODE_ERROR", span["name"]
193+
174194
# AC-04: every span of the group chat run shares one trace id.
175195
group_trace = names["conversation writer"][0]["traceId"]
176196
for name in ("speaker_selection", "conversation checking_agent", "invoke_agent critic"):
@@ -191,7 +211,6 @@ def test_ag2_classic_contract_content_off_round_trip():
191211
# Usage and cost live on LLM spans only. fi-collector promotes token and
192212
# cost keys on any span and Observe sums them over a trace, so an
193213
# aggregate on a conversation/agent span would count every call twice.
194-
llm_spans = [s for s in spans if _flatten(s).get("gen_ai.span.kind") == "LLM"]
195214
for span in spans:
196215
attrs = _flatten(span)
197216
if attrs.get("gen_ai.span.kind") == "LLM":
@@ -202,12 +221,14 @@ def test_ag2_classic_contract_content_off_round_trip():
202221
if key.startswith("gen_ai.usage.") or key in ("gen_ai.cost.total", "llm.cost.total")
203222
]
204223
assert not leaked, "usage/cost keys {0} on non-LLM span {1}".format(leaked, span["name"])
205-
assert len(llm_spans) == app["llm_requests"]
224+
# Each successful model call is one LLM span with 11 + 7 = 18 tokens; the
225+
# failed call carries none.
226+
assert len(ok_llm) == app["llm_requests"]
206227
per_trace: Dict[str, List[int]] = defaultdict(lambda: [0, 0])
207228
for span in spans:
208229
tokens = _flatten(span).get("gen_ai.usage.total_tokens", 0)
209230
per_trace[span["traceId"]][0] += tokens
210-
for span in llm_spans:
231+
for span in ok_llm:
211232
per_trace[span["traceId"]][1] += 1
212233
for trace_id, (total, calls) in per_trace.items():
213234
assert total == calls * 18, (trace_id, total, calls)

‎python/frameworks/ag2-classic/tests/test_setup.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
SECRET_PROMPT,
2929
TOOL_SECRET_CITY,
3030
broken_tool_chat,
31+
failing_llm_chat,
3132
group_chat,
3233
two_agent_tool_chat,
3334
)
@@ -424,6 +425,25 @@ def test_tool_exception_sets_error_status(pipeline, fake):
424425
assert tool.status.description == "ExecutionError"
425426

426427

428+
def test_llm_exception_keeps_error_status_and_error_type(pipeline, fake):
429+
"""The model call raises: upstream sets error.type and re-raises (llm_wrapper.py:97-101)."""
430+
_provider, exporter, make = pipeline
431+
tracing = make()
432+
assert failing_llm_chat(tracing, fake) == "BadRequestError"
433+
assert len(fake.failed_requests) == 1
434+
finished = exporter.get_finished_spans()
435+
llm = [s for s in finished if s.attributes.get("ag2.span.type") == "llm"]
436+
assert len(llm) == 1
437+
attrs = llm[0].attributes
438+
assert attrs["gen_ai.span.kind"] == "LLM"
439+
assert attrs["error.type"] == "BadRequestError"
440+
assert llm[0].status.status_code is StatusCode.ERROR
441+
# A failed call has no usage: nothing to count, nothing invented.
442+
for key in ("gen_ai.usage.total_tokens", "gen_ai.cost.total"):
443+
assert key not in attrs
444+
_assert_no_content_and_no_promoted_usage_off_llm(finished)
445+
446+
427447
# AC-04: group chat, one trace id ----------------------------------------------------
428448

429449

0 commit comments

Comments
 (0)