Skip to content

Commit a91fe66

Browse files
eavanvalkenburgeavanvalkenburg
authored
Python: Preserve shell outcome fidelity (#8934)
* Python: Preserve shell outcome fidelity * Python: Preserve raw shell tool results * Python: Preserve shell truncation notices
1 parent 68e58bf commit a91fe66

10 files changed

Lines changed: 508 additions & 52 deletions

File tree

‎docs/specs/004-python-function-calling-loop.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -432,7 +432,8 @@ that manually replay messages own the equivalent rule: do not resend an approval
432432
or a shell call marked with a local environment, paired with a configured local executor, can enter the local
433433
function loop and its approval boundary. Stateless replay preserves provider shell call/output items in their
434434
original shape and fails explicitly when that provider representation is unavailable or inconsistent. Locally
435-
generated shell outputs are sent in both service-side and stateless continuation modes.
435+
generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes and are
436+
sent in both service-side and stateless continuation modes.
436437

437438
### Reasoning-bound calls
438439

@@ -588,7 +589,7 @@ that manually replay messages own the equivalent rule: do not resend an approval
588589
| Parallel calls | Results retain the corresponding call ids and execution count. | `test_max_function_calls_limits_parallel_invocations`, `test_streaming_multiple_function_calls_parallel_execution` |
589590
| Sequential calls | Calls execute one at a time in model order, including after approval replay; mixed local/Host-owned siblings retain that order, approval-only batches wait for every decision before executing, each call has isolated context, and middleware termination takes effect after the accepted batch finishes. | `packages/core/tests/core/test_tools.py::test_sequential_function_invocation_runs_calls_in_model_order`, `test_sequential_function_invocation_isolates_context`, `test_sequential_function_invocation_finishes_batch_after_termination`, `packages/core/tests/core/test_harness_tool_approval.py::test_mixed_batch_hides_already_approved_request_until_approval_replay`, `test_sequential_approval_replay_preserves_model_order_when_responses_are_reversed`, `test_sequential_approval_replay_waits_for_the_complete_batch`, `packages/core/tests/core/test_function_invocation_logic.py::test_sequential_mixed_batch_preserves_approval_and_host_model_order`, `packages/core/tests/workflow/test_agent_executor_tool_calls.py::test_workflow_cancels_host_member_of_mixed_batch_with_terminal_result`, `test_workflow_host_cancellation_preserves_occurrence_with_reused_call_id` |
590591
| Informational-only call | The call is returned but not executed or approved. | `test_informational_only_function_call_is_not_invoked`, `test_informational_only_function_call_does_not_request_approval`, `test_streaming_informational_only_function_call_is_not_invoked` |
591-
| OpenAI hosted/local shell boundary | Hosted shell calls remain informational in streaming and non-streaming responses even when a local executor is configured; only valid explicit local-shell items or shell calls marked with a local environment can execute, local execution preserves its configured approval mode, stateless loops preserve the complete provider shell transcript or fail explicitly, and locally generated shell outputs are sent in every continuation mode. | `packages/openai/tests/openai/test_openai_chat_client.py::test_response_content_creation_with_shell_call_remains_hosted_with_local_tool`, `test_parse_chunk_from_openai_shell_call_done_remains_hosted`, `test_parse_chunk_from_openai_local_environment_shell_call_done_emits_command`, `test_mixed_shell_calls_only_invoke_explicit_local_shell_call`, `test_stateless_shell_transcript_without_provider_item_fails`, `test_prepare_messages_keeps_local_shell_output_under_storage`, `test_malformed_local_environment_shell_call_is_not_executable`, `test_response_content_creation_with_local_shell_call_maps_to_function_call`, `test_malformed_local_shell_call_is_not_executable`, `test_response_function_call_named_local_shell_is_informational`, `test_parse_chunk_function_call_named_local_shell_is_informational`, `test_local_shell_tool_requires_approval_before_function_loop_execution` |
592+
| OpenAI hosted/local shell boundary | Hosted shell calls remain informational in streaming and non-streaming responses even when a local executor is configured; only valid explicit local-shell items or shell calls marked with a local environment can execute, local execution preserves its configured approval mode, stateless loops preserve the complete provider shell transcript or fail explicitly, and locally generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes in every continuation mode. | `packages/openai/tests/openai/test_openai_chat_client.py::test_response_content_creation_with_shell_call_remains_hosted_with_local_tool`, `test_parse_chunk_from_openai_shell_call_done_remains_hosted`, `test_parse_chunk_from_openai_local_environment_shell_call_done_emits_command`, `test_mixed_shell_calls_only_invoke_explicit_local_shell_call`, `test_stateless_shell_transcript_without_provider_item_fails`, `test_prepare_messages_keeps_local_shell_output_under_storage`, `test_shell_output_payloads_use_structured_function_result_items`, `test_malformed_local_environment_shell_call_is_not_executable`, `test_response_content_creation_with_local_shell_call_maps_to_function_call`, `test_malformed_local_shell_call_is_not_executable`, `test_response_function_call_named_local_shell_is_informational`, `test_parse_chunk_function_call_named_local_shell_is_informational`, `test_local_shell_tool_requires_approval_before_function_loop_execution` |
592593
| Declaration-only call | The call is surfaced as user input and is not executed; streaming arguments appear once while finalized request metadata remains available. | `test_declaration_only_tool`, `test_streaming_declaration_only_tool_preserves_metadata_without_duplicate_arguments` |
593594
| Function invocation disabled | The client bypasses the invocation loop without losing invocation kwargs. | `test_function_invocation_config_enabled_false`, `test_function_invocation_config_enabled_false_preserves_invocation_kwargs`, `test_streaming_function_invocation_config_enabled_false` |
594595
| Runtime tool changes | Added tools become available on the next iteration and retain approval behavior. | `test_add_tools_available_next_iteration`, `test_add_tools_with_approval_required_tool` |

‎python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py‎

Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@
6868
FunctionShellAction,
6969
FunctionShellCallOutputContent,
7070
FunctionShellCallOutputExitOutcome,
71+
FunctionShellCallOutputTimeoutOutcome,
7172
Item,
7273
ItemReasoningItem,
7374
LocalEnvironmentResource,
@@ -2432,14 +2433,19 @@ async def handle(
24322433
if content.outputs:
24332434
for out in content.outputs:
24342435
exit_code = getattr(out, "exit_code", None)
2436+
outcome = (
2437+
FunctionShellCallOutputTimeoutOutcome(type="timeout")
2438+
if getattr(out, "timed_out", False)
2439+
else FunctionShellCallOutputExitOutcome(
2440+
type="exit",
2441+
exit_code=exit_code if exit_code is not None else 0,
2442+
)
2443+
)
24352444
output_items.append(
24362445
FunctionShellCallOutputContent(
24372446
stdout=getattr(out, "stdout", "") or "",
24382447
stderr=getattr(out, "stderr", "") or "",
2439-
outcome=FunctionShellCallOutputExitOutcome(
2440-
type="exit",
2441-
exit_code=exit_code if exit_code is not None else 0,
2442-
),
2448+
outcome=outcome,
24432449
)
24442450
)
24452451
async for event in self._stream.output_item_function_shell_call_output(
@@ -2853,6 +2859,17 @@ def _computer_screenshot_to_output(screenshot: Content) -> dict[str, Any]:
28532859
return output
28542860

28552861

2862+
def _shell_command_output_to_content(output: Mapping[str, Any]) -> Content:
2863+
outcome = output["outcome"]
2864+
outcome_type = outcome.get("type")
2865+
return Content.from_shell_command_output(
2866+
stdout=output.get("stdout") or "",
2867+
stderr=output.get("stderr") or "",
2868+
exit_code=outcome.get("exit_code") if outcome_type == "exit" else None,
2869+
timed_out=True if outcome_type == "timeout" else False if outcome_type == "exit" else None,
2870+
)
2871+
2872+
28562873
async def _item_to_message(
28572874
item: Item,
28582875
*,
@@ -2970,14 +2987,7 @@ async def _item_to_message(
29702987
)
29712988

29722989
if item["type"] == "shell_call_output":
2973-
outputs = [
2974-
Content.from_shell_command_output(
2975-
stdout=out["stdout"] or "",
2976-
stderr=out["stderr"] or "",
2977-
exit_code=out["outcome"].get("exit_code"),
2978-
)
2979-
for out in (item["output"] or [])
2980-
]
2990+
outputs = [_shell_command_output_to_content(out) for out in (item["output"] or [])]
29812991
return Message(
29822992
role="tool",
29832993
contents=[

‎python/packages/foundry_hosting/tests/test_responses.py‎

Lines changed: 115 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3898,6 +3898,51 @@ async def test_shell_call_preserves_execution_limits(self) -> None:
38983898
"max_output_length": 4096,
38993899
}
39003900

3901+
async def test_shell_result_preserves_outcomes(self) -> None:
3902+
agent = _make_agent(
3903+
response=AgentResponse(
3904+
messages=[
3905+
Message(
3906+
role="tool",
3907+
contents=[
3908+
Content.from_shell_tool_result(
3909+
call_id="shell_1",
3910+
outputs=[
3911+
Content.from_shell_command_output(
3912+
stdout="ok",
3913+
stderr="",
3914+
exit_code=0,
3915+
timed_out=False,
3916+
),
3917+
Content.from_shell_command_output(
3918+
stdout="",
3919+
stderr="command failed",
3920+
exit_code=3,
3921+
timed_out=False,
3922+
),
3923+
Content.from_shell_command_output(
3924+
stdout="partial",
3925+
stderr="timeout details",
3926+
timed_out=True,
3927+
),
3928+
],
3929+
)
3930+
],
3931+
)
3932+
]
3933+
)
3934+
)
3935+
3936+
resp = await _post(_make_server(agent), stream=False)
3937+
3938+
assert resp.status_code == 200
3939+
shell_item = next(item for item in resp.json()["output"] if item["type"] == "shell_call_output")
3940+
assert shell_item["output"] == [
3941+
{"stdout": "ok", "stderr": "", "outcome": {"type": "exit", "exit_code": 0}},
3942+
{"stdout": "", "stderr": "command failed", "outcome": {"type": "exit", "exit_code": 3}},
3943+
{"stdout": "partial", "stderr": "timeout details", "outcome": {"type": "timeout"}},
3944+
]
3945+
39013946
async def test_hosted_mcp_call_and_result_persist_as_single_mcp_call(self) -> None:
39023947
agent = _make_agent(
39033948
response=AgentResponse(
@@ -4865,8 +4910,8 @@ async def test_shell_call_output(self) -> None:
48654910

48664911
output: FunctionShellCallOutputContent = {
48674912
"stdout": "file.txt",
4868-
"stderr": "",
4869-
"outcome": cast(FunctionShellCallOutputExitOutcome, {"exit_code": 0}),
4913+
"stderr": "warning",
4914+
"outcome": cast(FunctionShellCallOutputExitOutcome, {"type": "exit", "exit_code": 3}),
48704915
}
48714916
item: OutputItemFunctionShellCallOutput = {
48724917
"type": "shell_call_output",
@@ -4880,6 +4925,40 @@ async def test_shell_call_output(self) -> None:
48804925
assert msg.role == "tool"
48814926
assert msg.contents[0].type == "shell_tool_result"
48824927
assert msg.contents[0].call_id == "call_sc"
4928+
assert msg.contents[0].outputs is not None
4929+
assert msg.contents[0].outputs[0].stdout == "file.txt"
4930+
assert msg.contents[0].outputs[0].stderr == "warning"
4931+
assert msg.contents[0].outputs[0].exit_code == 3
4932+
assert msg.contents[0].outputs[0].timed_out is False
4933+
4934+
async def test_shell_call_output_preserves_timeout(self) -> None:
4935+
from azure.ai.agentserver.responses.models import (
4936+
FunctionShellCallOutputContent,
4937+
FunctionShellCallOutputTimeoutOutcome,
4938+
OutputItemFunctionShellCallOutput,
4939+
)
4940+
4941+
output: FunctionShellCallOutputContent = {
4942+
"stdout": "partial",
4943+
"stderr": "timeout details",
4944+
"outcome": cast(FunctionShellCallOutputTimeoutOutcome, {"type": "timeout"}),
4945+
}
4946+
item: OutputItemFunctionShellCallOutput = {
4947+
"type": "shell_call_output",
4948+
"id": "sco-timeout",
4949+
"call_id": "call_timeout",
4950+
"status": "completed",
4951+
"output": [output],
4952+
"max_output_length": 1024,
4953+
}
4954+
4955+
msg = await _output_item_to_message(item)
4956+
4957+
assert msg.contents[0].outputs is not None
4958+
assert msg.contents[0].outputs[0].stdout == "partial"
4959+
assert msg.contents[0].outputs[0].stderr == "timeout details"
4960+
assert msg.contents[0].outputs[0].exit_code is None
4961+
assert msg.contents[0].outputs[0].timed_out is True
48834962

48844963
async def test_local_shell_call(self) -> None:
48854964
from azure.ai.agentserver.responses.models import LocalShellExecAction, OutputItemLocalShellToolCall
@@ -5463,8 +5542,8 @@ async def test_shell_call_output(self) -> None:
54635542

54645543
output: FunctionShellCallOutputContentParam = {
54655544
"stdout": "file.txt",
5466-
"stderr": "",
5467-
"outcome": cast(FunctionShellCallOutputExitOutcomeParam, {"exit_code": 0}),
5545+
"stderr": "warning",
5546+
"outcome": cast(FunctionShellCallOutputExitOutcomeParam, {"type": "exit", "exit_code": 3}),
54685547
}
54695548
item: FunctionShellCallOutputItemParam = {
54705549
"type": "shell_call_output",
@@ -5477,6 +5556,38 @@ async def test_shell_call_output(self) -> None:
54775556
assert msg.role == "tool"
54785557
assert msg.contents[0].type == "shell_tool_result"
54795558
assert msg.contents[0].call_id == "call_sc"
5559+
assert msg.contents[0].outputs is not None
5560+
assert msg.contents[0].outputs[0].stdout == "file.txt"
5561+
assert msg.contents[0].outputs[0].stderr == "warning"
5562+
assert msg.contents[0].outputs[0].exit_code == 3
5563+
assert msg.contents[0].outputs[0].timed_out is False
5564+
5565+
async def test_shell_call_output_preserves_timeout(self) -> None:
5566+
from azure.ai.agentserver.responses.models import (
5567+
FunctionShellCallOutputContentParam,
5568+
FunctionShellCallOutputItemParam,
5569+
FunctionShellCallOutputTimeoutOutcomeParam,
5570+
)
5571+
5572+
output: FunctionShellCallOutputContentParam = {
5573+
"stdout": "partial",
5574+
"stderr": "timeout details",
5575+
"outcome": cast(FunctionShellCallOutputTimeoutOutcomeParam, {"type": "timeout"}),
5576+
}
5577+
item: FunctionShellCallOutputItemParam = {
5578+
"type": "shell_call_output",
5579+
"call_id": "call_timeout",
5580+
"output": [output],
5581+
"max_output_length": 1024,
5582+
}
5583+
5584+
msg = await _item_to_message(item)
5585+
5586+
assert msg.contents[0].outputs is not None
5587+
assert msg.contents[0].outputs[0].stdout == "partial"
5588+
assert msg.contents[0].outputs[0].stderr == "timeout details"
5589+
assert msg.contents[0].outputs[0].exit_code is None
5590+
assert msg.contents[0].outputs[0].timed_out is True
54805591

54815592
async def test_local_shell_call(self) -> None:
54825593
from azure.ai.agentserver.responses.models import ItemLocalShellToolCall, LocalShellExecAction

0 commit comments

Comments
 (0)