Skip to content

Commit 758ec03

Browse files
anishmehta24claude
andauthored
Python: mark hosted /responses incomplete on content_filter and length finish reasons (microsoft#8478)
* Python: mark hosted /responses incomplete on content_filter and length finish reasons The hosted /responses path iterated each AgentResponseUpdate's contents but never read its finish_reason, so a turn the model stopped early (Azure OpenAI content filter, token limit) ended with status "completed" and no trace of the cut-off. Callers could only detect a filtered turn by matching the canned refusal text. _OutputItemTracker now records truncating finish reasons from both the plain-agent and workflow update loops, and _handle_response ends the response with response.incomplete plus incomplete_details.reason ("content_filter" / "max_output_tokens") instead of response.completed, mirroring the OpenAI Responses shape. The refusal text is still delivered as an output item. Fixes microsoft#8475 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * Persist the incomplete reason across resilient checkpoint recovery Recovery rebuilds _OutputItemTracker from the persisted response, so a marker held only in memory was lost if the crash landed between a filtered update and a later one. Mirror it into the stream's internal_metadata (persisted with every checkpoint, like the last checkpoint id) and restore it in the tracker constructor. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * Handle updates through one tracker entry point and pair snapshots with the checkpoint an update follows _OutputItemTracker.handle_update records the update's finish reason and then handles its contents, so the plain-agent and workflow loops share one ordering rule. The workflow loop snapshotted the response against the latest checkpoint in storage at the time an update was consumed. _SignalledIterator drives the workflow one update ahead, so by then the runner could already have checkpointed past the update; a crash after that snapshot resumed the workflow with the update never replayed and its output (and any incomplete reason) lost. _SignalledIterator now takes an optional stamp coroutine that the driver awaits right after each item is produced, before advancing the wrapped iterator again, and the loop pairs snapshots with that stamped checkpoint id. After the workflow completes, its final checkpoint is paired with the full output so recovery does not replay the last superstep. The real-crash recovery integration tests that were xfailed against microsoft#7809 pass with this. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * Fix pre-commit and typing failures - Drop the redundant `None, None` Generator parameters flagged by pyupgrade. - Type the `finish_reason` test parameters as `FinishReasonLiteral` instead of `str` + `type: ignore`, which `ty` does not honour. - Read the text delta with `.get()` so the `ResponseStreamEvent` TypedDict union does not need narrowing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 4a592df commit 758ec03

2 files changed

Lines changed: 348 additions & 41 deletions

File tree

‎python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py‎

Lines changed: 120 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
OutputItem,
6666
OutputItemReasoningItem,
6767
OutputMessageContent,
68+
ResponseIncompleteReason,
6869
ResponseStreamEvent,
6970
ResponseUsage,
7071
ResponseUsageInputTokensDetails,
@@ -171,15 +172,27 @@ class _SignalledIterator(Generic[_T]):
171172
task -- and the real agent/workflow run it's pumping -- would otherwise be silently abandoned.
172173
"""
173174

174-
def __init__(self, iterator: AsyncIterator[_T], *events: asyncio.Event) -> None:
175+
def __init__(
176+
self,
177+
iterator: AsyncIterator[_T],
178+
*events: asyncio.Event,
179+
stamp: Callable[[], Awaitable[Any]] | None = None,
180+
) -> None:
175181
"""Wrap an async iterator, stopping early if any of ``events`` fires.
176182
177183
Args:
178184
iterator: The async iterator to wrap.
179185
events: One or more asyncio.Event objects to watch for. If any of them is set, iteration stops early.
186+
stamp: Optional coroutine function the driver awaits right after the wrapped iterator produces an
187+
item and before it is advanced again. Its result is exposed as :attr:`stamp` while that item
188+
is the current one, which lets a consumer observe state (e.g. the latest persisted workflow
189+
checkpoint) as it was when the item was produced rather than when it is consumed: the driver
190+
runs one item ahead, so by consumption time the wrapped iterator may already have moved on.
180191
"""
181192
self._iterator = iterator
182193
self._events = events
194+
self._stamp_fn = stamp
195+
self._stamp: Any = None
183196
self._signalled = False
184197
# The queue is used to communicate items from the background driver task to the main iteration loop.
185198
self._queue: asyncio.Queue[Any] = asyncio.Queue(maxsize=1)
@@ -195,6 +208,11 @@ def signalled(self) -> bool:
195208
"""
196209
return self._signalled
197210

211+
@property
212+
def stamp(self) -> Any:
213+
"""The ``stamp`` result taken when the current item was produced (``None`` without a ``stamp``)."""
214+
return self._stamp
215+
198216
def __aiter__(self) -> _SignalledIterator[_T]:
199217
return self
200218

@@ -204,13 +222,14 @@ async def _drive(self) -> None:
204222
while True:
205223
try:
206224
item: Any = await self._iterator.__anext__()
225+
stamp = await self._stamp_fn() if self._stamp_fn is not None else None
207226
except StopAsyncIteration:
208227
await self._queue.put(_STOP_SENTINEL)
209228
return
210229
except Exception as exc:
211230
await self._queue.put(exc)
212231
return
213-
await self._queue.put(item)
232+
await self._queue.put((item, stamp))
214233
finally:
215234
iterator: AsyncIterator[_T] = self._iterator
216235
if isinstance(iterator, ResponseStream):
@@ -251,6 +270,7 @@ async def __anext__(self) -> _T:
251270
raise StopAsyncIteration
252271
if isinstance(item, Exception):
253272
raise item
273+
item, self._stamp = item
254274
return cast(_T, item)
255275

256276
async def aclose(self) -> None:
@@ -274,6 +294,9 @@ async def aclose(self) -> None:
274294
# checkpoint in storage (if any), or replay the original input if none exists as no output was ever
275295
# durably persisted.
276296
_LATEST_CHECKPOINT_ID_KEY = "_last_checkpoint_id"
297+
# ``internal_metadata`` key carrying a truncating finish reason across resilient checkpoints, so a
298+
# turn cut short before a crash still ends ``incomplete`` after recovery.
299+
_INCOMPLETE_REASON_KEY = "_incomplete_reason"
277300

278301

279302
# Foundry Toolbox Auth integration
@@ -797,8 +820,9 @@ async def _handle_prepared_response(
797820
for event in tracker.close():
798821
yield event
799822

800-
if tracker.oauth_consent_requested:
801-
yield response_event_stream.emit_incomplete(usage=tracker.usage)
823+
incomplete_reason = tracker.incomplete_reason
824+
if tracker.oauth_consent_requested or incomplete_reason is not None:
825+
yield response_event_stream.emit_incomplete(reason=incomplete_reason, usage=tracker.usage)
802826
else:
803827
yield response_event_stream.emit_completed(usage=tracker.usage)
804828
except Exception as ex:
@@ -964,11 +988,8 @@ async def _handle_inner_agent(
964988
)
965989
async with aclosing(agent_stream):
966990
async for update in agent_stream:
967-
for content in update.contents:
968-
async for event in tracker.handle(
969-
content, message_id=update.message_id, approval_storage=approval_storage
970-
):
971-
yield event
991+
async for event in tracker.handle_update(update, approval_storage=approval_storage):
992+
yield event
972993
except (asyncio.CancelledError, GeneratorExit):
973994
request_interrupted = True
974995
raise
@@ -1161,41 +1182,57 @@ async def _handle_inner_workflow(
11611182
checkpoint_storage=checkpoint_storage,
11621183
)
11631184

1164-
main_iter = _SignalledIterator(run_stream, context.shutdown, cancellation_signal)
1185+
workflow_name = agent.workflow.name
1186+
1187+
async def latest_checkpoint_id() -> str | None:
1188+
latest = await checkpoint_storage.get_latest(workflow_name=workflow_name)
1189+
return latest.checkpoint_id if latest is not None else None
1190+
1191+
def snapshot_response(
1192+
checkpoint_id: str | None,
1193+
) -> Generator[ResponseStreamEvent | ResponseCheckpointEvent]:
1194+
# Pair the response output emitted so far with the workflow checkpoint it corresponds
1195+
# to, so recovery from that checkpoint replays exactly the updates that came after it.
1196+
if checkpoint_id is None or checkpoint_id == response_event_stream.internal_metadata.get(
1197+
_LATEST_CHECKPOINT_ID_KEY
1198+
):
1199+
return
1200+
yield from tracker.close()
1201+
response_event_stream.internal_metadata[_LATEST_CHECKPOINT_ID_KEY] = checkpoint_id
1202+
yield response_event_stream.checkpoint()
1203+
1204+
main_iter = _SignalledIterator(
1205+
run_stream,
1206+
context.shutdown,
1207+
cancellation_signal,
1208+
# The runner creates a checkpoint at the end of each superstep, inside the generator
1209+
# that produces the updates (see RunnerImpl.run_until_convergence). Stamping each
1210+
# update with the latest checkpoint as it was produced tells which checkpoint the
1211+
# update follows; the driver runs one update ahead, so by the time an update is
1212+
# consumed the workflow may already have checkpointed past it.
1213+
stamp=latest_checkpoint_id if self._resilient_background else None,
1214+
)
11651215
async with aclosing(main_iter):
11661216
async for update in main_iter:
11671217
if self._resilient_background:
1168-
latest_checkpoint = await checkpoint_storage.get_latest(workflow_name=agent.workflow.name)
1169-
if (
1170-
latest_checkpoint is not None
1171-
and latest_checkpoint.checkpoint_id
1172-
!= response_event_stream.internal_metadata.get(_LATEST_CHECKPOINT_ID_KEY)
1173-
):
1174-
# A new checkpoint is created when we pull the next item from the stream
1175-
# (see RunnerImpl.run_until_convergence). We only take a snapshot of the
1176-
# response (response_event_stream.checkpoint()) once the checkpoint is
1177-
# durably persisted. This means all items from the previous superstep
1178-
# has been pulled thus we can safely close the tracker. The latest checkpoint
1179-
# now reflects the state of the workflow that matches the response output.
1180-
# Note that if a workflow crashes before any update is created, no response
1181-
# snapshot is taken. However, upon recovery the workflow will still be resumed
1182-
# from the latest checkpoint.
1183-
for event in tracker.close():
1184-
yield event
1185-
response_event_stream.internal_metadata[_LATEST_CHECKPOINT_ID_KEY] = (
1186-
latest_checkpoint.checkpoint_id
1187-
)
1188-
yield response_event_stream.checkpoint()
1189-
1190-
for content in update.contents:
1191-
async for event in tracker.handle(
1192-
content, message_id=update.message_id, approval_storage=approval_storage
1193-
):
1218+
# Every update before this one belongs to the stamped checkpoint (or an
1219+
# earlier one), so the output so far can be snapshotted against it. If the
1220+
# workflow crashes before any update is produced, no snapshot is taken and
1221+
# recovery still resumes from the latest workflow checkpoint.
1222+
for event in snapshot_response(main_iter.stamp):
11941223
yield event
1224+
1225+
async for event in tracker.handle_update(update, approval_storage=approval_storage):
1226+
yield event
11951227
# Cancellation needs no extra action here (the loop above already stopped); shutdown
11961228
# does, but only if it's what actually stopped the loop, not a natural completion.
11971229
if main_iter.signalled and context.shutdown.is_set():
11981230
await context.exit_for_recovery()
1231+
elif self._resilient_background and not main_iter.signalled:
1232+
# The workflow ran to completion: pair its final checkpoint with the full output, so
1233+
# recovery after this point does not replay the last superstep.
1234+
for event in snapshot_response(await latest_checkpoint_id()):
1235+
yield event
11991236
except Exception:
12001237
logger.exception("Failed to produce response for workflow agent")
12011238
raise
@@ -1291,6 +1328,16 @@ def __init__(self, stream: ResponseEventStream) -> None:
12911328
self._mcp_builder: OutputItemMcpCallBuilder | None = None
12921329
self._outstanding_function_calls: dict[str, str | None] = {}
12931330
self._oauth_consent_requests: set[tuple[str, str]] = set()
1331+
# Set when an agent update reports the model stopped early (content filter, token
1332+
# limit); the response then ends as ``incomplete`` instead of ``completed`` so callers
1333+
# can tell a cut-short turn from a successful one. Mirrored into the stream's
1334+
# ``internal_metadata`` so it survives a resilient checkpoint/recovery cycle, which
1335+
# rebuilds this tracker from the persisted response.
1336+
self._incomplete_reason: ResponseIncompleteReason | None = None
1337+
persisted_reason = stream.internal_metadata.get(_INCOMPLETE_REASON_KEY)
1338+
if isinstance(persisted_reason, str):
1339+
with suppress(ValueError):
1340+
self._incomplete_reason = ResponseIncompleteReason(persisted_reason)
12941341
for item in stream.response.get("output", []):
12951342
if not isinstance(item, Mapping):
12961343
continue
@@ -1329,6 +1376,43 @@ def oauth_consent_requested(self) -> bool:
13291376
"""Return whether this response emitted an OAuth consent request."""
13301377
return bool(self._oauth_consent_requests)
13311378

1379+
@property
1380+
def incomplete_reason(self) -> ResponseIncompleteReason | None:
1381+
"""Return why the turn was cut short, if any update reported a truncating finish reason."""
1382+
return self._incomplete_reason
1383+
1384+
def record_finish_reason(self, finish_reason: str | None) -> None:
1385+
"""Note the finish reason of an agent update.
1386+
1387+
Only finish reasons that mean the model stopped early are retained, mapped onto the
1388+
Responses ``incomplete_details.reason`` vocabulary. A content filter is kept in
1389+
preference to a token limit if both are seen during a multi-step turn, since it is the
1390+
more actionable signal for the caller.
1391+
"""
1392+
if finish_reason == "content_filter":
1393+
self._incomplete_reason = ResponseIncompleteReason.CONTENT_FILTER
1394+
elif finish_reason == "length" and self._incomplete_reason is None:
1395+
self._incomplete_reason = ResponseIncompleteReason.MAX_OUTPUT_TOKENS
1396+
else:
1397+
return
1398+
self._stream.internal_metadata[_INCOMPLETE_REASON_KEY] = self._incomplete_reason.value
1399+
1400+
async def handle_update(
1401+
self,
1402+
update: AgentResponseUpdate,
1403+
*,
1404+
approval_storage: FunctionApprovalStore | None = None,
1405+
) -> AsyncGenerator[ResponseStreamEvent]:
1406+
"""Process one agent update: note its finish reason, then handle each of its contents.
1407+
1408+
This is the single entry point for both the plain-agent and the workflow loops, so the
1409+
finish reason cannot be forgotten on one of them.
1410+
"""
1411+
self.record_finish_reason(update.finish_reason)
1412+
for content in update.contents:
1413+
async for event in self.handle(content, message_id=update.message_id, approval_storage=approval_storage):
1414+
yield event
1415+
13321416
async def handle(
13331417
self,
13341418
content: Content,

0 commit comments

Comments
 (0)