Skip to content

Commit 563eb3a

Browse files
.NET: terminate chat completions SSE stream with data: [DONE] (#8532)
* .NET: terminate chat completions SSE stream with data: [DONE] The agent chat completions endpoint ended its SSE stream by simply closing the connection. OpenAI-compatible clients treat a final `data: [DONE]` frame as the completion signal, so they waited for a timeout or hung instead. GetStreamingChunksAsync now yields a null-payload sentinel after the agent stream completes, and the item formatter writes it as `[DONE]`. Routing the terminator through SseFormatter keeps that component the single owner of the `data: ` prefix and the blank-line frame terminator. The sentinel is yielded after the loop, so it only follows a stream that ran to completion: if the agent throws or the caller aborts, the exception propagates out of the enumerator and no terminator is written. A client must never read a truncated stream as a complete one. The Responses endpoint is deliberately unchanged -- that API signals completion with typed `response.*` events, and the recorded trace for it contains no [DONE] frame. The existing streaming conformance test could not catch this: it asserts through ParseChatCompletionChunksFromSse, which skips the `[DONE]` line because it is not JSON. The new test asserts on the raw SSE body instead. Fixes #8526 * .NET: address review — pin the failure-path invariant and correct the rationale comment Two review comments from Copilot on #8532. The sentinel comment asserted that clients "wait for the connection to drop, which reads as a hang or a timeout". Live testing disproved that: the response is chunked, so the zero-length terminating chunk ends the body definitively and the official OpenAI client completes without the sentinel. The comment now states the actual interoperability problem -- consumers that treat the sentinel rather than end-of-body as the completion signal cannot recognize the stream as finished -- and names the one this repo ships, SseResponseIdCapture. The failure-path invariant is now covered. An earlier attempt at this test was discarded because a client-side assertion has no teeth: when the agent throws, the server aborts and the client receives no body, so the test passed even against a try/finally refactor that writes the terminator unconditionally. This version tees the response body into a buffer through test middleware, so it observes what the server actually wrote. Verified to fail against that refactor. The failing chat client and the tee stream are private to the test file rather than added to the shared TestHelpers, keeping the change to one test file. * .NET: tidy the failure-path test after a review pass Review findings on the test added in 2240252. No behaviour change; the test still fails against the try/finally append refactor it exists to block. - Dispose the capture buffer (`using var recorded`), which was leaked. - Hoist the trace load into Arrange, matching the Arrange/Act/Assert split the rest of the file uses; it was sitting in the Act block. - Record why the host is built inline instead of through ConformanceTestBase.CreateTestServerAsync: this test needs response-body middleware and a throwing chat client, neither of which belongs in the shared harness for one caller. Without the note the duplication reads as an oversight and invites a refactor back into the base class. --------- Co-authored-by: westey <[email protected]>
1 parent 4964bdd commit 563eb3a

2 files changed

Lines changed: 211 additions & 4 deletions

File tree

‎dotnet/src/Microsoft.Agents.AI.Hosting.OpenAI/ChatCompletions/AIAgentChatCompletionsProcessor.cs‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Copyright (c) Microsoft. All rights reserved.
22

33
using System;
4+
using System.Buffers;
45
using System.Collections.Generic;
56
using System.Linq;
67
using System.Net.ServerSentEvents;
@@ -79,14 +80,27 @@ public Task ExecuteAsync(HttpContext httpContext)
7980
destination: response.Body,
8081
itemFormatter: (sseItem, bufferWriter) =>
8182
{
83+
var chunk = sseItem.Data;
84+
85+
// A null payload is the end-of-stream sentinel from GetStreamingChunksAsync.
86+
// It is written through the formatter rather than appended to the response body
87+
// so that SseFormatter stays the single owner of the "data: " prefix and the
88+
// blank-line frame terminator; duplicating that framing here would let the two
89+
// representations drift apart.
90+
if (chunk is null)
91+
{
92+
bufferWriter.Write("[DONE]"u8);
93+
return;
94+
}
95+
8296
using var writer = new Utf8JsonWriter(bufferWriter);
83-
JsonSerializer.Serialize(writer, sseItem.Data, ChatCompletionsJsonContext.Default.ChatCompletionChunk);
97+
JsonSerializer.Serialize(writer, chunk, ChatCompletionsJsonContext.Default.ChatCompletionChunk);
8498
writer.Flush();
8599
},
86100
cancellationToken);
87101
}
88102

89-
private async IAsyncEnumerable<SseItem<ChatCompletionChunk>> GetStreamingChunksAsync([EnumeratorCancellation] CancellationToken cancellationToken = default)
103+
private async IAsyncEnumerable<SseItem<ChatCompletionChunk?>> GetStreamingChunksAsync([EnumeratorCancellation] CancellationToken cancellationToken = default)
90104
{
91105
// The Unix timestamp (in seconds) of when the chat completion was created. Each chunk has the same timestamp.
92106
DateTimeOffset? createdAt = null;
@@ -167,6 +181,18 @@ private async IAsyncEnumerable<SseItem<ChatCompletionChunk>> GetStreamingChunksA
167181

168182
yield return new(chunk);
169183
}
184+
185+
// The OpenAI wire format ends a streaming chat completion with a literal
186+
// "data: [DONE]" frame, and consumers that treat that sentinel -- rather than
187+
// end-of-body -- as the completion signal cannot recognize the stream as finished
188+
// without it. This repo ships one such consumer: SseResponseIdCapture in the DevUI
189+
// Aspire integration matches "[DONE]"u8 explicitly.
190+
//
191+
// Emitted after the loop so it only follows a stream that ran to completion: if the
192+
// agent throws, or the caller aborts the request, the exception propagates out of the
193+
// enumerator above and no terminator is written. A client must never be able to read a
194+
// truncated stream as a complete one.
195+
yield return new(null);
170196
}
171197
}
172198
}

‎dotnet/tests/Microsoft.Agents.AI.Hosting.OpenAI.UnitTests/OpenAIChatCompletionsConformanceTests.cs‎

Lines changed: 183 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,23 @@
11
// Copyright (c) Microsoft. All rights reserved.
22

3+
using System;
34
using System.Collections.Generic;
5+
using System.IO;
46
using System.Linq;
57
using System.Net.Http;
8+
using System.Runtime.CompilerServices;
9+
using System.Text;
610
using System.Text.Json;
11+
using System.Threading;
712
using System.Threading.Tasks;
13+
using Microsoft.Agents.AI;
814
using Microsoft.Agents.AI.Hosting.OpenAI.Tests;
15+
using Microsoft.AspNetCore.Builder;
16+
using Microsoft.AspNetCore.Hosting.Server;
17+
using Microsoft.AspNetCore.TestHost;
918
using Microsoft.Extensions.AI;
19+
using Microsoft.Extensions.DependencyInjection;
20+
using Microsoft.Extensions.Hosting;
1021

1122
namespace Microsoft.Agents.AI.Hosting.OpenAI.UnitTests;
1223

@@ -230,6 +241,43 @@ public async Task StreamingRequestResponseAsync()
230241
Assert.NotEmpty(accumulatedText);
231242
}
232243

244+
[Fact]
245+
public async Task StreamingResponseEndsWithDoneSentinelAsync()
246+
{
247+
// The recorded OpenAI trace for this endpoint ends with "data: [DONE]", and
248+
// OpenAI-compatible clients use that frame to detect completion rather than waiting for
249+
// the connection to drop. StreamingRequestResponseAsync above cannot catch a missing
250+
// terminator: it asserts through ParseChatCompletionChunksFromSse, which skips the
251+
// "[DONE]" line because it is not JSON.
252+
253+
// Arrange
254+
string requestJson = LoadChatCompletionsTraceFile("streaming/request.json");
255+
HttpClient client = await this.CreateTestServerAsync("done-sentinel-agent", "You are a helpful assistant.", "Hello there.");
256+
257+
// Act
258+
HttpResponseMessage httpResponse = await this.SendChatCompletionRequestAsync(client, "done-sentinel-agent", requestJson);
259+
string responseSse = await httpResponse.Content.ReadAsStringAsync();
260+
261+
// Assert - the terminator is present, is the final frame, and is not duplicated.
262+
// The body must end with the blank line that closes the frame: a client reading frame by
263+
// frame never sees a "data: [DONE]" that is not followed by "\n\n", so asserting on a
264+
// trimmed body would pass even if the final frame were left unterminated.
265+
Assert.Equal("text/event-stream", httpResponse.Content.Headers.ContentType?.MediaType);
266+
Assert.Contains("data: [DONE]", responseSse);
267+
Assert.EndsWith("data: [DONE]\n\n", responseSse, StringComparison.Ordinal);
268+
269+
var dataLines = responseSse.Split('\n')
270+
.Select(line => line.TrimEnd('\r'))
271+
.Where(line => line.StartsWith("data: ", StringComparison.Ordinal))
272+
.ToList();
273+
Assert.Single(dataLines, line => line == "data: [DONE]");
274+
275+
// Assert - the terminator follows the payload chunks rather than replacing them, so a
276+
// client that stops reading at [DONE] still receives the whole completion.
277+
Assert.True(dataLines.Count > 1, "the stream should carry chat completion chunks before the terminator");
278+
Assert.All(dataLines[..^1], line => Assert.StartsWith("data: {", line, StringComparison.Ordinal));
279+
}
280+
233281
[Fact]
234282
public async Task FunctionCallingRequestResponseAsync()
235283
{
@@ -343,7 +391,7 @@ public async Task SystemMessageRequestResponseAsync()
343391
AssertJsonPropertyEquals(systemMessage, "role", "system");
344392
AssertJsonPropertyExists(systemMessage, "content");
345393
string systemContent = systemMessage.GetProperty("content").GetString()!;
346-
Assert.Contains("pirate", systemContent, System.StringComparison.OrdinalIgnoreCase);
394+
Assert.Contains("pirate", systemContent, StringComparison.OrdinalIgnoreCase);
347395

348396
var userMessage = messages[1];
349397
AssertJsonPropertyEquals(userMessage, "role", "user");
@@ -598,7 +646,7 @@ private static List<JsonElement> ParseChatCompletionChunksFromSse(string sseCont
598646
{
599647
var line = lines[i].TrimEnd('\r');
600648

601-
if (line.StartsWith("data: ", System.StringComparison.Ordinal))
649+
if (line.StartsWith("data: ", StringComparison.Ordinal))
602650
{
603651
var jsonData = line.Substring("data: ".Length);
604652

@@ -622,4 +670,137 @@ private static List<JsonElement> ParseChatCompletionChunksFromSse(string sseCont
622670

623671
return chunks;
624672
}
673+
674+
[Fact]
675+
public async Task StreamingResponseOmitsDoneSentinelWhenAgentStreamFailsAsync()
676+
{
677+
// The sentinel must only ever follow a stream that ran to completion; a truncated response
678+
// must never be marked complete. Asserting that from the client does not work: when the
679+
// agent throws, the server aborts and the client receives no body at all, so a client-side
680+
// check passes even against a try/finally refactor that writes the terminator
681+
// unconditionally. This captures what the server actually wrote to the response body, which
682+
// does distinguish the two.
683+
684+
// Arrange - a host whose response body is tee'd into a buffer we can inspect afterwards.
685+
// The host is built here rather than through ConformanceTestBase.CreateTestServerAsync
686+
// because this test needs response-body middleware and a throwing chat client, and neither
687+
// belongs in the shared harness for a single caller. If the base class's wiring changes,
688+
// this setup does not follow it automatically -- keep them in step deliberately.
689+
string requestJson = LoadChatCompletionsTraceFile("streaming/request.json");
690+
using var recorded = new MemoryStream();
691+
using var failingChatClient = new FailingStreamChatClient(updatesBeforeFailure: 2);
692+
693+
WebApplicationBuilder builder = WebApplication.CreateBuilder();
694+
builder.WebHost.UseTestServer();
695+
builder.Services.AddKeyedSingleton<IChatClient>("chat-client", failingChatClient);
696+
builder.AddAIAgent("failing-agent", "You are a helpful assistant.", chatClientServiceKey: "chat-client");
697+
builder.AddOpenAIChatCompletions();
698+
699+
await using WebApplication app = builder.Build();
700+
app.Use(async (context, next) =>
701+
{
702+
Stream original = context.Response.Body;
703+
context.Response.Body = new TeeStream(original, recorded);
704+
try
705+
{
706+
await next(context);
707+
}
708+
finally
709+
{
710+
context.Response.Body = original;
711+
}
712+
});
713+
AIAgent agent = app.Services.GetRequiredKeyedService<AIAgent>("failing-agent");
714+
app.MapOpenAIChatCompletions(agent, path: null, PermissiveMapOptions.ChatCompletions());
715+
await app.StartAsync();
716+
717+
TestServer testServer = (TestServer)app.Services.GetRequiredService<IServer>();
718+
using HttpClient client = testServer.CreateClient();
719+
720+
// Act - the agent yields two updates and then throws partway through the body
721+
try
722+
{
723+
using HttpResponseMessage response = await this.SendChatCompletionRequestAsync(client, "failing-agent", requestJson);
724+
_ = await response.Content.ReadAsStringAsync();
725+
}
726+
catch (Exception ex) when (ex is HttpRequestException or IOException or InvalidOperationException)
727+
{
728+
// Expected: the server aborts the response rather than completing it.
729+
}
730+
731+
await app.StopAsync();
732+
733+
// Assert - the server wrote payload frames but never the terminator
734+
string written = Encoding.UTF8.GetString(recorded.ToArray());
735+
Assert.Contains("chat.completion.chunk", written);
736+
Assert.DoesNotContain("[DONE]", written);
737+
}
738+
739+
/// <summary>A chat client whose streaming response yields a few updates and then throws.</summary>
740+
private sealed class FailingStreamChatClient(int updatesBeforeFailure) : IChatClient
741+
{
742+
public Task<ChatResponse> GetResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
743+
=> throw new InvalidOperationException("mock streaming failure");
744+
745+
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
746+
IEnumerable<ChatMessage> messages,
747+
ChatOptions? options = null,
748+
[EnumeratorCancellation] CancellationToken cancellationToken = default)
749+
{
750+
for (int i = 0; i < updatesBeforeFailure; i++)
751+
{
752+
await Task.Delay(1, cancellationToken);
753+
yield return new ChatResponseUpdate { Contents = [new TextContent($"chunk{i} ")], Role = ChatRole.Assistant };
754+
}
755+
756+
throw new InvalidOperationException("mock streaming failure");
757+
}
758+
759+
public object? GetService(Type serviceType, object? serviceKey = null) => serviceType.IsInstanceOfType(this) ? this : null;
760+
761+
public void Dispose()
762+
{
763+
}
764+
}
765+
766+
/// <summary>
767+
/// Forwards writes to the real response body while copying them into a buffer. The copy happens
768+
/// first so that bytes the server attempted to write are recorded even if the real write fails
769+
/// because the response is already being torn down.
770+
/// </summary>
771+
private sealed class TeeStream(Stream inner, Stream capture) : Stream
772+
{
773+
public override bool CanRead => false;
774+
public override bool CanSeek => false;
775+
public override bool CanWrite => true;
776+
public override long Length => throw new NotSupportedException();
777+
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
778+
779+
public override void Write(byte[] buffer, int offset, int count)
780+
{
781+
capture.Write(buffer, offset, count);
782+
inner.Write(buffer, offset, count);
783+
}
784+
785+
public override void Write(ReadOnlySpan<byte> buffer)
786+
{
787+
capture.Write(buffer);
788+
inner.Write(buffer);
789+
}
790+
791+
public override async ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default)
792+
{
793+
await capture.WriteAsync(buffer, cancellationToken).ConfigureAwait(false);
794+
await inner.WriteAsync(buffer, cancellationToken).ConfigureAwait(false);
795+
}
796+
797+
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
798+
=> this.WriteAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask();
799+
800+
public override void Flush() => inner.Flush();
801+
public override Task FlushAsync(CancellationToken cancellationToken) => inner.FlushAsync(cancellationToken);
802+
public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException();
803+
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
804+
public override void SetLength(long value) => throw new NotSupportedException();
805+
}
625806
}

0 commit comments

Comments
 (0)