Repository navigation
fix(server): default a zero SSE write timeout to 120s - #1686
Conversation
adkrest and agentengine set the write deadline to now + the configured SSE timeout. Left at zero, the deadline was the moment the request arrived, so every /run_sse response (and every Agent Engine query) was lost and the client saw EOF. Treat zero or negative as 120s, the launcher's --sse-write-timeout default, and document it on ServerConfig.SSEWriteTimeout and agentengine.NewHandler. Closes: google#1685
wolo-lab
left a comment
There was a problem hiding this comment.
Defaulting a zero timeout to 120s is the right fix, and both new tests fail without it. One change before merge: please make a negative timeout mean "no deadline" instead of 120s.
Right now there is no way to turn the deadline off. It is set once when the request arrives, so it caps the whole stream rather than the gap between events, and a long agent run that is still streaming gets cut off at 120s. adk-python sets no timeout on /run_sse, so the same agent runs to completion there. http.Server uses zero for "no deadline", but here zero has to keep meaning the default so that a zero-value ServerConfig works. That leaves a negative value as the natural opt-out, and settling it in this PR avoids changing what a negative value means after release.
What that needs:
- Constructors: apply the default only when the timeout is
== 0, in bothNewRuntimeAPIControllerWithConfigandNewAgentEngineAPIController. - Handlers: when the timeout is negative, pass a zero
time.TimetoSetWriteDeadlinerather than skipping the call. The web launcher'shttp.Serverhas its ownWriteTimeout, 15s by default (web.go#L369), and it stays in force unless the handler clears it. - Doc comments: update the three that say "zero or negative means 120 seconds" (
ServerConfig.SSEWriteTimeout,NewAgentEngineAPIController,agentengine.NewHandler). Please also add one toRuntimeAPIControllerConfig.SSETimeout, which has none, so someone mounting the controller directly can find the default. - Tests: add a negative case to each new test, for example by having the test's
SetWriteDeadlinestub record its argument and asserting it is zero. The current tests don't exercise a negative value at all.
Zero still means the 120s default so a zero-value ServerConfig works. A negative timeout now passes a zero time to SetWriteDeadline, which also clears the http.Server's own WriteTimeout, so long agent runs can stream to completion as they do in adk-python.
|
@wolo-lab Thanks, that makes sense. Zero now falls back to 120s and a negative value means no deadline: the handlers pass a zero time to |
* fix(server): default a zero SSE write timeout to 120s (#1686) (cherry picked from commit abb1dda) * fix(adkrest): render nested workflow clusters in graph images (#1724) (cherry picked from commit 49904c0) * fix(adkrest): bound /run_live messages, dead peers and concurrent sessions (#1648) (cherry picked from commit 713449b) * test(adkrest): pin the 40s live keepalive default as a literal The default row compared against defaultLiveKeepaliveTimeout itself, so changing the constant kept the suite green. On 1.x there is no field to override it, which makes the value part of the behavior.
Linked issue
2. Describe the change:
Problem:
RuntimeAPIControllerandAgentEngineAPIControllerset the write deadline totime.Now().Add(sseTimeout). When the timeout is left at zero (the zero value ofadkrest.ServerConfig.SSEWriteTimeout, or0passed toagentengine.NewHandler), the deadline has already passed, so every/run_sseresponse and every Agent Engine query is lost and the client seesEOF. The launcher always passes 120s, so it only hits people who build the server themselves, and the field was undocumented.Solution:
Treat a zero or negative timeout as 120s, the launcher's
--sse-write-timeoutdefault, in both controller constructors (same place an unsetAuthorizergets its default). Document the behaviour onServerConfig.SSEWriteTimeoutandagentengine.NewHandler.I went with a default rather than "zero means no deadline" (as in
http.Server.WriteTimeout) to keep the slow-client protection. Happy to switch if maintainers prefer the other.Behavior change
A server built without a timeout now serves
/run_sseand Agent Engine queries with a 120s write deadline instead of failing every request. Servers that set a positive timeout are unchanged.Testing Plan
Unit Tests:
With your source change reverted and your tests kept, which test fails?
Both run against a real
httptest.Server, sincehttptest.ResponseRecorderdoes not support write deadlines.With the change,
go test -race ./server/... ./cmd/...passes, andgofmt/golangci-lintare clean.Manual End-to-End (E2E) Tests:
Found while embedding
adkrest.NewServerin a small program without setting the timeout:/run_ssereturnedEOFuntilSSEWriteTimeoutwas set.Checklist