Skip to content

Commit d21d9de

Browse files
committed
fix(observability-proxy): end silent tails before the Gateway does
Behind a Cilium Gateway, Envoy resets a stream that carries no data for 300s, the default stream idle timeout, and the proxy's own tail idle limit was the same 5m. A tail that went quiet could then be cut by Envoy rather than ended by the proxy, which the client sees as a reset stream instead of a finished response. The HTTPRoute cannot raise Envoy's limit, so the proxy ends a silent tail a little earlier. Assisted-by: LLM Signed-off-by: Maxim Belyy <[email protected]>
1 parent 773a3a9 commit d21d9de

5 files changed

Lines changed: 20 additions & 4 deletions

File tree

‎packages/system/observability-proxy/templates/deployment.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ spec:
3838
- --cluster-domain={{ $clusterDomain }}
3939
- --root-tenant={{ .Values.observabilityProxy.rootTenant }}
4040
- --auth-cache-ttl={{ .Values.observabilityProxy.authCacheTTL }}
41+
{{- /* Below the 300s stream idle timeout of a Cilium Gateway, so a
42+
published tail that stops sending lines is ended by the proxy,
43+
not reset by Envoy. */}}
44+
- --tail-idle-timeout=4m30s
4145
ports:
4246
- name: http
4347
containerPort: 8080

‎packages/system/observability-proxy/templates/httproute.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,9 @@ spec:
2222
port: 80
2323
{{- /* Without a request timeout Cilium leaves Envoy's 15s route default
2424
in place, which would cut every tail. The proxy bounds each
25-
request itself, so the route sets none. */}}
25+
request itself, so the route disables it. Envoy's 300s stream
26+
idle timeout still applies; the proxy's tail idle limit stays
27+
below it (see deployment.yaml). */}}
2628
timeouts:
2729
request: 0s
2830
{{- end }}

‎packages/system/observability-proxy/templates/ingress.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ metadata:
2525
in batches. */}}
2626
nginx.ingress.kubernetes.io/proxy-buffering: "off"
2727
{{- /* The controller default is 10s. The longest silence the proxy
28-
allows is a tail with no new line for --tail-idle-timeout (5m),
29-
after which it ends the response itself. */}}
28+
allows is a tail with no new line for --tail-idle-timeout (4m30s),
29+
after which it ends the response itself; before the first byte
30+
the token and access reviews may add up to --request-timeout (1m). */}}
3031
nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
3132
spec:
3233
ingressClassName: {{ $exposeIngress }}

‎packages/system/observability-proxy/tests/deployment_test.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,15 @@ tests:
4040
path: spec.template.spec.containers[0].args
4141
content: --auth-cache-ttl=10s
4242

43+
# Envoy behind a Cilium Gateway resets a stream after 300s without data
44+
# (envoy.streamIdleTimeoutDurationSeconds), so a silent tail has to be
45+
# ended by the proxy first, or the client sees a reset instead of an end.
46+
- it: ends a silent tail before the Gateway stream idle timeout
47+
asserts:
48+
- contains:
49+
path: spec.template.spec.containers[0].args
50+
content: --tail-idle-timeout=4m30s
51+
4352
- it: falls back to cozy.local when the platform passes no cluster domain
4453
asserts:
4554
- contains:

‎packages/system/observability-proxy/tests/publishing_test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ suite: observability-proxy public endpoint
33
# Kubernetes bearer token, so the route must pass Authorization and the
44
# /apis/... path through untouched, and must not end a request before the
55
# proxy does: a non-tail request may take up to --request-timeout (1m), and
6-
# a tail may stay silent up to --tail-idle-timeout (5m). Whole objects are
6+
# a tail may stay silent up to --tail-idle-timeout (4m30s). Whole objects are
77
# compared rather than single keys, so an annotation or a field that should
88
# not be there fails the test too.
99
templates:

0 commit comments

Comments
 (0)