Skip to content

Commit c5e9062

Browse files
committed
fix(opensearch-operator): bound every call to a cluster's HTTP API
Nothing limited how long a call to a cluster's HTTP API could take. The client builds its own http.Transport, and a fresh transport has no dial, no TLS handshake and no response-header timeout; the config sets no http.Client, so there is no timeout there either; and every call the client makes on its own behalf passes context.Background(). The Timeout field the request types carry is the server-side query parameter, which bounds how long the server waits before answering and helps only once something answers. Each controller runs a single worker, so one call that never returns stops every OpenSearchCluster from being reconciled, and the finalizer on the next deletion waits out its whole termination timeout with nothing in the log to say why. The measurement is in #4332: the operator wrote its last line seconds after excluding a bootstrap pod from the voting configuration, which is an HTTP call, to a pod that was NotReady, and stayed Running with no restarts and nothing further in the log. The patch gives the client's own calls a 30s deadline, and the transport a 10s dial, a 10s TLS handshake and a 60s response-header timeout. The transport bound is deliberately the looser one: it is the only bound on the calls that take a context from the caller and hand the still-open body back, where cancelling on return would fail the caller's read, and on the product check opensearch-go runs on a client's first request, which carries a context.Background() of its own inside the library and cannot be reached from here. This bounds the wait; it does not make an unresponsive cluster answer. A call that passes its deadline now fails the reconcile, which requeues, instead of retiring the worker. Upstream is the better home for the change, and the patch goes away with the first release that carries one. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
1 parent 8799a4e commit c5e9062

2 files changed

Lines changed: 410 additions & 3 deletions

File tree

‎packages/system/opensearch-operator/images/opensearch-operator/Dockerfile‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,23 @@
2525
# upstream as opensearch-project/opensearch-k8s-operator#1585, fixed by #1586;
2626
# drop it on the first release carrying that.
2727
#
28-
# The patches are applied in the order the glob expands, which is alphabetical,
29-
# and skip-unusable-drain-exclusion builds on restart-on-permanently-yellow. A
30-
# rename that reorders them fails the build rather than producing a wrong binary.
28+
# patches/timeout-cluster-api-calls.diff bounds the calls the operator makes to
29+
# a cluster's HTTP API. Upstream builds its own http.Transport, which carries no
30+
# dial, handshake or response-header timeout; it passes context.Background() to
31+
# every call it makes on its own behalf; and it configures no http.Client, so
32+
# there is no timeout there either. A cluster that accepts a connection and then
33+
# says nothing therefore holds the call for as long as the process lives, and
34+
# with one worker per controller that one call stops every OpenSearchCluster
35+
# from being reconciled. The deletion that then waits out its whole termination
36+
# timeout is the visible end of it. Tracked as cozystack/cozystack#4337; it
37+
# belongs upstream, and there is no upstream issue for it yet.
38+
#
39+
# The patches are applied in the order the glob expands, which is alphabetical.
40+
# skip-unusable-drain-exclusion builds on restart-on-permanently-yellow, and
41+
# timeout-cluster-api-calls rewrites os_client.go call sites that
42+
# skip-unusable-drain-exclusion both adds and quotes as context, so it has to
43+
# come last. A rename that reorders them fails the build rather than producing a
44+
# wrong binary.
3145
FROM golang:1.26@sha256:079e59808d2d252516e27e3f3a9c003740dee7f75e55aa71528766d52bcfc16a AS builder
3246

3347
# Keep in step with the chart version pulled by `make update`: the chart's

0 commit comments

Comments
 (0)