Skip to content

Commit a759190

Browse files
authored
fix(opensearch-operator): bound every call to a cluster's HTTP API (#4341)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A call to a cluster's HTTP API has no bound of its own. `NewOsClusterClient` builds its own `http.Transport`, and a fresh transport carries no dial, TLS-handshake or response-header timeout; the config sets no `http.Client`, so there is no timeout there either; the calls the client makes on its own behalf pass `context.Background()`; and the calls that take a context from the caller are handed one with no deadline on it, since nothing on the reconcile path sets one. The `Timeout` field the request types carry is the server-side query parameter: it bounds how long the server waits before answering, which helps only once something answers. Each controller runs a single worker, so one call that never returns stops every `OpenSearchCluster` from being reconciled, and the finalizer on the next deletion waits out its whole termination timeout with nothing in the log to say why. Reported in #4337, measured in #4332. A fourth patch on the image bounds every request the operator's client sends, at two levels. - Every request goes out through a `RoundTripper` that puts a 30s deadline on it. That is the only level that reaches the request opensearch-go builds for itself: before a client's first request the library runs a product check with a `context.Background()` of its own and decodes the body of the answer, and a client is built per reconcile, so that check runs on every one. The deadline cannot be released when `RoundTrip` returns, because the body goes back to the caller still open, so the cancel rides on the body and is released by `Close`, which is how `http.Client` applies its own `Timeout`. - That bounds one attempt. opensearch-go retries a request up to three more times on 502, 503 and 504, and the transport's deadline starts over on each of them, so the calls carry a 30s deadline a second time for the series: the 15 calls in `os_client.go` that request and decode before returning through `requestContext`, and the 28 that go through the `doHTTP*` helpers in `performRequest`. - The transport keeps a 10s dial and a 10s TLS handshake, so an unreachable node fails in a third of the budget with an error that names what went wrong. `ResponseHeaderTimeout` is gone from the transport. It stops counting the moment the headers arrive, so it is no bound on a body that starts and then stops, and at 60s it sat above a request deadline that now covers everything it covered. This bounds the wait, it does not make an unresponsive cluster answer: a call that passes its deadline fails the reconcile, which requeues. The change belongs upstream, and there is no upstream issue for it yet. The Dockerfile entry carries the drop condition the other entries carry. ### Tests Seven, in the package the image's own `go test -race` step already runs: - a call to a cluster that accepts the request and answers nothing returns instead of running forever; - a new client, against a cluster whose product check flushes its headers and then stops, returns instead of running forever; - a request through the operator's transport, to a server that accepts the connection and then says nothing, returns instead of running forever; - a body that arrives with its headers and then stops is abandoned; - a call retried on 503 is bounded as a series rather than per attempt; - a body is still readable after the call that fetched it has returned; - the transport carries the handshake bound by value, and the dial bound through the dialer it is built from. Every bound was checked against the mutation that removes it, and all seven turn a test red: the transport wrapper dropped from the constructor, the deadline dropped from `RoundTrip`, the cancel moved off the body there, the deadline dropped from `performRequest`, one call site reverted to `context.Background()`, and the dial and handshake timeouts dropped in turn. Waits are measured from the test goroutine, so an unbounded call fails the test rather than hanging the run until CI kills it. The silent-server test holds its accepted connections and forces a collection while the call is in flight, because an unreachable `net.Conn` is closed by the collector (`net/fd_posix.go` puts a finalizer on `netFD`) and that turns the silent server into one that resets the connection. The full four-patch sequence applies to a pristine v2.8.0 tree in the glob order the build relies on, with no offsets and no fuzz; `gofmt` and `go vet` are clean; and `docker buildx build images/opensearch-operator` builds the image on linux/arm64 with the `go test -race` step green. ### Screenshots Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map against the diff: one comment block in `packages/system/opensearch-operator/images/opensearch-operator/Dockerfile` and one new file in that image's `patches/`. No package added, renamed or removed, no values key, no schema, no enum, no `hack/` file, no make target, no namespace and no `ApplicationDefinition`. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(opensearch-operator): time out calls to a cluster's HTTP API, so one unanswered request no longer stops the operator reconciling ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Added timeout handling for OpenSearch cluster connections, including connection, TLS-handshake, response-header, and request deadlines. * Prevented requests from hanging indefinitely when a cluster is unavailable, unresponsive, or stops returning data. * Ensured response bodies remain readable after the initial request completes when applicable. * Bounded the full request lifecycle, including response-body reads and retried calls. * **Tests** * Added coverage for connection, response, retry, and body-read timeout scenarios. * Enabled race detection for relevant package tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents ff0b851 + 48058dc commit a759190

2 files changed

Lines changed: 811 additions & 4 deletions

File tree

‎packages/system/opensearch-operator/images/opensearch-operator/Dockerfile‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,24 @@
2525
# upstream as opensearch-project/opensearch-k8s-operator#1585, fixed by #1586;
2626
# drop it on the first release carrying that.
2727
#
28+
# patches/timeout-cluster-api-calls.diff bounds the calls the operator makes to
29+
# a cluster's HTTP API. Upstream builds its own http.Transport, which carries no
30+
# dial, handshake or response-header timeout; it passes context.Background() to
31+
# every call it makes on its own behalf; the calls that take a context from the
32+
# caller are handed one with no deadline on it; and no http.Client is
33+
# configured, so there is no timeout there either. A cluster that accepts a
34+
# connection and then says nothing therefore holds the call for as long as the
35+
# process lives, and with one worker per controller that one call stops every
36+
# OpenSearchCluster from being reconciled. The deletion that then waits out its
37+
# whole termination timeout is the visible end of it. The patch bounds a request
38+
# on the RoundTripper, the one level that also reaches the product check
39+
# opensearch-go runs for itself before a client's first request, and bounds the
40+
# call again where the call site can be reached, because the library retries a
41+
# request up to three more times on 502, 503 and 504 and the transport's
42+
# deadline starts over on each. Tracked as cozystack/cozystack#4337;
43+
# it belongs upstream, there is no upstream issue for it yet, and the patch goes
44+
# away with the first release that carries a fix.
45+
#
2846
# patches/trace-stalled-component-reconcilers.diff names the component a
2947
# reconcile is stuck in. The cluster controller runs nine component reconcilers
3048
# in sequence and none of them logs unconditionally when it is entered, so a
@@ -37,9 +55,12 @@
3755
# It reports rather than repairs: drop it once the stall itself is understood
3856
# and fixed.
3957
#
40-
# The patches are applied in the order the glob expands, which is alphabetical,
41-
# and skip-unusable-drain-exclusion builds on restart-on-permanently-yellow. A
42-
# rename that reorders them fails the build rather than producing a wrong binary.
58+
# The patches are applied in the order the glob expands, which is alphabetical.
59+
# skip-unusable-drain-exclusion builds on restart-on-permanently-yellow, and
60+
# timeout-cluster-api-calls rewrites os_client.go call sites that
61+
# skip-unusable-drain-exclusion both adds and quotes as context, so it has to
62+
# come after it. A rename that reorders them fails the build rather than
63+
# producing a wrong binary.
4364
FROM golang:1.26@sha256:079e59808d2d252516e27e3f3a9c003740dee7f75e55aa71528766d52bcfc16a AS builder
4465

4566
# Keep in step with the chart version pulled by `make update`: the chart's
@@ -63,7 +84,10 @@ WORKDIR /workspace/opensearch-operator
6384
# them: the build below compiles main.go, so a test file that had stopped
6485
# compiling, or stopped asserting, would still produce an image. These two
6586
# package trees are the ones the patches touch, and both run without a cluster.
66-
RUN go test ./opensearch-gateway/... ./pkg/reconcilers/...
87+
# -race is on because the timeout tests run a listener in one goroutine and
88+
# inspect what it accepted from another: an unsynchronised access there passes
89+
# the plain run, so nothing else would catch it. The builder has cgo and gcc.
90+
RUN go test -race ./opensearch-gateway/... ./pkg/reconcilers/...
6791

6892
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -a -o manager main.go
6993

0 commit comments

Comments
 (0)