Repository navigation
Commit cf36387
committed
fix(opensearch-operator): bound every call to a cluster's HTTP API
Nothing limited how long a call to a cluster's HTTP API could take. The client
builds its own http.Transport, and a fresh transport has no dial, no TLS
handshake and no response-header timeout; the config sets no http.Client, so
there is no timeout there either; and every call the client makes on its own
behalf passes context.Background(). The Timeout field the request types carry is
the server-side query parameter, which bounds how long the server waits before
answering and helps only once something answers.
Each controller runs a single worker, so one call that never returns stops every
OpenSearchCluster from being reconciled, and the finalizer on the next deletion
waits out its whole termination timeout with nothing in the log to say why. The
measurement is in #4332: the operator wrote its last line seconds after excluding
a bootstrap pod from the voting configuration, which is an HTTP call, to a pod
that was NotReady, and stayed Running with no restarts and nothing further in the
log.
The patch gives the client's own calls a 30s deadline, and the transport a 10s
dial, a 10s TLS handshake and a 60s response-header timeout. The transport bound
is deliberately the looser one: it is the only bound on the calls that take a
context from the caller and hand the still-open body back, where cancelling on
return would fail the caller's read, and on the product check opensearch-go runs
on a client's first request, which carries a context.Background() of its own
inside the library and cannot be reached from here.
This bounds the wait; it does not make an unresponsive cluster answer. A call
that passes its deadline now fails the reconcile, which requeues, instead of
retiring the worker.
Upstream is the better home for the change, and the patch goes away with the
first release that carries one.
Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>1 parent b18281e commit cf36387
2 files changed
Lines changed: 410 additions & 3 deletions
File tree
- packages/system/opensearch-operator/images/opensearch-operator
- patches
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | | - | |
30 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
31 | 45 | | |
32 | 46 | | |
33 | 47 | | |
| |||
0 commit comments