…8314)

## Description

`/hello` and `/ready` served a stale "pass" status forever when Redis
failed before the Dashboard (regression in 5.8.14 / 5.13.0).

Root cause: the liveness probe `Ping()` shared `sendHeartBeat`, whose
403 branch calls `DashService.Register(ctx)` — since #8188 an unbounded
5s retry loop — with `context.Background()`. With Redis down and the
Dashboard up, the Dashboard answers the heartbeat with 403 and every
registration attempt with `409 + Status:"Error"`, so `Register()` never
returned, the dashboard probe goroutine never released
`gatherHealthChecks`' `wg.Wait()`, the cached health map was never
refreshed, and the synchronous ticker loop wedged for the duration of
the outage. Kubernetes kept routing traffic to Gateways without Redis.

The fix, in three layers (none of which reverts the TT-16865 behaviour —
startup registration retry is unchanged):

1. **The probe no longer triggers re-registration.** The heartbeat HTTP
exchange is extracted into `doHeartBeat()`, returning a sentinel on 403.
`sendHeartBeat()` (heartbeat loop) keeps 403 → `Register(ctx)`; `Ping()`
treats 403 as "dashboard reachable → healthy", preserving v5.8.13
endpoint payloads exactly. Re-registration is owned by the
`StartBeating` loop.
2. **The probe is bounded.** `Ping()` runs under a context derived from
`gw.ctx`, bounded at half the configured `liveness_check.check_duration`
(so the probe always reports its own error before the round's barrier
expires), so no probe request can outlive a check interval.
3. **The health-check barrier is wedge-proof.** `gatherHealthChecks()`
waits for its probes only up to the check interval, commits a race-safe
snapshot of what completed, and reports any unfinished component as
`fail` / `"health check timed out"` — no single hung dependency can ever
freeze `/hello` and `/ready` again, whatever the failure order.

## Related Issue

https://tyktech.atlassian.net/browse/TT-17486

## Motivation and Context

In Tyk Pro deployments on Kubernetes the readiness probe is the only
signal gating traffic to Gateway Pods. With this regression, a Gateway
that lost Redis kept answering `/ready` with `200/pass` indefinitely, so
traffic continued to be routed to instances that could not serve it. The
failure was order-dependent (only when Redis died before the Dashboard),
which is what made it hard to diagnose; the storage-library theory was
ruled out (all good/bad tags pin `storage v1.3.1`; `health_check.go` is
byte-identical across them — only the #8188 `Register()` rewrite tracks
the good/bad split).

## How This Has Been Tested

TDD throughout — every regression test was watched failing on the
unfixed code first:

- `TestPing_RedisDownDashboardUp_DoesNotBlockOrReRegister` — stubs the
Dashboard's Redis-down wire behaviour (heartbeat 403, registration `409
+ Status:"Error"`). Before: `Ping()` blocked >10s retrying registration;
after: returns `nil` in ~20ms with **zero** registration attempts.
- `TestPing_HangingDashboard_BoundedByTimeout` — unresponsive Dashboard
with the production-default 30s client; probe now fails at ~5s.
- `TestGatherHealthChecks_HungProbeDoesNotWedgeTheLoop` — injected
`DashService` whose `Ping` blocks forever; the round completes within
the interval with `redis: pass`, `dashboard: fail/"health check timed
out"`.
- `TestSendHeartBeat_Forbidden_ReRegisters` — pins the recovery path:
the heartbeat loop still re-registers on 403 (node identity and nonce
refreshed).
- Characterization tests written before the refactor pin preserved
behaviour: 200 heartbeat updates the nonce; transport errors still
report `"dashboard is down? Heartbeat is failing"`.
- Full dashboard-register + health-check suites green, including the
TT-16865 tests (`TestRegister_Retries`,
`TestRegister_DuplicateSession409`); `-race` clean on the new tests; `go
vet`/`gofmt` clean.
- Differential validation of the root cause: the Redis-down probe test
passes on a v5.8.13 worktree unmodified and fails on 5.8.14+ code.

Not covered here (tracked in the ticket's test plan for QA/e2e):
full-stack failure-order matrix, recovery/flapping soak, and the
Kubernetes readiness scenario.

## Types of changes

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

- [x] I ensured that the documentation is up to date
- [x] I explained why this PR updates go.mod in detail with reasoning
why it's required (go.mod untouched)
- [x] I would like a code coverage CI quality gate exception and have
explained why (not needed)

---------

Co-authored-by: Kofo Okesola <[email protected]>
Co-authored-by: Ilija Bojanovic <[email protected]>

…on node re-registration (#8314) (#8327)

[TT-17486] fix: stop liveness probe blocking on node re-registration
(#8314)

## Description

`/hello` and `/ready` served a stale "pass" status forever when Redis
failed before the Dashboard (regression in 5.8.14 / 5.13.0).

Root cause: the liveness probe `Ping()` shared `sendHeartBeat`, whose
403 branch calls `DashService.Register(ctx)` — since #8188 an unbounded
5s retry loop — with `context.Background()`. With Redis down and the
Dashboard up, the Dashboard answers the heartbeat with 403 and every
registration attempt with `409 + Status:"Error"`, so `Register()` never
returned, the dashboard probe goroutine never released
`gatherHealthChecks`' `wg.Wait()`, the cached health map was never
refreshed, and the synchronous ticker loop wedged for the duration of
the outage. Kubernetes kept routing traffic to Gateways without Redis.

The fix, in three layers (none of which reverts the TT-16865 behaviour —
startup registration retry is unchanged):

1. **The probe no longer triggers re-registration.** The heartbeat HTTP
exchange is extracted into `doHeartBeat()`, returning a sentinel on 403.
`sendHeartBeat()` (heartbeat loop) keeps 403 → `Register(ctx)`; `Ping()`
treats 403 as "dashboard reachable → healthy", preserving v5.8.13
endpoint payloads exactly. Re-registration is owned by the
`StartBeating` loop.
2. **The probe is bounded.** `Ping()` runs under a context derived from
`gw.ctx`, bounded at half the configured `liveness_check.check_duration`
(so the probe always reports its own error before the round's barrier
expires), so no probe request can outlive a check interval.
3. **The health-check barrier is wedge-proof.** `gatherHealthChecks()`
waits for its probes only up to the check interval, commits a race-safe
snapshot of what completed, and reports any unfinished component as
`fail` / `"health check timed out"` — no single hung dependency can ever
freeze `/hello` and `/ready` again, whatever the failure order.

## Related Issue

https://tyktech.atlassian.net/browse/TT-17486

## Motivation and Context

In Tyk Pro deployments on Kubernetes the readiness probe is the only
signal gating traffic to Gateway Pods. With this regression, a Gateway
that lost Redis kept answering `/ready` with `200/pass` indefinitely, so
traffic continued to be routed to instances that could not serve it. The
failure was order-dependent (only when Redis died before the Dashboard),
which is what made it hard to diagnose; the storage-library theory was
ruled out (all good/bad tags pin `storage v1.3.1`; `health_check.go` is
byte-identical across them — only the #8188 `Register()` rewrite tracks
the good/bad split).

## How This Has Been Tested

TDD throughout — every regression test was watched failing on the
unfixed code first:

- `TestPing_RedisDownDashboardUp_DoesNotBlockOrReRegister` — stubs the
Dashboard's Redis-down wire behaviour (heartbeat 403, registration `409
+ Status:"Error"`). Before: `Ping()` blocked >10s retrying registration;
after: returns `nil` in ~20ms with **zero** registration attempts.
- `TestPing_HangingDashboard_BoundedByTimeout` — unresponsive Dashboard
with the production-default 30s client; probe now fails at ~5s.
- `TestGatherHealthChecks_HungProbeDoesNotWedgeTheLoop` — injected
`DashService` whose `Ping` blocks forever; the round completes within
the interval with `redis: pass`, `dashboard: fail/"health check timed
out"`.
- `TestSendHeartBeat_Forbidden_ReRegisters` — pins the recovery path:
the heartbeat loop still re-registers on 403 (node identity and nonce
refreshed).
- Characterization tests written before the refactor pin preserved
behaviour: 200 heartbeat updates the nonce; transport errors still
report `"dashboard is down? Heartbeat is failing"`.
- Full dashboard-register + health-check suites green, including the
TT-16865 tests (`TestRegister_Retries`,
`TestRegister_DuplicateSession409`); `-race` clean on the new tests; `go
vet`/`gofmt` clean.
- Differential validation of the root cause: the Redis-down probe test
passes on a v5.8.13 worktree unmodified and fails on 5.8.14+ code.

Not covered here (tracked in the ticket's test plan for QA/e2e):
full-stack failure-order matrix, recovery/flapping soak, and the
Kubernetes readiness scenario.

## Types of changes

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

- [x] I ensured that the documentation is up to date
- [x] I explained why this PR updates go.mod in detail with reasoning
why it's required (go.mod untouched)
- [x] I would like a code coverage CI quality gate exception and have
explained why (not needed)

---------

Co-authored-by: Kofo Okesola <[email protected]>
Co-authored-by: Ilija Bojanovic <[email protected]>

[TT-17486]:
https://tyktech.atlassian.net/browse/TT-17486?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: Matias <[email protected]>
Co-authored-by: Kofo Okesola <[email protected]>
Co-authored-by: Ilija Bojanovic <[email protected]>

… on node re-registration (#8314) (#8328)

[TT-17486] fix: stop liveness probe blocking on node re-registration
(#8314)

## Description

`/hello` and `/ready` served a stale "pass" status forever when Redis
failed before the Dashboard (regression in 5.8.14 / 5.13.0).

Root cause: the liveness probe `Ping()` shared `sendHeartBeat`, whose
403 branch calls `DashService.Register(ctx)` — since #8188 an unbounded
5s retry loop — with `context.Background()`. With Redis down and the
Dashboard up, the Dashboard answers the heartbeat with 403 and every
registration attempt with `409 + Status:"Error"`, so `Register()` never
returned, the dashboard probe goroutine never released
`gatherHealthChecks`' `wg.Wait()`, the cached health map was never
refreshed, and the synchronous ticker loop wedged for the duration of
the outage. Kubernetes kept routing traffic to Gateways without Redis.

The fix, in three layers (none of which reverts the TT-16865 behaviour —
startup registration retry is unchanged):

1. **The probe no longer triggers re-registration.** The heartbeat HTTP
exchange is extracted into `doHeartBeat()`, returning a sentinel on 403.
`sendHeartBeat()` (heartbeat loop) keeps 403 → `Register(ctx)`; `Ping()`
treats 403 as "dashboard reachable → healthy", preserving v5.8.13
endpoint payloads exactly. Re-registration is owned by the
`StartBeating` loop.
2. **The probe is bounded.** `Ping()` runs under a context derived from
`gw.ctx`, bounded at half the configured `liveness_check.check_duration`
(so the probe always reports its own error before the round's barrier
expires), so no probe request can outlive a check interval.
3. **The health-check barrier is wedge-proof.** `gatherHealthChecks()`
waits for its probes only up to the check interval, commits a race-safe
snapshot of what completed, and reports any unfinished component as
`fail` / `"health check timed out"` — no single hung dependency can ever
freeze `/hello` and `/ready` again, whatever the failure order.

## Related Issue

https://tyktech.atlassian.net/browse/TT-17486

## Motivation and Context

In Tyk Pro deployments on Kubernetes the readiness probe is the only
signal gating traffic to Gateway Pods. With this regression, a Gateway
that lost Redis kept answering `/ready` with `200/pass` indefinitely, so
traffic continued to be routed to instances that could not serve it. The
failure was order-dependent (only when Redis died before the Dashboard),
which is what made it hard to diagnose; the storage-library theory was
ruled out (all good/bad tags pin `storage v1.3.1`; `health_check.go` is
byte-identical across them — only the #8188 `Register()` rewrite tracks
the good/bad split).

## How This Has Been Tested

TDD throughout — every regression test was watched failing on the
unfixed code first:

- `TestPing_RedisDownDashboardUp_DoesNotBlockOrReRegister` — stubs the
Dashboard's Redis-down wire behaviour (heartbeat 403, registration `409
+ Status:"Error"`). Before: `Ping()` blocked >10s retrying registration;
after: returns `nil` in ~20ms with **zero** registration attempts.
- `TestPing_HangingDashboard_BoundedByTimeout` — unresponsive Dashboard
with the production-default 30s client; probe now fails at ~5s.
- `TestGatherHealthChecks_HungProbeDoesNotWedgeTheLoop` — injected
`DashService` whose `Ping` blocks forever; the round completes within
the interval with `redis: pass`, `dashboard: fail/"health check timed
out"`.
- `TestSendHeartBeat_Forbidden_ReRegisters` — pins the recovery path:
the heartbeat loop still re-registers on 403 (node identity and nonce
refreshed).
- Characterization tests written before the refactor pin preserved
behaviour: 200 heartbeat updates the nonce; transport errors still
report `"dashboard is down? Heartbeat is failing"`.
- Full dashboard-register + health-check suites green, including the
TT-16865 tests (`TestRegister_Retries`,
`TestRegister_DuplicateSession409`); `-race` clean on the new tests; `go
vet`/`gofmt` clean.
- Differential validation of the root cause: the Redis-down probe test
passes on a v5.8.13 worktree unmodified and fails on 5.8.14+ code.

Not covered here (tracked in the ticket's test plan for QA/e2e):
full-stack failure-order matrix, recovery/flapping soak, and the
Kubernetes readiness scenario.

## Types of changes

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

- [x] I ensured that the documentation is up to date
- [x] I explained why this PR updates go.mod in detail with reasoning
why it's required (go.mod untouched)
- [x] I would like a code coverage CI quality gate exception and have
explained why (not needed)

---------

Co-authored-by: Kofo Okesola <[email protected]>
Co-authored-by: Ilija Bojanovic <[email protected]>

[TT-17486]:
https://tyktech.atlassian.net/browse/TT-17486?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: Matias <[email protected]>
Co-authored-by: Kofo Okesola <[email protected]>
Co-authored-by: Ilija Bojanovic <[email protected]>