Skip to content

[TT-16068] Enhance Graceful Shutdown Sequence to introduce a waiting period for Gateway Deregistration#8241

Merged
lghiur merged 11 commits into
masterfrom
TT-16068-enhance-graceful-shutdown-sequence-to-introduce-a-waiting-period-for-gateway-deregistration
Jun 10, 2026
Merged

[TT-16068] Enhance Graceful Shutdown Sequence to introduce a waiting period for Gateway Deregistration#8241
lghiur merged 11 commits into
masterfrom
TT-16068-enhance-graceful-shutdown-sequence-to-introduce-a-waiting-period-for-gateway-deregistration

Conversation

@MaciekMis

@MaciekMis MaciekMis commented May 21, 2026

Copy link
Copy Markdown
Contributor

Description

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

Ticket Details

TT-16068
Status In Test
Summary Enhance Graceful Shutdown Sequence to introduce a waiting period for Gateway Deregistration

Generated at: 2026-06-03 11:29:38

@probelabs

probelabs Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

This PR enhances the graceful shutdown sequence of the Tyk Gateway by introducing a configurable waiting period. This allows orchestration platforms like Kubernetes to detect that the gateway is shutting down via its readiness probe, remove it from the load balancer's pool, and stop routing new traffic to it before the gateway process terminates. This helps prevent dropped requests during rolling updates and enables zero-downtime deployments.

The key change is the introduction of a new configuration parameter, graceful_shutdown_delay_seconds. When a shutdown signal is received, the gateway immediately marks itself as shutting down, causing the readiness endpoint to return a 503 Service Unavailable status. It then waits for the configured delay before proceeding with the existing shutdown logic, providing a window for traffic managers to react.

Files Changed Analysis

The changes are concise and targeted across 5 files:

  • config/config.go: Adds the GracefulShutdownDelaySeconds field to the main configuration struct.
  • cli/linter/schema.json: Updates the configuration schema to include the new graceful_shutdown_delay_seconds property for validation.
  • gateway/server.go: Introduces a shuttingDown atomic boolean to the Gateway struct. The shutdown signal handler is updated to set this flag and then wait for the configured delay before proceeding.
  • gateway/health_check.go: Modifies the readinessHandler to check the shuttingDown flag and return an HTTP 503 Service Unavailable if it is set.
  • gateway/health_check_test.go: Adds a unit test to verify that the readiness probe correctly returns a 503 status when the gateway is in the shutdown state.

Architecture & Impact Assessment

  • What this PR accomplishes: It improves the gateway's operational behavior in orchestrated environments, making rolling updates more reliable by preventing traffic from being sent to terminating instances.

  • Key technical changes introduced:

    1. Configurable Shutdown Delay: A new graceful_shutdown_delay_seconds parameter allows operators to define a waiting period.
    2. Immediate State Flagging: Upon receiving a termination signal, an atomic boolean shuttingDown is immediately set to true.
    3. Proactive Readiness Failure: The readiness probe begins failing with a 503 error as soon as the shuttingDown state is active.
    4. Delayed Shutdown: The actual server shutdown process is postponed by the configured delay, creating a window for traffic redirection.
  • Affected system components:

    • Gateway Configuration
    • Gateway Server Lifecycle & Signal Handling
    • Health & Readiness Check Endpoints
  • Shutdown Sequence Visualization:

sequenceDiagram
participant Orchestrator as "Orchestrator (e.g., K8s)"
participant LoadBalancer as "Load Balancer"
participant Gateway

Note over Gateway: Normal Operation
Orchestrator->>Gateway: GET /hello (Readiness Probe)
Gateway-->>Orchestrator: 200 OK

Note over Gateway: Shutdown Initiated
Orchestrator->>Gateway: Sends SIGTERM
Gateway->>Gateway: Sets shuttingDown = true

Note over Gateway: Readiness probe now fails
Orchestrator->>Gateway: GET /hello (Readiness Probe)
Gateway-->>Orchestrator: 503 Service Unavailable
Orchestrator->>LoadBalancer: Deregister Gateway instance
Note right of LoadBalancer: Stops sending new traffic

Gateway->>Gateway: Waits for `graceful_shutdown_delay_seconds`
Gateway->>Gateway: Proceeds with existing graceful shutdown

## Scope Discovery & Context Expansion
The impact of this change is primarily operational, enhancing the gateway's behavior as a component within modern cloud-native environments. The change is self-contained and does not affect other Tyk services.

- **Orchestration (Kubernetes/Nomad)**: A failing readiness probe is the standard mechanism to signal that a pod should be removed from a service's load balancing pool. This change makes the gateway a better citizen in such systems.
- **Load Balancers (AWS ALB, NGINX)**: External load balancers that use the readiness endpoint for health checks will correctly identify the node as unhealthy and remove it from the active backend pool before connections are forcefully terminated.


<details>
  <summary>Metadata</summary>

  - Review Effort: 2 / 5
  - Primary Label: enhancement


</details>
<!-- visor:section-end id="overview" -->

<!-- visor:thread-end key="TykTechnologies/tyk#8241@bcfdfa9" -->

---

*Powered by [Visor](https://probelabs.com/visor) from [Probelabs](https://probelabs.com)*

*Last updated: 2026-06-09T18:08:32.137Z | Triggered by: pr_updated | Commit: bcfdfa9*

💡 **TIP:** You can chat with Visor using `/visor ask <your question>`
<!-- /visor-comment-id:visor-thread-overview-TykTechnologies/tyk#8241 -->

@probelabs

probelabs Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Security Issues (1)

Severity Location Issue
🟡 Warning gateway/server.go:2225
The `graceful_shutdown_delay_seconds` configuration, if set to an excessively large value, could lead to a denial-of-service condition by preventing the gateway from shutting down in a timely manner. While this is an administrative setting, adding a reasonable upper limit would prevent accidental or malicious misconfiguration from causing prolonged service unavailability during shutdown or restart cycles.
💡 SuggestionEnforce a reasonable maximum value for `GracefulShutdownDelaySeconds`. For example, a limit of 300 seconds (5 minutes) would accommodate most graceful shutdown scenarios while preventing abuse. This check should be performed after loading the configuration.
🔧 Suggested Fix
const maxShutdownDelay = 300 // 5 minutes

// In Start() function, after config is loaded:
if gwConfig.GracefulShutdownDelaySeconds > maxShutdownDelay {
mainLog.Warningf("Configured graceful shutdown delay of %d seconds is too high. Capping at %d seconds.", gwConfig.GracefulShutdownDelaySeconds, maxShutdownDelay)
gwConfig.GracefulShutdownDelaySeconds = maxShutdownDelay
}

// ... existing shutdown logic ...
if gwConfig.GracefulShutdownDelaySeconds > 0 {
mainLog.Infof("Delaying graceful shutdown for %d seconds", gwConfig.GracefulShutdownDelaySeconds)

select {
case &lt;-time.After(time.Duration(gwConfig.GracefulShutdownDelaySeconds) * time.Second):
}

}

Security Issues (1)

Severity Location Issue
🟡 Warning gateway/server.go:2225
The `graceful_shutdown_delay_seconds` configuration, if set to an excessively large value, could lead to a denial-of-service condition by preventing the gateway from shutting down in a timely manner. While this is an administrative setting, adding a reasonable upper limit would prevent accidental or malicious misconfiguration from causing prolonged service unavailability during shutdown or restart cycles.
💡 SuggestionEnforce a reasonable maximum value for `GracefulShutdownDelaySeconds`. For example, a limit of 300 seconds (5 minutes) would accommodate most graceful shutdown scenarios while preventing abuse. This check should be performed after loading the configuration.
🔧 Suggested Fix
const maxShutdownDelay = 300 // 5 minutes

// In Start() function, after config is loaded:
if gwConfig.GracefulShutdownDelaySeconds > maxShutdownDelay {
mainLog.Warningf("Configured graceful shutdown delay of %d seconds is too high. Capping at %d seconds.", gwConfig.GracefulShutdownDelaySeconds, maxShutdownDelay)
gwConfig.GracefulShutdownDelaySeconds = maxShutdownDelay
}

// ... existing shutdown logic ...
if gwConfig.GracefulShutdownDelaySeconds > 0 {
mainLog.Infof("Delaying graceful shutdown for %d seconds", gwConfig.GracefulShutdownDelaySeconds)

select {
case &lt;-time.After(time.Duration(gwConfig.GracefulShutdownDelaySeconds) * time.Second):
}

}

\n\n ### Architecture Issues (1)
Severity Location Issue
🟡 Warning gateway/server.go:2226-2228
The use of `select` with a single `time.After` case to create a delay is functionally correct but less direct than using `time.Sleep`. Since the delay does not need to be interruptible by other events (like context cancellation or another signal), `time.Sleep` would be a simpler and more idiomatic way to express a fixed-duration pause, improving code clarity.
💡 SuggestionReplace the `select` block with a `time.Sleep` call for better readability and simplicity, as there are no other channels to select from. This makes the intent of a fixed-duration pause more explicit. The current implementation is functionally equivalent but slightly more verbose for a simple delay.

✅ Performance Check Passed

No performance issues found – changes LGTM.

Quality Issues (1)

Severity Location Issue
🟡 Warning gateway/server.go:2223-2229
The new graceful shutdown delay logic, which is critical for ensuring zero-downtime deployments, is not covered by any automated tests. The current implementation inside the signal-handling goroutine in `Start()` is difficult to test. This lack of coverage creates a risk of future regressions going undetected.
💡 SuggestionTo improve testability, consider refactoring the shutdown logic into a separate, testable function. This function could accept dependencies like a timer, which can be mocked during tests. This would allow for verifying that the delay is correctly applied when `GracefulShutdownDelaySeconds` is positive and skipped when it is zero or negative.

Powered by Visor from Probelabs

Last updated: 2026-06-09T18:08:11.935Z | Triggered by: pr_updated | Commit: bcfdfa9

💡 TIP: You can chat with Visor using /visor ask <your question>

@radkrawczyk radkrawczyk reopened this May 27, 2026
…e-to-introduce-a-waiting-period-for-gateway-deregistration
@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

🎯 Recommended Merge Targets

Based on JIRA ticket TT-16068: Enhance Graceful Shutdown Sequence to introduce a waiting period for Gateway Deregistration

Fix Version: Tyk 5.14.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

📋 Workflow

  1. Merge this PR to master first

…e-to-introduce-a-waiting-period-for-gateway-deregistration
…e-to-introduce-a-waiting-period-for-gateway-deregistration
…e-to-introduce-a-waiting-period-for-gateway-deregistration
Comment thread gateway/server.go
mainLog.Infof("Shutdown signal received: %v. Initiating graceful shutdown...", sig)

gw.shuttingDown.Store(true)
if gwConfig.GracefulShutdownDelaySeconds > 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MaciekMis you can move this into it's own function so it's easier to test. Wdyt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, we have already created a ticket to refactor this area and cover it with tests: https://tyktech.atlassian.net/browse/TT-17354

…e-to-introduce-a-waiting-period-for-gateway-deregistration
…e-to-introduce-a-waiting-period-for-gateway-deregistration
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: bcfdfa9
Failed at: 2026-06-09 18:07:02 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to get Jira issue: failed to fetch Jira issue TT-16068: Issue does not exist or you do not have permission to see it.: request failed. Please analyze the request body for more details. Status code: 404

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@sonarqubecloud

sonarqubecloud Bot commented Jun 9, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
42.9% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@lghiur
lghiur merged commit 83c50e7 into master Jun 10, 2026
36 of 55 checks passed
@lghiur
lghiur deleted the TT-16068-enhance-graceful-shutdown-sequence-to-introduce-a-waiting-period-for-gateway-deregistration branch June 10, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants