Skip to content

Commit 0824268

Browse files
committed
fix(ha): refresh gateway membership before shutdown redirects
A stopping replica redirected its supervisors using its cached ring, which the membership worker refreshes only every 10s. During a rolling update the replacement for the previous pod is often only seconds old, so the cached ring missed it and still named the departed pod. The redirect then went nowhere or to a dead pod, whose 10s connect timeout outlasted the handoff grace, and the sandbox was demoted mid-roll. Re-read live membership, bounded to 2s, just before session shutdown computes redirects. Signed-off-by: Piotr Mlocek <[email protected]>
1 parent 7a45dec commit 0824268

3 files changed

Lines changed: 88 additions & 2 deletions

File tree

‎crates/openshell-server/src/gateway_members.rs‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ pub const MEMBER_REFRESH_INTERVAL: Duration = Duration::from_secs(10);
2828
/// Upper bound on replicas read back when building the ring.
2929
const MEMBER_LIST_LIMIT: u32 = 1024;
3030

31+
/// How long shutdown waits to re-read membership before redirecting sessions.
32+
pub const SHUTDOWN_RING_REFRESH_TIMEOUT: Duration = Duration::from_secs(2);
33+
3134
fn member_object_id(replica_id: &str) -> String {
3235
format!("gateway-member:{replica_id}")
3336
}
@@ -145,6 +148,59 @@ impl GatewayMemberIndex {
145148
let ring = GatewayRing::new(members.iter().map(|member| member.replica_id.as_str()));
146149
Ok((ring, members))
147150
}
151+
152+
/// Build the ring and peer endpoints a stopping replica redirects to: the
153+
/// live replicas other than `replica_id`.
154+
pub async fn ring_without(
155+
&self,
156+
replica_id: &str,
157+
) -> PersistenceResult<(GatewayRing, HashMap<String, String>)> {
158+
let members = self
159+
.live_members()
160+
.await?
161+
.into_iter()
162+
.filter(|member| member.replica_id != replica_id)
163+
.collect::<Vec<_>>();
164+
let ring = GatewayRing::new(members.iter().map(|member| member.replica_id.as_str()));
165+
let peers = members
166+
.into_iter()
167+
.map(|member| (member.replica_id, member.peer_endpoint))
168+
.collect();
169+
Ok((ring, peers))
170+
}
171+
}
172+
173+
/// Re-read membership just before shutdown redirects supervisor sessions.
174+
///
175+
/// The worker refreshes the ring only every [`MEMBER_REFRESH_INTERVAL`]. During
176+
/// a rolling update the cached ring can miss a replacement that just started
177+
/// and still name a replica that already left, so redirects built from it send
178+
/// supervisors nowhere or to a dead pod. The read is bounded by `timeout`; on
179+
/// failure the cached ring stays in place.
180+
pub async fn refresh_ring_for_shutdown(state: &crate::ServerState, timeout: Duration) {
181+
if state.store.is_single_replica() || state.peer_endpoint.is_none() {
182+
return;
183+
}
184+
let index = GatewayMemberIndex::new(state.store.clone(), MEMBER_TTL);
185+
match tokio::time::timeout(timeout, index.ring_without(&state.replica_id)).await {
186+
Ok(Ok((ring, peers))) => {
187+
if let Ok(mut slot) = state.gateway_ring.write() {
188+
*slot = ring;
189+
}
190+
if let Ok(mut slot) = state.gateway_peers.write() {
191+
*slot = peers;
192+
}
193+
}
194+
Ok(Err(error)) => {
195+
tracing::warn!(%error, "gateway membership: shutdown ring refresh failed");
196+
}
197+
Err(_) => {
198+
tracing::warn!(
199+
timeout_ms = timeout.as_millis(),
200+
"gateway membership: shutdown ring refresh timed out"
201+
);
202+
}
203+
}
148204
}
149205

150206
/// Keep this replica's membership record fresh and refresh the local ring.
@@ -259,6 +315,28 @@ mod tests {
259315
);
260316
}
261317

318+
#[tokio::test]
319+
async fn shutdown_ring_includes_new_replicas_and_excludes_departed_ones() {
320+
let index = test_index(MEMBER_TTL).await;
321+
index.register("gw-old", "https://gw-old").await.unwrap();
322+
index
323+
.register("gw-stopping", "https://gw-stopping")
324+
.await
325+
.unwrap();
326+
// A rolling update replaces gw-old after the stopping replica last
327+
// refreshed its cached ring.
328+
index.deregister("gw-old").await.unwrap();
329+
index.register("gw-new", "https://gw-new").await.unwrap();
330+
331+
let (ring, peers) = index.ring_without("gw-stopping").await.unwrap();
332+
333+
assert_eq!(ring.owner_for("sb-1"), Some("gw-new"));
334+
assert_eq!(
335+
peers,
336+
HashMap::from([("gw-new".to_string(), "https://gw-new".to_string())])
337+
);
338+
}
339+
262340
#[tokio::test]
263341
async fn register_is_idempotent_and_updates_the_endpoint() {
264342
let index = test_index(MEMBER_TTL).await;

‎crates/openshell-server/src/lib.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -987,6 +987,13 @@ pub(crate) async fn run_server(
987987
}
988988

989989
let compute_cleanup = state.compute.cleanup_on_shutdown().await;
990+
// Closing sessions redirect their supervisors through the ring, so pick up
991+
// replicas that joined since the last periodic refresh.
992+
gateway_members::refresh_ring_for_shutdown(
993+
&state,
994+
gateway_members::SHUTDOWN_RING_REFRESH_TIMEOUT,
995+
)
996+
.await;
990997
// A stopped supervisor may still have a detached task deleting its owner
991998
// record. Drain it even when compute cleanup failed before exiting Tokio.
992999
let session_cleanup = state

‎docs/kubernetes/high-availability.mdx‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,9 @@ When a non-owner gateway pod stops, Kubernetes removes it from ready Service
178178
endpoints and new requests go to another replica. Existing requests through the
179179
stopped pod disconnect and must be retried.
180180

181-
When an owner gateway pod stops gracefully, it redirects its connected
182-
supervisors to the replicas that now own them. After a crash, supervisors
181+
When an owner gateway pod stops gracefully, it re-reads gateway membership and
182+
redirects its connected supervisors to the replicas that now own them,
183+
including replicas that started moments earlier. After a crash, supervisors
183184
reconnect through the gateway Service. Either way, they publish new ownership. Other replicas re-read ownership
184185
while waiting to relay a request. A request fails if no fresh, reachable owner
185186
appears before its normal deadline.

0 commit comments

Comments
 (0)