Skip to content

Commit dea07e4

Browse files
committed
fix(middleware): accept websocket sequence gaps
Signed-off-by: Piotr Mlocek <[email protected]>
1 parent 7c05b7e commit dea07e4

4 files changed

Lines changed: 110 additions & 16 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ openshell logs <sandbox-name> --tail --source sandbox
104104

105105
The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if `Describe` is unavailable, a manifest exposes duplicate operation/phase bindings, the registration claims the reserved `openshell/` namespace, or body, message, and timeout limits are invalid. Supported V1 bindings are `HTTP_REQUEST/PRE_CREDENTIALS` and `WEBSOCKET_MESSAGE/PRE_CREDENTIALS`. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its `network_middlewares` config.
106106

107-
At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the HTTP request or closes the WebSocket, while `fail_open` bypasses only that stage and emits a detection finding. A broken fail-open WebSocket stream is disabled for later messages on that connection and emits `openshell.middleware.websocket_stage_disabled`. Confirm preflight, session-start, monotonically increasing message sequences, and session-end in service logs. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
107+
At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the HTTP request or closes the WebSocket, while `fail_open` bypasses only that stage and emits a detection finding. A fail-open per-message capacity failure bypasses that message without disabling the stage. A timeout, transport failure, stream closure, missing or invalid response, duplicate or regressed sequence, or other failure that makes an established WebSocket stream unreliable disables that stage for later messages on the connection and emits `openshell.middleware.websocket_stage_disabled`. Confirm preflight, session-start, and session-end in service logs. WebSocket message sequences are allocated session-wide; each stage receives a strictly increasing subset, so gaps are valid when messages are not delivered to that stage. Zero, duplicate, or regressed sequences are protocol errors. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
108108

109109
### Step 4: Check Docker-Backed Gateways
110110

@@ -401,7 +401,7 @@ openshell logs <sandbox-name>
401401
| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response |
402402
| Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields |
403403
| HTTP request returns `middleware_failed`, or WebSocket closes with `1008` | Selected stage failed or explicitly denied admitted traffic | Sandbox OCSF logs; policy-local middleware config; service availability; binding operation; `on_error` |
404-
| WebSocket messages stop reaching middleware after one failure | A fail-open stage stream was disabled for the rest of the connection | `openshell.middleware.websocket_stage_disabled`; middleware timeout/stream logs; reconnect to create a fresh stream |
404+
| WebSocket messages stop reaching middleware after one failure | A fail-open stage stream was disabled for the rest of the connection | `openshell.middleware.websocket_stage_disabled`; middleware timeout/stream/protocol logs. A per-message capacity bypass alone leaves the stage active. Reconnect to create a fresh stream after a genuine stream failure |
405405
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
406406
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
407407
| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |

‎crates/openshell-supervisor-middleware-builtins/src/lib.rs‎

Lines changed: 46 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ impl BuiltinMiddlewareService {
6161
tokio::spawn(async move {
6262
let mut config = None;
6363
let mut started = false;
64-
let mut next_sequence = 1u64;
64+
let mut sequence_lower_bound = Some(1u64);
6565

6666
while let Some(request) = requests.next().await {
6767
let request = match request {
@@ -112,10 +112,11 @@ impl BuiltinMiddlewareService {
112112
Ok(None)
113113
}
114114
Some(web_socket_evaluation_request::Request::Message(message)) if started => {
115-
if message.sequence != next_sequence {
116-
Err(Status::invalid_argument(
117-
"WebSocket message sequence is not monotonic",
118-
))
115+
if let Err(error) = advance_sequence_lower_bound(
116+
&mut sequence_lower_bound,
117+
message.sequence,
118+
) {
119+
Err(error)
119120
} else if message.direction != WebSocketDirection::ClientToUpstream as i32
120121
|| message.message_type != WebSocketMessageType::Text as i32
121122
{
@@ -130,16 +131,13 @@ impl BuiltinMiddlewareService {
130131
&message.payload,
131132
selected_config,
132133
) {
133-
Ok(result) => {
134-
next_sequence = next_sequence.saturating_add(1);
135-
Ok(Some(WebSocketEvaluationResponse {
136-
response: Some(
137-
web_socket_evaluation_response::Response::MessageResult(
138-
result,
139-
),
134+
Ok(result) => Ok(Some(WebSocketEvaluationResponse {
135+
response: Some(
136+
web_socket_evaluation_response::Response::MessageResult(
137+
result,
140138
),
141-
}))
142-
}
139+
),
140+
})),
143141
Err(error) => Err(Status::invalid_argument(error.to_string())),
144142
}
145143
}
@@ -172,6 +170,24 @@ impl BuiltinMiddlewareService {
172170
}
173171
}
174172

173+
fn advance_sequence_lower_bound(
174+
lower_bound: &mut Option<u64>,
175+
sequence: u64,
176+
) -> std::result::Result<(), Status> {
177+
let Some(current_lower_bound) = *lower_bound else {
178+
return Err(Status::invalid_argument(
179+
"WebSocket message sequence must be strictly increasing",
180+
));
181+
};
182+
if sequence < current_lower_bound {
183+
return Err(Status::invalid_argument(
184+
"WebSocket message sequence must be strictly increasing",
185+
));
186+
}
187+
*lower_bound = sequence.checked_add(1);
188+
Ok(())
189+
}
190+
175191
#[tonic::async_trait]
176192
impl SupervisorMiddleware for BuiltinMiddlewareService {
177193
type EvaluateWebSocketStream = WebSocketResponseStream;
@@ -428,4 +444,20 @@ mod tests {
428444
.is_err()
429445
);
430446
}
447+
448+
#[test]
449+
fn websocket_sequence_lower_bound_accepts_gaps_and_rejects_reuse() {
450+
let mut lower_bound = Some(1);
451+
advance_sequence_lower_bound(&mut lower_bound, 2).expect("first delivered sequence");
452+
assert_eq!(lower_bound, Some(3));
453+
assert!(advance_sequence_lower_bound(&mut lower_bound, 2).is_err());
454+
assert!(advance_sequence_lower_bound(&mut lower_bound, 1).is_err());
455+
456+
advance_sequence_lower_bound(&mut lower_bound, 7).expect("forward gap");
457+
assert_eq!(lower_bound, Some(8));
458+
459+
advance_sequence_lower_bound(&mut lower_bound, u64::MAX).expect("last sequence");
460+
assert_eq!(lower_bound, None);
461+
assert!(advance_sequence_lower_bound(&mut lower_bound, u64::MAX).is_err());
462+
}
431463
}

‎crates/openshell-supervisor-middleware/src/lib.rs‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4170,6 +4170,61 @@ mod tests {
41704170
.await;
41714171
}
41724172

4173+
#[tokio::test]
4174+
async fn builtin_regex_redacts_after_fail_open_message_capacity_gap() {
4175+
let runner = builtin_runner();
4176+
let chain = [entry("regex-redactor", OnError::FailOpen)];
4177+
let preflight = runner
4178+
.preflight_websocket(
4179+
&chain,
4180+
WebSocketPreflightInput {
4181+
session_id: "builtin-regex-gap-session".into(),
4182+
request_id: "request".into(),
4183+
sandbox_id: "sandbox".into(),
4184+
scheme: "wss".into(),
4185+
host: "api.openai.com".into(),
4186+
port: 443,
4187+
path: "/v1/responses".into(),
4188+
requested_subprotocols: vec!["realtime".into()],
4189+
},
4190+
)
4191+
.await
4192+
.expect("preflight");
4193+
assert!(preflight.allowed);
4194+
let mut session = preflight.session.expect("built-in chose to inspect");
4195+
assert!(session.start("realtime").await.allowed);
4196+
4197+
let oversized_payload = vec![b'a'; 256 * 1024 + 1];
4198+
let oversized = session.evaluate_text(oversized_payload.clone()).await;
4199+
assert!(oversized.allowed);
4200+
assert_eq!(oversized.payload, oversized_payload);
4201+
assert_eq!(oversized.invocations[0].sequence, Some(1));
4202+
assert_eq!(
4203+
oversized.invocations[0].outcome,
4204+
WebSocketInvocationOutcome::FailOpen
4205+
);
4206+
assert!(!oversized.invocations[0].stage_disabled);
4207+
4208+
let original = br#"{"type":"response.create","response":{"input":"sk-ABCDEFGHIJKLMNOP"}}"#;
4209+
let redacted = session.evaluate_text(original.to_vec()).await;
4210+
assert!(redacted.allowed);
4211+
assert_eq!(
4212+
String::from_utf8(redacted.payload).expect("transformed UTF-8"),
4213+
r#"{"type":"response.create","response":{"input":"[REDACTED]"}}"#
4214+
);
4215+
assert_eq!(redacted.invocations[0].sequence, Some(2));
4216+
assert_eq!(
4217+
redacted.invocations[0].outcome,
4218+
WebSocketInvocationOutcome::Allow
4219+
);
4220+
assert!(redacted.invocations[0].transformed);
4221+
assert!(!redacted.invocations[0].stage_disabled);
4222+
4223+
session
4224+
.end(openshell_core::proto::WebSocketSessionEndReason::NormalClose)
4225+
.await;
4226+
}
4227+
41734228
#[tokio::test]
41744229
async fn in_process_websocket_endpoint_redacts_openai_event() {
41754230
let service = OpenAiRedactionService::default();

‎proto/supervisor_middleware.proto‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,9 @@ enum WebSocketSessionEndReason {
157157
}
158158

159159
// WebSocketEvaluationRequest is one ordered event in a stage-local stream.
160+
// Message sequence numbers identify logical messages session-wide. A stage
161+
// receives a strictly increasing subset of those numbers; gaps are valid when
162+
// session messages are not delivered to that stage.
160163
message WebSocketEvaluationRequest {
161164
oneof request {
162165
WebSocketPreflight preflight = 1;
@@ -195,6 +198,9 @@ message WebSocketSessionStart {
195198

196199
// WebSocketMessage contains one complete reconstructed logical message.
197200
message WebSocketMessage {
201+
// Session-global sequence starting at 1. Values delivered to one stage must
202+
// strictly increase but need not be contiguous. Reject zero, duplicates, and
203+
// regressions; accept gaps.
198204
uint64 sequence = 1;
199205
WebSocketDirection direction = 2;
200206
WebSocketMessageType message_type = 3;
@@ -221,6 +227,7 @@ message WebSocketPreflightDecision {
221227
// one message. A replacement preserves the input message type; text
222228
// replacements must be valid UTF-8.
223229
message WebSocketMessageResult {
230+
// Must exactly match the sequence of the corresponding WebSocketMessage.
224231
uint64 sequence = 1;
225232
Decision decision = 2;
226233
bytes replacement = 3;

0 commit comments

Comments
 (0)