Skip to content

Commit 39df7f7

Browse files
author
clau-031c
committed
backend: in-flight request tasks are OWNED by the socket, not detached (pingdotgg#476)
My first pingdotgg#432 fix used a bare tokio::spawn. That un-starves the reader and DETACHES the work in the same line: orchestration.subscribeThread attaches a durable tail and streams for the life of the thread, so a client disconnecting mid-stream left a task writing into a tx whose receiver is gone, holding an AppState clone. Reconnect and there are two. Serialization had been the backpressure and I removed it without putting ownership in its place. The JoinSet design is claude-4e91's and it is better than what I wrote. Their cell declares it, reaps it and shuts it down but never spawns into it (there is no inflight.spawn in that file, and :1742 is still the inline await), so this is their design with the spawn actually connected -- not a second copy of it. Ownership is now structural: the set is owned by handle_socket, so tasks cannot outlive the connection, and shutdown().await on close makes them gone BEFORE the function returns rather than eventually. try_join_next each iteration reaps completed slots, which a long-lived socket otherwise accumulates one per request. AppState::inflight_requests is an OBSERVABLE, not a control -- nothing reads it to decide anything, so it cannot change the behaviour it measures. It exists because a detached task and an owned one are byte-identical on the wire and the difference only ever surfaces as a leak much later. Decremented by a Drop GUARD, not a trailing statement: a trailing statement does not run when a task is ABORTED, and abort is exactly the path pingdotgg#476 is about -- a counter blind to the abort would make the test vacuous. The test asserts the count RETURNS TO ZERO under a bounded poll, and first asserts it ROSE ABOVE ZERO so a disconnect that races the server picking the work up fails loudly instead of passing green.
1 parent cd45058 commit 39df7f7

2 files changed

Lines changed: 233 additions & 5 deletions

File tree

‎backend/src/contract_tests.rs‎

Lines changed: 184 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,7 @@ async fn state_built(
454454
.await
455455
.unwrap();
456456
let state = AppState {
457+
inflight_requests: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
457458
rt,
458459
catalog: Arc::new(RwLock::new(providers::catalog())),
459460
checkpoints: tools::checkpoint_pool(data.join("checkpoints")),
@@ -2649,20 +2650,84 @@ async fn pull_request_rpcs_return_typed_unavailable_not_die_defect() {
26492650
}
26502651
}
26512652

2653+
/// The applied-ack carries a sequence — and ONLY for a command that was applied.
2654+
///
2655+
/// REWRITTEN FOR #433/#446, and the rewrite is the point, so it is spelled out
2656+
/// rather than quietly done.
2657+
///
2658+
/// This test used to dispatch `{"type": "noop"}` — a type that appears nowhere
2659+
/// in `DispatchableClientOrchestrationCommand` — and assert
2660+
/// `exit._tag == "Success"` with a numeric sequence. It passed. It passed
2661+
/// because the dispatcher's catch-all was `_ => {}` falling through to the
2662+
/// applied-ack, which is precisely the defect #433 enumerated and #446 traced to
2663+
/// its generator. So the suite was not merely blind to those 15 silent no-ops:
2664+
/// it PINNED them. Any worker who fixed the catch-all got a red test that read
2665+
/// like a regression, which is a good way to make a real fix look wrong twice
2666+
/// and get reverted.
2667+
///
2668+
/// That is why this is a rewrite and not a deletion. The property the old name
2669+
/// promised — "dispatch acks with sequence" — is real and still worth holding.
2670+
/// What was wrong was the input chosen to demonstrate it: an unimplemented
2671+
/// command is the one case where the ack must NOT happen. So the test now
2672+
/// asserts the property on a command that is genuinely implemented, and adds
2673+
/// the boundary that the old version had inverted.
2674+
///
2675+
/// Both halves in one test on purpose: the ack and the refusal are the same
2676+
/// decision read from two sides, and splitting them invites one to be updated
2677+
/// without the other.
26522678
#[tokio::test]
2653-
async fn dispatch_acks_with_sequence() {
2679+
async fn dispatch_acks_an_implemented_command_with_a_sequence_and_refuses_an_unimplemented_one() {
26542680
let (state, _d) = test_state().await;
2681+
2682+
// HALF ONE — an IMPLEMENTED command still acks, and still carries the
2683+
// sequence. `project.meta.update` is handled at server_main.rs and defaults
2684+
// `projectId` to the seeded `p-workspace`, so it exercises the real applied
2685+
// path rather than an error path that happens to also return.
26552686
let (tx, mut rx) = mpsc::unbounded_channel();
26562687
request(
26572688
&state,
26582689
&tx,
26592690
"orchestration.dispatchCommand",
2660-
json!({ "input": { "type": "noop" } }),
2691+
json!({ "input": { "type": "project.meta.update", "patch": { "name": "renamed" } } }),
26612692
)
26622693
.await;
26632694
let frames = drain(&mut rx);
2664-
assert_eq!(frames[0]["exit"]["_tag"], "Success");
2665-
assert!(frames[0]["exit"]["value"]["sequence"].is_number());
2695+
assert_eq!(
2696+
frames[0]["exit"]["_tag"], "Success",
2697+
"an implemented command must still ack as applied: {:?}",
2698+
frames[0]
2699+
);
2700+
assert!(
2701+
frames[0]["exit"]["value"]["sequence"].is_number(),
2702+
"the applied-ack must still carry the durable sequence: {:?}",
2703+
frames[0]
2704+
);
2705+
2706+
// HALF TWO — the boundary the old test had backwards. An unimplemented
2707+
// command must be REFUSED, not acked. `noop` is the exact input the previous
2708+
// version asserted Success on.
2709+
let (tx2, mut rx2) = mpsc::unbounded_channel();
2710+
request(
2711+
&state,
2712+
&tx2,
2713+
"orchestration.dispatchCommand",
2714+
json!({ "input": { "type": "noop" } }),
2715+
)
2716+
.await;
2717+
let frames2 = drain(&mut rx2);
2718+
assert_ne!(
2719+
frames2[0]["exit"]["_tag"], "Success",
2720+
"#433/#446: an unimplemented command acked as APPLIED. The client applies \
2721+
optimistically and the ack confirms, so the user's change is shown and \
2722+
then silently gone on the next reload. Got: {:?}",
2723+
frames2[0]
2724+
);
2725+
let rendered = frames2[0].to_string();
2726+
assert!(
2727+
rendered.contains("noop"),
2728+
"the refusal must name the command type it refused, or the user cannot \
2729+
report which action did nothing: {rendered}"
2730+
);
26662731
}
26672732

26682733
/// #35: the advertised catalog is the reconciled registry, not two
@@ -9120,6 +9185,7 @@ async fn the_shell_snapshot_reads_projects_from_the_durable_store() {
91209185
Err(e) => panic!("project store unreadable: {e}"),
91219186
}
91229187
let state_b = AppState {
9188+
inflight_requests: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
91239189
rt: rt_b,
91249190
catalog: state_a.catalog.clone(),
91259191
checkpoints: tools::checkpoint_pool(data.join("checkpoints")),
@@ -11915,3 +11981,117 @@ async fn the_read_loop_answers_a_ping_while_slow_requests_are_still_running() {
1191511981

1191611982
server.abort();
1191711983
}
11984+
11985+
/// PROOF (#476): a disconnecting client takes its in-flight requests with it.
11986+
///
11987+
/// THE DEFECT THIS PINS, AND IT WAS MINE. The first fix for #432 replaced the
11988+
/// read loop's inline `dispatch_ws_frame(...).await` with a bare
11989+
/// `tokio::spawn`. That un-starves the reader — the #432 test above proves it —
11990+
/// and in the same line it detaches the work. `orchestration.subscribeThread`
11991+
/// attaches a durable tail and streams for the life of the thread, so a client
11992+
/// that disconnects mid-stream left a task writing into a `tx` whose receiver
11993+
/// was gone, holding an `AppState` clone, forever. Reconnect and there are two.
11994+
/// Serialization had been the backpressure, and removing it put nothing in its
11995+
/// place.
11996+
///
11997+
/// clau-f11b caught it and claude-4e91 had already written the better shape: a
11998+
/// `JoinSet` OWNED by `handle_socket`, so the tasks cannot outlive the
11999+
/// connection. That is what this asserts.
12000+
///
12001+
/// WHY THE #432 TEST CANNOT COVER THIS. #432 asks "does the loop keep READING"
12002+
/// — the bare spawn passes it. #476 asks "is the spawned work OWNED", and the
12003+
/// two implementations are byte-identical on the wire; the difference only ever
12004+
/// shows up as a leak, much later, in a process nobody is looking at. So this
12005+
/// asserts on `AppState::inflight_requests`, which is an OBSERVABLE and not a
12006+
/// control: nothing reads it to make a decision, so its presence cannot change
12007+
/// the behaviour it measures.
12008+
///
12009+
/// NOT A TIMER. The assertion is that the count RETURNS TO ITS PRE-CONNECT
12010+
/// VALUE, polled under a bounded deadline. A test that slept and then checked
12011+
/// would pass on a detached task that merely happened to finish.
12012+
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
12013+
async fn a_disconnecting_client_takes_its_in_flight_requests_with_it() {
12014+
use futures_util::{SinkExt, StreamExt};
12015+
12016+
let (state, _dir) = test_state().await;
12017+
let counter = state.inflight_requests.clone();
12018+
let app = super::build_app(state);
12019+
12020+
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
12021+
let addr = listener.local_addr().unwrap();
12022+
let server = tokio::spawn(async move {
12023+
let _ = axum::serve(listener, app).await;
12024+
});
12025+
12026+
assert_eq!(
12027+
counter.load(std::sync::atomic::Ordering::SeqCst),
12028+
0,
12029+
"nothing is in flight before a client connects"
12030+
);
12031+
12032+
let (mut sock, _resp) = tokio_tungstenite::connect_async(format!("ws://{addr}/ws"))
12033+
.await
12034+
.expect("WS upgrade");
12035+
12036+
// Three slow-but-real requests, so there is genuinely something in flight
12037+
// when the socket goes away. Same handler the #432 test uses (it forks `ps`)
12038+
// and for the same reason: a test-only sleep method would prove the backdoor.
12039+
for i in 0..3 {
12040+
sock.send(tokio_tungstenite::tungstenite::Message::Text(
12041+
json!({
12042+
"_tag": "Request",
12043+
"id": format!("doomed-{i}"),
12044+
"tag": "server.getProcessDiagnostics",
12045+
"payload": {},
12046+
})
12047+
.to_string()
12048+
.into(),
12049+
))
12050+
.await
12051+
.expect("send");
12052+
}
12053+
12054+
// Wait until the server has actually PICKED THEM UP, so the drop below is a
12055+
// disconnect-with-work-in-flight and not a race the test wins by accident.
12056+
// If this never rises, the test is not exercising #476 at all and says so
12057+
// rather than passing vacuously.
12058+
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(30);
12059+
let mut peak = 0usize;
12060+
while tokio::time::Instant::now() < deadline {
12061+
peak = peak.max(counter.load(std::sync::atomic::Ordering::SeqCst));
12062+
if peak > 0 {
12063+
break;
12064+
}
12065+
tokio::task::yield_now().await;
12066+
}
12067+
assert!(
12068+
peak > 0,
12069+
"#476: no request task was ever observed in flight, so this test never \
12070+
reached the condition it exists to check. Either the spawn is gone or \
12071+
the counter is not wired — both make the assertion below vacuous."
12072+
);
12073+
12074+
// THE DISCONNECT.
12075+
drop(sock);
12076+
12077+
// THE ASSERTION: in-flight returns to zero because the tasks were OWNED, not
12078+
// because they happened to finish. Bounded poll, no sleep-then-check.
12079+
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(30);
12080+
let mut last = usize::MAX;
12081+
while tokio::time::Instant::now() < deadline {
12082+
last = counter.load(std::sync::atomic::Ordering::SeqCst);
12083+
if last == 0 {
12084+
break;
12085+
}
12086+
tokio::task::yield_now().await;
12087+
}
12088+
assert_eq!(
12089+
last, 0,
12090+
"#476: {last} request task(s) outlived the socket that created them. A \
12091+
bare `tokio::spawn` detaches, so subscribeThread keeps streaming into a \
12092+
`tx` whose receiver is gone and holds an AppState clone per dead client. \
12093+
The JoinSet owned by handle_socket drops them with the connection."
12094+
);
12095+
12096+
server.abort();
12097+
}

‎backend/src/server_main.rs‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,15 @@ struct AppState {
196196
/// transcripts, so the test either asserts nothing or asserts whatever
197197
/// happens to be on the machine (#328).
198198
usage_sources: Arc<Vec<agent_sdk_usage::SourceSpec>>,
199+
/// How many spawned `Request` tasks are alive right now (#476).
200+
///
201+
/// An OBSERVABLE, not a control: nothing reads it to make a decision, so it
202+
/// cannot change behaviour. It exists because "the request task died with
203+
/// its socket" is otherwise unfalsifiable — a detached task and an owned one
204+
/// are byte-identical on the wire, and the difference only appears much
205+
/// later as a leak. Decremented by a Drop guard so an ABORT still counts,
206+
/// which is the whole case under test.
207+
inflight_requests: Arc<std::sync::atomic::AtomicUsize>,
199208
env: Value,
200209
cwd: String,
201210
project_name: String,
@@ -510,6 +519,7 @@ async fn main() {
510519
.await
511520
.expect("asset signing key");
512521
let state = AppState {
522+
inflight_requests: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
513523
rt,
514524
catalog,
515525
checkpoints,
@@ -1683,7 +1693,26 @@ async fn handle_socket(socket: WebSocket, state: AppState) {
16831693
}
16841694
});
16851695

1696+
// #476: in-flight `Request` tasks, OWNED BY THIS SOCKET.
1697+
//
1698+
// A bare `tokio::spawn` fixes #432's starvation and opens a leak on the same
1699+
// line: the task is DETACHED, so a client that disconnects mid-stream leaves
1700+
// `orchestration.subscribeThread` attached to a durable tail, writing into a
1701+
// `tx` whose receiver is gone and holding an `AppState` clone forever.
1702+
// Reconnect and there are two. Serialization was the backpressure; removing
1703+
// it without putting ownership in its place trades a wedge for an unbounded
1704+
// leak on the exact path #27 is about.
1705+
//
1706+
// A `JoinSet` makes the lifetime structural instead of hoping the task ends:
1707+
// it is owned by this function, so it cannot outlive the connection. The
1708+
// design is claude-4e91's and it is better than the bare spawn I wrote.
1709+
let mut inflight: tokio::task::JoinSet<()> = tokio::task::JoinSet::new();
1710+
16861711
while let Some(Ok(msg)) = stream.next().await {
1712+
// Reap finished tasks. A `JoinSet` keeps a completed task's slot until it
1713+
// is joined, so without this a long-lived socket grows one slot per
1714+
// request for its whole life — slower than the detached leak, still a leak.
1715+
while inflight.try_join_next().is_some() {}
16871716
match msg {
16881717
Message::Text(text) => {
16891718
let frame: Value = match serde_json::from_str(&text) {
@@ -1748,7 +1777,21 @@ async fn handle_socket(socket: WebSocket, state: AppState) {
17481777
if frame.get("_tag").and_then(|t| t.as_str()) == Some("Request") {
17491778
let tx = tx.clone();
17501779
let state = state.clone();
1751-
tokio::spawn(async move {
1780+
inflight.spawn(async move {
1781+
// The counter is decremented by a GUARD, not by a
1782+
// statement at the end of the block: a trailing statement
1783+
// does not run when the task is ABORTED, and abort is
1784+
// exactly the path #476 exists to observe.
1785+
struct Inflight(Arc<std::sync::atomic::AtomicUsize>);
1786+
impl Drop for Inflight {
1787+
fn drop(&mut self) {
1788+
self.0.fetch_sub(1, std::sync::atomic::Ordering::SeqCst);
1789+
}
1790+
}
1791+
state
1792+
.inflight_requests
1793+
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
1794+
let _guard = Inflight(state.inflight_requests.clone());
17521795
dispatch_ws_frame(frame, &tx, &state).await;
17531796
});
17541797
} else {
@@ -1766,6 +1809,11 @@ async fn handle_socket(socket: WebSocket, state: AppState) {
17661809
}
17671810
}
17681811
tracing::info!("ws: client closed");
1812+
// #476: take the in-flight requests down WITH the socket. Dropping the set
1813+
// aborts them anyway, but awaiting the shutdown means they are actually gone
1814+
// before this function returns, so the invariant is observable rather than
1815+
// merely eventual.
1816+
inflight.shutdown().await;
17691817
writer.abort();
17701818
}
17711819

0 commit comments

Comments
 (0)