Summary

A connection that is already Drained and then processes another stateless-reset datagram pushes a second EndpointEventInner::Draining event. The endpoint decrements active_connections once per Draining event with no per-connection deduplication, so the counter is decremented twice for a single increment. Once the accumulated imbalance exceeds the number of currently active connections, the endpoint driver hits:

thread 'tokio-rt-worker' panicked at noq-1.0.1/src/endpoint.rs:648:17:
attempt to subtract with overflow

With default release settings (no overflow checks) the counter instead wraps, after which active_connections == 0 never holds again: wait_all_draining() hangs forever and all_draining waiters are never notified.

This is distinct from #717 (increment asymmetry on accept-after-close), which is already in the 1.0.1 release; the panic above is observed on 1.0.1. Affects released noq/noq-proto 1.0.1 and current main (54bae7f).

Mechanism

1. A connection is established: ConnectionSet::insert does active_connections += 1 (noq/src/endpoint.rs:779).

2. The peer endpoint loses its connection state (e.g. process restart). Per RFC 9000 §10.3 it answers each subsequent packet with a stateless reset, so several reset datagrams for the same connection can be in flight at once — one per packet sent before the sender notices.

3. The reset datagrams are routed to the connection and queued on its event channel. The connection task's process_conn_events loop (noq/src/connection.rs:1556) processes every queued event in a single poll, with no early exit once the connection reaches a terminal state. Nothing else discards datagrams for a drained connection either: early_discard_packet (noq-proto/src/connection/mod.rs) checks paths and handshake state, but not connection state.

4. First reset: handle_packet detects the reset and produces Err(ConnectionError::Reset) (noq-proto/src/connection/mod.rs:4271). The error-transition block runs:

   ConnectionError::Reset
   | ConnectionError::TransportError(TransportError { code: TransportErrorCode::AEAD_LIMIT_REACHED, .. }) => {
       let was_draining = self.state.move_to_drained(Some(conn_err));
       if !was_draining {
           self.endpoint_events.push_back(EndpointEventInner::Draining);
       }
   }

   State goes Established → Drained, was_draining == false, one Draining event is pushed. Correct so far.

5. Second reset (same poll): reaches the same arm. Two things go wrong together:

   • State::move_to_drained(Some(_)) (noq-proto/src/connection/state.rs:71) does not reject a Drained → Drained transition when an error is supplied — only the error: None branch has the panic!("invalid state transition drained -> drained") guard.
   • Its return value is matches!(self.inner, InnerState::Draining { .. }), which is false for Drained. So was_draining == false and a duplicate Draining event is pushed.

6. The endpoint decrements once per Draining event (noq/src/endpoint.rs:646-651), with no tracking of whether that connection already reported draining → net −2 against a single +1.

Note the asymmetry inside handle_packet itself: the function already captures let was_drained = self.state.is_drained(); at entry (mod.rs:4259 on main) and uses it to make the Drained event edge-triggered:

if !was_drained && self.state.is_drained() {
    self.endpoint_events.push_back(EndpointEventInner::Drained);
    ...
}

so a repeated reset does not duplicate Drained — only Draining lacks the same edge condition.

The AEAD_LIMIT_REACHED error shares the same arm and has the same effect.

Trigger conditions

Any situation where a connection receives more than one terminal-error datagram, most commonly two or more stateless resets arriving close together (the batch-processing loop in process_conn_events makes same-poll processing routine). A restarting peer under active traffic produces exactly this pattern. Observed repeatedly in production on 1.0.1 during ordinary connection teardown when a remote endpoint had recently restarted.

Proposed fix

Make the Draining push edge-triggered on the terminal state, consistent with how the same function already handles Drained — using the was_drained local it already captures:

let was_draining = self.state.move_to_drained(Some(conn_err));
if !was_draining && !was_drained {
    self.endpoint_events.push_back(EndpointEventInner::Draining);
}

Complementary hardening, if wanted:

• Discard incoming datagrams for connections already in Drained (e.g. in early_discard_packet); a drained connection has nothing left to do with them.
• Have move_to_drained(Some(_)) treat Drained as terminal (return "was already terminal" or debug-assert), so future callers can't reintroduce the double event.
• A debug_assert!(self.recv_state.connections.active_connections > 0) before the decrement in noq/src/endpoint.rs would have pointed straight at the event bug. A plain saturating_sub alone would be counterproductive: it converts the loud panic into the silent wait_all_draining() hang described above unless the event accounting is fixed first.

Regression test sketch

Proto-level, in the style of the #732 test: establish a pair, then deliver two stateless-reset datagrams for the same connection via handle_event back-to-back. Assert poll_endpoint_events yields exactly one Draining (and one Drained).

Related

• fix(proto): Avoid active_connections underflow #717 — fixed the increment-side asymmetry; included in 1.0.1; this report reproduces on 1.0.1, so it is a separate decrement-side bug.
• EndpointDriver::drop panics on a poisoned mutex, escalating any accept-loop panic to a process abort #723 — when this panic fires inside the endpoint driver, the poisoned state mutex escalates it exactly as described there (subsequent lock().unwrap() panics, then abort in EndpointDriver::drop), turning one accounting bug into a process kill.