Thank you for the PR. Can you say a little more about what motivated you to fix this? Were you hitting this bug in practice? If so, what was your setup like? Knowing these things might help us understand how to test noq better.

IIUC there is a compounding issue that Connection::discard_path is not telling the endpoint to remove those local CIDs so that it stops routing them to the connection. #322 is about this.

Instead it should send a stateless reset to the peer upon receiving this packet, which would kill the peer. This is why the above question of how this occurred is really important: because firstly this shouldn't normally happen and secondly it would kill the connection on the remote side.

However even if we did do that, because revoking the CIDs at the endpoint is also an asynchronous event this bug could still be triggered. So this fix is still needed.

Hi @flub ,

Thanks for looking into this - I saw the following panic on one of my servers running https://github.com/losfair/iroh-http-gateway. Only happened once so far - unfortunately log level was unset, so no trace-level logs:

Jun 29 13:21:35 <redacted> iroh-http-gateway[3027151]: {"timestamp":"2026-06-29T05:21:35.709315Z","level":"WARN","fields":{"message":"HTTP connection failed","remote_addr":"127.0.0.1:45184","error":"connection closed before message completed"},"target":"iroh_http_gateway"}
Jun 29 13:21:36 <redacted> iroh-http-gateway[3027151]: {"timestamp":"2026-06-29T05:21:36.385583Z","level":"WARN","fields":{"message":"HTTP connection failed","remote_addr":"127.0.0.1:45192","error":"connection closed before message completed"},"target":"iroh_http_gateway"}
Jun 29 13:21:36 <redacted> iroh-http-gateway[3027151]: {"timestamp":"2026-06-29T05:21:36.909711Z","level":"WARN","fields":{"message":"HTTP connection failed","remote_addr":"127.0.0.1:45206","error":"connection closed before message completed"},"target":"iroh_http_gateway"}
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: thread 'tokio-rt-worker' (3027152) panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/noq-proto-1.0.0/src/connection/mod.rs:4148:14:
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: known path
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: thread 'tokio-rt-worker' (3027152) panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/noq-1.0.0/src/mutex.rs:138:42:
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: called `Result::unwrap()` on an `Err` value: PoisonError { .. }
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: stack backtrace:
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]:    0:          0x1ebe9fc - <unknown>
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]:    1:          0x1ed7408 - <unknown>
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]:    2:          0x1ec53a8 - <unknown>
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]:    3:          0x1ea5eb4 - <unknown>
...
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: thread 'tokio-rt-worker' (3027152) panicked at /rustc/ac68faa20c58cbccd01ee7208bf3b6e93a7d7f96/library/core/src/panicking.rs:233:5:
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: panic in a destructor during cleanup
Jun 29 13:21:37 <redacted> iroh-http-gateway[3027151]: thread caused non-unwinding panic. aborting.

@losfair can you reproduce this? because like this there isn't enough information, so please configure logging and run with RUST_LOG=trace and than we should be able to get more details about what triggered this as it will show the exact packets that were being sent and received.

@flub I added a test that reproduces the issue properly and fixed the patch. Running with trace level logs for ~10 hrs and haven't seen a reproduction yet though...

So somehow the first datagram from the server is duplicated on the network and is then received again after the connection has closed PathId::ZERO? That does seem like something that could occur.

But I still really want to understand why this happens. Can you share logs of a server and client where this happens (before the fix)? I have some hunches but I need logs to check this. Duplicated packets are probably created by the server-side. But normally it would deduplicate them. I am guessing that the initial EndpointAddr contains both a RelayUrl and an IP that functions, how big is the latency difference? I assume that the IP connection wins and that for some reason the connection that very quickly migrates to a different IP address, closing PathId::ZERO, again, why? And possibly this happens faster than the relay connection has delivered the first datagram from the client to the relay. At which point the relay connection may see this as a genuinely new connection attempt. But I'm just guessing. Please give us some evidence.

Please fix clippy as well, pref tests won't work so that's fine.

Can you share logs of a server and client where this happens (before the fix)?

Still waiting for this to happen again so that I have trace level logs...

I am guessing that the initial EndpointAddr contains both a RelayUrl and an IP that functions, how big is the latency difference?

I'm connecting with endpoint id only - so I guess it has to rely on relay initially before direct path discovery works?

I have some custom routing setup on the client machine - something like:

chain output_route_1000 {
        meta protocol ip6 counter reject
        ip daddr { 100.64.0.0/10 } counter return
        counter meta mark set 0x00000020 # policy routing - routed via a node ~200ms away
}

chain output_route {
        type route hook output priority filter; policy accept;
        fib daddr type local counter accept
        meta skuid 1000 jump output_route_1000
}

So client <-> server are reachable via three different paths:

• Direct: Tailscale 100.64.0.0/10 network - ~15ms
• Direct: Public IPv4 - ~400ms
• Relay - 300-400ms

They might ask you to name the test regression_ tho
a comment that this used to panic inside the test would help too

Time to finally share my spicy take on this: distinguishing regression tests from other tests is... fun but not helpful in any way.