Repository navigation
fix(mesh): owner-control listener must not share the mesh relay slot - #1064
michaelneale wants to merge 2 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
Repins the mesh-llm-* git deps from tag v0.73.1 to rev c441ea7f328692b18b7f49ad819c7b1a603cbdcb, the v0.73.1-based backport of the owner-control / main-mesh relay endpoint-id collision fix (Mesh-LLM/mesh-llm#1064). Without the fix, a consumer with an owner identity (buzz always sets one for admission) starts a second iroh endpoint sharing the node's endpoint id; that endpoint evicts the main mesh endpoint's relay registration, silently killing relay fallback. Any consumer that cannot reach the host directly then never joins and every inference 429/503s. The rev still reports version 0.73.1, so the native runtime download resolves to the existing v0.73.1 release unchanged. Hand-verified in the release app on the meshllm community: mini consumed M5's shared gemma with owner identity present on both sides (the exact pre-fix failure condition), inference returned 200. NOTE: interim pin to an unmerged rev; retarget to a v0.73.x tag once #1064 merges.
|
this may not be needed - will be superseeded by other work. DO NOT MERGE |
A node with a verified owner identity started a second iroh endpoint (the owner-control listener, ALPN mesh-llm-control/1) using the SAME secret key as the main mesh endpoint, so both endpoints presented the SAME iroh endpoint id. The control protocol requires this: it validates the dialed target_node_id against the main endpoint id (verify_control_plane_target_node). But an iroh relay keeps only ONE active connection per endpoint id. The control listener enabled relay and bound AFTER the main mesh endpoint, so it evicted the main endpoint's relay registration (relay: 'Another endpoint connected with the same endpoint id. No more messages will be received.'). The main mesh endpoint then went dark on the relay: all relay-delivered mesh traffic (gossip, joins, inference routing) stopped, so any peer that could not reach the node directly never completed its join, never synced the model catalog, and every inference returned 429/503. Fix: bind the owner-control listener with RelayMode::Disabled so it never contends for the shared id's relay slot, leaving the main mesh endpoint as the sole relay registrant. The main mesh endpoint is unchanged and keeps relay/NAT traversal. Also clear_ip_transports() on the control endpoint so an explicit IPv4 bind does not leave iroh's implicit [::]:0 socket (which can trip MultipathNotNegotiated with relay disabled), matching the main endpoint's LAN-only handling. Trade-off: owner-control is now reachable over its direct/advertised address only; relay-assisted remote owner-control across NAT is not supported while the control and mesh endpoints share one id. Regression test: control_plane_listener_token_carries_no_relay_urls.
…ack IP) Defence-in-depth invariant for the option-c relay-slot fix (#1064): the owner-control token must advertise ZERO relay URLs AND only loopback IPs by default, keeping the control plane non-routable off-box (preserving the split-control separation intent of #539) unless an explicit advertise addr is configured. Complements the existing no-relay-URLs assertion. Assisted-by: goose
f66a323 to
4692d7b
Compare
|
may actually be building on this - I think this is necessary |
|
Superseded by #1071, which contains these two commits plus the iroh 1.0.3 modernization. Consolidating to one PR. |
…ol relay fix (#1071) * fix(mesh): owner-control listener must not share the mesh relay slot A node with a verified owner identity started a second iroh endpoint (the owner-control listener, ALPN mesh-llm-control/1) using the SAME secret key as the main mesh endpoint, so both endpoints presented the SAME iroh endpoint id. The control protocol requires this: it validates the dialed target_node_id against the main endpoint id (verify_control_plane_target_node). But an iroh relay keeps only ONE active connection per endpoint id. The control listener enabled relay and bound AFTER the main mesh endpoint, so it evicted the main endpoint's relay registration (relay: 'Another endpoint connected with the same endpoint id. No more messages will be received.'). The main mesh endpoint then went dark on the relay: all relay-delivered mesh traffic (gossip, joins, inference routing) stopped, so any peer that could not reach the node directly never completed its join, never synced the model catalog, and every inference returned 429/503. Fix: bind the owner-control listener with RelayMode::Disabled so it never contends for the shared id's relay slot, leaving the main mesh endpoint as the sole relay registrant. The main mesh endpoint is unchanged and keeps relay/NAT traversal. Also clear_ip_transports() on the control endpoint so an explicit IPv4 bind does not leave iroh's implicit [::]:0 socket (which can trip MultipathNotNegotiated with relay disabled), matching the main endpoint's LAN-only handling. Trade-off: owner-control is now reachable over its direct/advertised address only; relay-assisted remote owner-control across NAT is not supported while the control and mesh endpoints share one id. Regression test: control_plane_listener_token_carries_no_relay_urls. * test(mesh): assert owner-control token is loopback-only (no non-loopback IP) Defence-in-depth invariant for the option-c relay-slot fix (#1064): the owner-control token must advertise ZERO relay URLs AND only loopback IPs by default, keeping the control plane non-routable off-box (preserving the split-control separation intent of #539) unless an explicit advertise addr is configured. Complements the existing no-relay-URLs assertion. Assisted-by: goose * chore(deps): upgrade iroh 1.0.1 -> 1.0.3 Bundled with the owner-control relay-slot fix so there is a single proven change. Workspace declares iroh = "1.0.0" (caret), so this is a Cargo.lock bump only. 1.0.2 = transport lanes fairness fix; 1.0.3 = ALPN connection-error handling + relay/DNS improvements. Compiles clean, no API changes. Assisted-by: goose * refactor(mesh): adopt modern iroh defaults for direct paths Remove home-grown networking workarounds that predate mature iroh NAT traversal (added at iroh 1.0.0-rc.0): - drop startup_transport_config keep-alive/idle overrides (iroh clamps per-path idle to 15s and defaults 5s keep-alive; the 300s override was silently ignored) - default effective_quic_bind_ip to None (wildcard sockets) instead of pinning the detected LAN IPv4 (which created a 2nd UDP socket) - remove start_direct_path_maintenance reverse-dial loop (iroh QNT does relay->direct migration itself, incl. hard-NAT since rc.1) Rationale + cross-NAT proof that bare iroh 1.0.3 hole-punches without these: mesh-llm issue #1065. NOTE: modernized mesh CLIENT still rides relay in live mini<->M5 testing; bisection shows the serve side punches direct to a bare-iroh dialer, so the residual fault is client-side and not yet root-caused. * refactor(mesh): retire legacy direct-path sender * test(api): restore HOME after control-plane tests --------- Co-authored-by: Michael Neale <[email protected]>
Problem
A node with a verified owner identity starts a second iroh endpoint — the
owner-control listener (
ALPN mesh-llm-control/1) — using the same secret keyas the main mesh endpoint, so both present the same iroh endpoint id. The
control protocol requires the shared id: it validates the dialed
target_node_idagainst the main endpoint id (
verify_control_plane_target_node).An iroh relay keeps only one active connection per endpoint id
(
iroh-relayserver/clients.rs: a second same-id registration doesstd::mem::replace(&mut state.active, client)and the old one "can still sendmessages, but won't receive anything"). The control listener enables relay and
binds after the main mesh endpoint, so it evicts the main endpoint's relay
registration — logged as:
The main mesh endpoint then goes dark on the relay. All relay-delivered mesh
traffic (gossip, joins, inference routing) stops. Any peer that cannot reach the
node directly never completes its join, never syncs the model catalog, and
every inference returns 429/503.
Impact
Consumers on a network where the direct path does not establish (common: NAT /
LAN-hairpin) are left with no relay fallback, because the fallback path was
silently destroyed. Consume appears to "connect but never join".
Fix
Bind the owner-control listener with
RelayMode::Disabledso it never contendsfor the shared id's relay slot. The main mesh endpoint is unchanged and keeps
relay + NAT traversal; it becomes the sole relay registrant for the id. Also
clear_ip_transports()on the control endpoint so an explicit IPv4 bind does notleave iroh's implicit
[::]:0socket (which can tripMultipathNotNegotiatedwith relay disabled) — mirroring the main endpoint's LAN-only handling.
Removed the now-unused
configure_control_relayhelper, itsrelayparameter,and the control-endpoint relay
online()wait.Trade-off
Owner-control becomes reachable over its direct / advertised address only;
relay-assisted remote owner-control across NAT is unsupported while control and
mesh share one id. Giving control its own relay presence would require a distinct
endpoint id, which the control target validation forbids (would be a
protocol-level change). Buzz — the downstream that hit this — never dials
owner-control, so this trade-off is invisible there.
Verification
control_plane_listener_token_carries_no_relay_urlsasserts the control token advertises zero relay URLs while keeping the main
endpoint id. All 6
control_plane_listenertests pass.Notes
v0.73.1release layout(branch
micn/fix-owner-control-relay-collision-v0.73.1) for downstreamspinning that tag.
mesh's direct LAN path is flaky where raw iroh's is not) is filed separately as
a blocker issue — it is not caused by this change (reproduces on unpatched
v0.73.1too) and is orthogonal to this relay-slot fix.