Skip to content

fix(mesh): owner-control listener must not share the mesh relay slot - #1064

Closed
michaelneale wants to merge 2 commits into
mainfrom
micn/fix-owner-control-relay-collision
Closed

michaelneale wants to merge 2 commits into
mainfrom
micn/fix-owner-control-relay-collision

Conversation

@michaelneale

Copy link
Copy Markdown
Collaborator

Problem

A node with a verified owner identity starts a second iroh endpoint — the
owner-control listener (ALPN mesh-llm-control/1) — using the same secret key
as the main mesh endpoint, so both present the same iroh endpoint id. The
control protocol requires the shared id: it validates the dialed target_node_id
against the main endpoint id (verify_control_plane_target_node).

An iroh relay keeps only one active connection per endpoint id
(iroh-relay server/clients.rs: a second same-id registration does
std::mem::replace(&mut state.active, client) and the old one "can still send
messages, but won't receive anything"). The control listener enables relay and
binds after the main mesh endpoint, so it evicts the main endpoint's relay
registration
— logged as:

Relay server reports problem: Another endpoint connected with the same endpoint id.
No more messages will be received.

The main mesh endpoint then goes dark on the relay. All relay-delivered mesh
traffic (gossip, joins, inference routing) stops. Any peer that cannot reach the
node directly never completes its join, never syncs the model catalog, and
every inference returns 429/503.

Impact

Consumers on a network where the direct path does not establish (common: NAT /
LAN-hairpin) are left with no relay fallback, because the fallback path was
silently destroyed. Consume appears to "connect but never join".

Fix

Bind the owner-control listener with RelayMode::Disabled so it never contends
for the shared id's relay slot. The main mesh endpoint is unchanged and keeps
relay + NAT traversal; it becomes the sole relay registrant for the id. Also
clear_ip_transports() on the control endpoint so an explicit IPv4 bind does not
leave iroh's implicit [::]:0 socket (which can trip MultipathNotNegotiated
with relay disabled) — mirroring the main endpoint's LAN-only handling.

Removed the now-unused configure_control_relay helper, its relay parameter,
and the control-endpoint relay online() wait.

Trade-off

Owner-control becomes reachable over its direct / advertised address only;
relay-assisted remote owner-control across NAT is unsupported while control and
mesh share one id. Giving control its own relay presence would require a distinct
endpoint id, which the control target validation forbids (would be a
protocol-level change). Buzz — the downstream that hit this — never dials
owner-control, so this trade-off is invisible there.

Verification

  • New regression test control_plane_listener_token_carries_no_relay_urls
    asserts the control token advertises zero relay URLs while keeping the main
    endpoint id. All 6 control_plane_listener tests pass.
  • Live A/B, two Macs, owner identity present both sides:
    • Before: consumer logs the "same endpoint id" eviction, never joins.
    • After: consumer joins over the relay and inference returns HTTP 200.

Notes

  • Also available as a minimal backport onto the v0.73.1 release layout
    (branch micn/fix-owner-control-relay-collision-v0.73.1) for downstreams
    pinning that tag.
  • A separate, pre-existing direct-hole-punch reliability problem (native
    mesh's direct LAN path is flaky where raw iroh's is not) is filed separately as
    a blocker issue — it is not caused by this change (reproduces on unpatched
    v0.73.1 too) and is orthogonal to this relay-slot fix.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: eea466cc-c4fb-45fb-ba4d-77ebf7e56148

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch micn/fix-owner-control-relay-collision

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

@michaelneale

Copy link
Copy Markdown
Collaborator Author

Related blocker (orthogonal, pre-existing): #1065 — native mesh direct LAN hole-punch times out where raw iroh succeeds. This PR restores relay fallback; #1065 tracks getting the fast direct path reliable.

michaelneale added a commit to block/buzz that referenced this pull request Jul 23, 2026
Repins the mesh-llm-* git deps from tag v0.73.1 to
rev c441ea7f328692b18b7f49ad819c7b1a603cbdcb, the v0.73.1-based backport of the
owner-control / main-mesh relay endpoint-id collision fix
(Mesh-LLM/mesh-llm#1064).

Without the fix, a consumer with an owner identity (buzz always sets one for
admission) starts a second iroh endpoint sharing the node's endpoint id; that
endpoint evicts the main mesh endpoint's relay registration, silently killing
relay fallback. Any consumer that cannot reach the host directly then never
joins and every inference 429/503s. The rev still reports version 0.73.1, so the
native runtime download resolves to the existing v0.73.1 release unchanged.

Hand-verified in the release app on the meshllm community: mini consumed M5's
shared gemma with owner identity present on both sides (the exact pre-fix
failure condition), inference returned 200.

NOTE: interim pin to an unmerged rev; retarget to a v0.73.x tag once #1064 merges.
@michaelneale

Copy link
Copy Markdown
Collaborator Author

this may not be needed - will be superseeded by other work. DO NOT MERGE

A node with a verified owner identity started a second iroh endpoint (the
owner-control listener, ALPN mesh-llm-control/1) using the SAME secret key as
the main mesh endpoint, so both endpoints presented the SAME iroh endpoint id.
The control protocol requires this: it validates the dialed target_node_id
against the main endpoint id (verify_control_plane_target_node).

But an iroh relay keeps only ONE active connection per endpoint id. The control
listener enabled relay and bound AFTER the main mesh endpoint, so it evicted the
main endpoint's relay registration (relay: 'Another endpoint connected with the
same endpoint id. No more messages will be received.'). The main mesh endpoint
then went dark on the relay: all relay-delivered mesh traffic (gossip, joins,
inference routing) stopped, so any peer that could not reach the node directly
never completed its join, never synced the model catalog, and every inference
returned 429/503.

Fix: bind the owner-control listener with RelayMode::Disabled so it never
contends for the shared id's relay slot, leaving the main mesh endpoint as the
sole relay registrant. The main mesh endpoint is unchanged and keeps relay/NAT
traversal. Also clear_ip_transports() on the control endpoint so an explicit
IPv4 bind does not leave iroh's implicit [::]:0 socket (which can trip
MultipathNotNegotiated with relay disabled), matching the main endpoint's
LAN-only handling.

Trade-off: owner-control is now reachable over its direct/advertised address
only; relay-assisted remote owner-control across NAT is not supported while the
control and mesh endpoints share one id.

Regression test: control_plane_listener_token_carries_no_relay_urls.
…ack IP)

Defence-in-depth invariant for the option-c relay-slot fix (#1064): the
owner-control token must advertise ZERO relay URLs AND only loopback IPs by
default, keeping the control plane non-routable off-box (preserving the
split-control separation intent of #539) unless an explicit advertise addr is
configured. Complements the existing no-relay-URLs assertion.

Assisted-by: goose
@michaelneale
michaelneale force-pushed the micn/fix-owner-control-relay-collision branch from f66a323 to 4692d7b Compare July 24, 2026 03:43
@michaelneale

Copy link
Copy Markdown
Collaborator Author

may actually be building on this - I think this is necessary

@michaelneale

Copy link
Copy Markdown
Collaborator Author

Superseded by #1071, which contains these two commits plus the iroh 1.0.3 modernization. Consolidating to one PR.

michaelneale added a commit that referenced this pull request Jul 25, 2026
…ol relay fix (#1071)

* fix(mesh): owner-control listener must not share the mesh relay slot

A node with a verified owner identity started a second iroh endpoint (the
owner-control listener, ALPN mesh-llm-control/1) using the SAME secret key as
the main mesh endpoint, so both endpoints presented the SAME iroh endpoint id.
The control protocol requires this: it validates the dialed target_node_id
against the main endpoint id (verify_control_plane_target_node).

But an iroh relay keeps only ONE active connection per endpoint id. The control
listener enabled relay and bound AFTER the main mesh endpoint, so it evicted the
main endpoint's relay registration (relay: 'Another endpoint connected with the
same endpoint id. No more messages will be received.'). The main mesh endpoint
then went dark on the relay: all relay-delivered mesh traffic (gossip, joins,
inference routing) stopped, so any peer that could not reach the node directly
never completed its join, never synced the model catalog, and every inference
returned 429/503.

Fix: bind the owner-control listener with RelayMode::Disabled so it never
contends for the shared id's relay slot, leaving the main mesh endpoint as the
sole relay registrant. The main mesh endpoint is unchanged and keeps relay/NAT
traversal. Also clear_ip_transports() on the control endpoint so an explicit
IPv4 bind does not leave iroh's implicit [::]:0 socket (which can trip
MultipathNotNegotiated with relay disabled), matching the main endpoint's
LAN-only handling.

Trade-off: owner-control is now reachable over its direct/advertised address
only; relay-assisted remote owner-control across NAT is not supported while the
control and mesh endpoints share one id.

Regression test: control_plane_listener_token_carries_no_relay_urls.

* test(mesh): assert owner-control token is loopback-only (no non-loopback IP)

Defence-in-depth invariant for the option-c relay-slot fix (#1064): the
owner-control token must advertise ZERO relay URLs AND only loopback IPs by
default, keeping the control plane non-routable off-box (preserving the
split-control separation intent of #539) unless an explicit advertise addr is
configured. Complements the existing no-relay-URLs assertion.

Assisted-by: goose

* chore(deps): upgrade iroh 1.0.1 -> 1.0.3

Bundled with the owner-control relay-slot fix so there is a single proven
change. Workspace declares iroh = "1.0.0" (caret), so this is a Cargo.lock
bump only. 1.0.2 = transport lanes fairness fix; 1.0.3 = ALPN connection-error
handling + relay/DNS improvements. Compiles clean, no API changes.

Assisted-by: goose

* refactor(mesh): adopt modern iroh defaults for direct paths

Remove home-grown networking workarounds that predate mature iroh NAT
traversal (added at iroh 1.0.0-rc.0):
- drop startup_transport_config keep-alive/idle overrides (iroh clamps
  per-path idle to 15s and defaults 5s keep-alive; the 300s override was
  silently ignored)
- default effective_quic_bind_ip to None (wildcard sockets) instead of
  pinning the detected LAN IPv4 (which created a 2nd UDP socket)
- remove start_direct_path_maintenance reverse-dial loop (iroh QNT does
  relay->direct migration itself, incl. hard-NAT since rc.1)

Rationale + cross-NAT proof that bare iroh 1.0.3 hole-punches without
these: mesh-llm issue #1065. NOTE: modernized mesh CLIENT still rides
relay in live mini<->M5 testing; bisection shows the serve side punches
direct to a bare-iroh dialer, so the residual fault is client-side and
not yet root-caused.

* refactor(mesh): retire legacy direct-path sender

* test(api): restore HOME after control-plane tests

---------

Co-authored-by: Michael Neale <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants