Skip to content

db: api: service: introduce system.client_routes table and related API endpoints#27323

Merged
scylladb-promoter merged 12 commits intoscylladb:masterfrom
andrzej-jackowski-scylladb:private-link-master
Dec 16, 2025
Merged

db: api: service: introduce system.client_routes table and related API endpoints#27323
scylladb-promoter merged 12 commits intoscylladb:masterfrom
andrzej-jackowski-scylladb:private-link-master

Conversation

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor

@andrzej-jackowski-scylladb andrzej-jackowski-scylladb commented Nov 27, 2025

system.client_routes is a system table that sets the target address and ports for each host_id, for one or more connection (e.g., Private Link) represented by connection_id. Cloud will write the table via REST, and drivers will read it via CQL to override values obtained from system.local and system.peers.

This patch series contains:

  • Introduction of CLIENT_ROUTES feature flag.
  • Implementation of raft-based system.client_routes table
  • Implementation of v2/client-routes POST/DELETE/GET endpoints
  • Implementation of new CLIENT_ROUTES_CHANGE event that is sent to drivers when system.client_routes is changed
  • New tests that verifies the aforementioned features

Ref: scylladb/scylla-enterprise#5699

For now, no automatic backport. However, the changes are planned to be release on 2025.4 either as a backport or a private build.

@andrzej-jackowski-scylladb andrzej-jackowski-scylladb added the backport/none Backport is not required label Nov 27, 2025
@andrzej-jackowski-scylladb andrzej-jackowski-scylladb mentioned this pull request Nov 27, 2025
Closed
@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Nov 27, 2025

v2:

  • Added missing @skip_mode('release', 'error injections are not supported in release mode') decorator to a test with injected errors

@scylladbbot
Copy link
Copy Markdown

scylladbbot commented Nov 28, 2025

🟢 CI State: SUCCESS

Mode Stage Status Node
Framework test ✅ spider9.cloudius-systems.com
dev Build ✅ spider17.cloudius-systems.com
test.py ✅ spider17.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider17.cloudius-systems.com
release Build ✅ spider3.cloudius-systems.com
test.py ✅ spider3.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider3.cloudius-systems.com
debug Build ✅ spider18.cloudius-systems.com
test.py ✅ spider18.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider18.cloudius-systems.com
dtest dtest with tablets ✅
dtest with gossip topology changes ✅
dtest with consistent topology changes ✅

Build Details:

  • Duration: 3 hr 28 min

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Nov 28, 2025

v3:

  • Removed rack and datacenter columns and related JSON fields
  • Changed GET endpoint to use CL=ONE instead of QUORUM
  • Fixed handling null terminators in JSON strings
  • Added test_client_routes_lost_quorum and test_client_routes_null_terminators

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Nov 28, 2025

@patjed41 please review the changes (most of the changes other than adding endpoint and tests are related to raft)

@scylladbbot
Copy link
Copy Markdown

scylladbbot commented Nov 29, 2025

🔴 CI State: FAILURE

Mode Stage Status Node
Framework test ✅ spider9.cloudius-systems.com
dev Build ✅ spider13.cloudius-systems.com
test.py ✅ spider13.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider13.cloudius-systems.com
release Build ✅ spider2.cloudius-systems.com
test.py ✅ spider2.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider2.cloudius-systems.com
debug Build ✅ spider8.cloudius-systems.com
test.py ✅ spider8.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_storage_service		 ✅ spider8.cloudius-systems.com
dtest dtest with gossip topology changes ❌
dtest with tablets ❌
dtest with consistent topology changes ❌

Failed Tests (11/88422):

Build Details:

  • Duration: 4 hr 57 min

Note: To re-trigger CI for this PR, comment: @scylladbbot trigger-ci

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Dec 1, 2025

Failed Tests (11/88422):

The new test_client_routes_upgrade requires an adjustment (additional wait) because just after the upgrade, a request can fail with:

set_client_routes requires all nodes to support the CLIENT_ROUTES cluster feature

I will update test_client_routes_upgrade.

@patjed41 patjed41 mentioned this pull request Dec 3, 2025
Open
@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Dec 3, 2025 •
edited
Loading

v4:

  • Rebased to the latest master to get reference to the latest python-driver

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Dec 3, 2025

v5:

  • Updated test_events to use the newest python driver and avoid monkey patching
  • Updated test_client_routes_upgrade to handle requires all nodes to support the CLIENT_ROUTES cluster feature error
  • Also moved a small change in storage_service between commits, because earlier it was accidentally in a wrong commit

@scylladbbot
Copy link
Copy Markdown

scylladbbot commented Dec 3, 2025

🔴 CI State: FAILURE

Mode Stage Status Node
Framework test ✅ spider9.cloudius-systems.com
dev Build ✅ spider4.cloudius-systems.com
test.py ❌ spider4.cloudius-systems.com
release Build ✅ spider14.cloudius-systems.com
test.py ❌ spider14.cloudius-systems.com
debug Build ✅ spider17.cloudius-systems.com
test.py ❌ spider17.cloudius-systems.com
dtest dtest with consistent topology changes ✅
dtest with tablets ✅
dtest with gossip topology changes ✅

Failed Tests (10/78756):

Build Details:

  • Duration: 2 hr 54 min

Note: To re-trigger CI for this PR, comment: @scylladbbot trigger-ci

@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Dec 4, 2025

Failed Tests (10/78756):

Those failures look like scylla-driver incompatibility issue i.e. the old scylla-driver was used in CI environment instead of a new one. However, I haven't found a single log that confirms it in 100%, so I added a new assert in test_events that explicitly checks the driver compatibility.

v6:

  • Added assert to test_events to check that the python driver is compatible

@scylladbbot
Copy link
Copy Markdown

scylladbbot commented Dec 16, 2025

🟢 CI State: SUCCESS

Mode Stage Status Node
Framework test ✅ spider9.cloudius-systems.com
dev Build ✅ spider8.cloudius-systems.com
test.py ✅ spider8.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_client_routes_api		 ✅ spider8.cloudius-systems.com
release Build ✅ spider17.cloudius-systems.com
test.py ✅ spider17.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_client_routes_api		 ✅ spider17.cloudius-systems.com
debug Build ✅ spider3.cloudius-systems.com
test.py ✅ spider3.cloudius-systems.com
Unit Tests Custom
🔹 cluster/test_client_routes
🔹 rest_api/test_client_routes_api		 ✅ spider3.cloudius-systems.com
dtest dtest with tablets ✅
dtest with gossip topology changes ✅
dtest with consistent topology changes ✅

Build Details:

  • Duration: 6 hr 21 min

@andrzej-jackowski-scylladb andrzej-jackowski-scylladb added the status/merge_candidate Item needs a maintainer's assistance in merging label Dec 16, 2025
@andrzej-jackowski-scylladb
Copy link
Copy Markdown
Contributor Author

andrzej-jackowski-scylladb commented Dec 16, 2025

@patjed41 Ci is green, please consider merging

@patjed41
Copy link
Copy Markdown
Contributor

patjed41 commented Dec 16, 2025

queued

@scylladb-promoter scylladb-promoter merged commit 73db5c9 into scylladb:master Dec 16, 2025
69 checks passed
This was referenced Dec 18, 2025
Closed
Closed
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 15, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
b9da986 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
b65f9ea 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
@dkropachev dkropachev mentioned this pull request Jan 16, 2026
Merged
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
cea6fe5 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
@dkropachev dkropachev mentioned this pull request Jan 16, 2026
Closed
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
5b81a79 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
4e2aac0 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
90d4092 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
3db8f7b 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
a1dbdbf 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
39f0fb4 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
dkropachev added a commit to scylladb/gocql that referenced this pull request Jan 16, 2026
@dkropachev
Implement scylla-specific ClientRoutes feature
be7f6f6 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
This was referenced Feb 9, 2026
Open
Closed
Closed
Open
Open
sylwiaszunejko added a commit to sylwiaszunejko/python-driver that referenced this pull request Feb 16, 2026
@sylwiaszunejko
Implement scylla-specific ClientRoutes feature
6223ff6 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
sylwiaszunejko added a commit to sylwiaszunejko/python-driver that referenced this pull request Feb 16, 2026
@sylwiaszunejko
Implement scylla-specific ClientRoutes feature
b2b13a2 
This feature was implemented in scylladb/scylladb#27323
Idea is to enable clients to dynamically learn address translation
information from the system.client_routes table.
When this table is updated drivers get CLIENT_ROUTES_CHANGE event with
scope of the change.

This PR adds ability to configure driver to read this table and events
and maintain address translation mapping updated.
@sylwiaszunejko sylwiaszunejko mentioned this pull request Feb 16, 2026
Merged
8 tasks
@nikagra nikagra mentioned this pull request Feb 17, 2026
Closed
7 tasks
@nikagra nikagra mentioned this pull request Feb 26, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nyh nyh nyh left review comments

@nuivall nuivall nuivall approved these changes

@calebxyz calebxyz calebxyz left review comments

@patjed41 patjed41 patjed41 approved these changes

Copilot code review Copilot Copilot left review comments

@ptrsmrn ptrsmrn ptrsmrn approved these changes

@tgrabiec tgrabiec Awaiting requested review from tgrabiec tgrabiec is a code owner

+2 more reviewers

@gusev-p gusev-p gusev-p left review comments

@gleb-cloudius gleb-cloudius gleb-cloudius left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport/none Backport is not required promoted-to-master status/merge_candidate Item needs a maintainer's assistance in merging

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.