alternator: Add stream support for tablets#28500
alternator: Add stream support for tablets#28500scylladb-promoter merged 5 commits intoscylladb:masterfrom
Conversation
Docs Preview 📖Docs Preview for this pull request is available here Changed Files: Note: This preview will be available for 30 days and will be automatically deleted after that period. You can manually trigger a new build by committing changes. |
🔴 CI State: FAILURE
Build Details:
Note: To re-trigger CI for this PR, comment: |
🔴 CI State: FAILURE
Build Details:
Note: To re-trigger CI for this PR, comment: |
0f8607d to
7a5ea6c
Compare
7a5ea6c to
56c186f
Compare
🔴 CI State: FAILURE
Build Details:
Note: To re-trigger CI for this PR, comment: |
|
@radoslawcybulski please add |
|
@ScyllaPiotr @nyh can do the review, but we can't merge it and i assume we might need some rebases. |
|
The build stage failed:
|
|
@ScyllaPiotr fixed. |
Did you push? |
|
@ScyllaPiotr i did. EDIT: (there's this nice "progress" thingy in top right corner, which shows "processing updates". i guess github is lagged or something else broke)
|
response.get('Records') can return None when the key is missing from the
response dictionary. While the code already guards iteration with
'if records:', using an explicit default empty list is more defensive
and makes the intent clearer.
Addresses review comment:
scylladb#28500 (comment)
7abc51a to
f2cb12e
Compare
1067007 to
d7edd9a
Compare
|
Patch updated - squashed review fix commits. Added #28189 as base branch, so the patch would run tests successfully. |
Drop the tablet-vs-streams error checks added by scylladb#28500. The next commit replaces them with a deferred enablement mechanism that makes Alternator Streams compatible with tablet tables.
🟢 CI State: SUCCESS
Build Details:
|
👏 😄 |
Remove the error checks that reject enabling Alternator Streams on tablet tables. This is the code-guard removal part of scylladb#28500; the documentation and test updates from that PR are not included here. The next commit replaces these guards with a deferred enablement mechanism that makes Alternator Streams compatible with tablet tables. This commit is intended to be dropped once scylladb#28500 is merged, as that PR carries the same removal together with the full cleanup.
|
@radoslawcybulski with #28189 merged, this branch naturally became conflicted. |
d7edd9a to
794f0d9
Compare
|
Patch updated - rebased. |
| future<std::map<db_clock::time_point, cdc::streams_version>> cdc_get_versioned_streams(db_clock::time_point not_older_than, context); | ||
|
|
||
| // Read current generation timestamp for the given table. Throws runtime_error (see `cql3::untyped_result_set::one()`) if table not found. | ||
| // NOTE: there's a sibling `read_cdc_for_tablets_current_generation_timestamp` in `system_keyspace`, that does the same for tables backed up by tablets. |
There was a problem hiding this comment.
Minor wording: "backed up by tablets" reads like backup/restore; this seems to mean "backed by tablets". Clarifying avoids confusion.
| // NOTE: there's a sibling `read_cdc_for_tablets_current_generation_timestamp` in `system_keyspace`, that does the same for tables backed up by tablets. | |
| // NOTE: there's a sibling `read_cdc_for_tablets_current_generation_timestamp` in `system_keyspace`, that does the same for tables backed by tablets. |
| # | ||
| # SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.1 | ||
|
|
||
| # Tests for stream operations' nuanses that are intristic to ScyllaDB (parent-children relationship on stream shards). |
There was a problem hiding this comment.
Comment has multiple spelling errors (e.g. "nuanses", "intristic") which makes the test module description harder to read; please correct the wording.
| # Tests for stream operations' nuanses that are intristic to ScyllaDB (parent-children relationship on stream shards). | |
| # Tests for stream operation nuances that are intrinsic to ScyllaDB (parent-child relationship on stream shards). |
|
|
||
| # Tests for stream operations' nuanses that are intristic to ScyllaDB (parent-children relationship on stream shards). | ||
|
|
||
| import time, random, collections |
There was a problem hiding this comment.
Combined imports on one line ("import time, random, collections") deviates from the import style used elsewhere in Alternator tests and makes diffs noisier when adding/removing imports. Split these into separate import statements.
| import time, random, collections | |
| import collections | |
| import random | |
| import time |
| # run a test, where we create two cdc log generations (parent and children) | ||
| # and children count is double of parents (thus parents are splited into children) | ||
| # validate split assumptions: | ||
| # - every parents has the two distinct children | ||
| # - every two children has the same parent |
There was a problem hiding this comment.
Spelling/grammar in this comment is off (e.g. "splited", "every parents"), which makes the test intent harder to parse. Please fix the wording.
| # run a test, where we create two cdc log generations (parent and children) | |
| # and children count is double of parents (thus parents are splited into children) | |
| # validate split assumptions: | |
| # - every parents has the two distinct children | |
| # - every two children has the same parent | |
| # Run a test where we create two CDC log generations (parents and children) | |
| # and the number of children is double the number of parents (that is, each parent is split into children). | |
| # Validate the split assumptions: | |
| # - every parent has two distinct children | |
| # - every pair of children has the same parent |
| // filter out cdc generations older than the table or now() - cdc::ttl (typically dynamodb_streams_max_window - 24h) | ||
| auto low_ts = std::max(as_timepoint(schema->id()), db_clock::now() - ttl); | ||
| if (schema->table().uses_tablets()) { | ||
| // We can't use table creation time here, as tablets might report a | ||
| // generation timestamp just before table creation. This is safe | ||
| // because CDC generations are per-table and cannot pre-date the | ||
| // table, so expanding the window won't pull in unrelated data. | ||
| auto low_ts = db_clock::now() - ttl; | ||
| topologies = co_await _system_keyspace.read_cdc_for_tablets_versioned_streams(bs->ks_name(), bs->cf_name(), low_ts); | ||
| } else { | ||
| auto normal_token_owners = _proxy.get_token_metadata_ptr()->count_normal_token_owners(); | ||
| auto low_ts = std::max(as_timepoint(schema->id()), db_clock::now() - ttl); | ||
| topologies = co_await _sdks.cdc_get_versioned_streams(low_ts, { normal_token_owners }); | ||
| } |
There was a problem hiding this comment.
This module computes describe_cl based on token owners, but in tablets mode DescribeStream now reads CDC generations from system_keyspace virtual tables using consistency_level::ONE only. Consider adjusting the audited consistency level for the tablets path (or computing it after the uses_tablets() check) so audit records reflect the actual read behavior.
| future<> read_cdc_streams_history(table_id table, std::optional<db_clock::time_point> from, noncopyable_function<future<>(table_id, db_clock::time_point, cdc::cdc_stream_diff)> f); | ||
|
|
||
| // Reads current generation timestamp for the given table. Throws runtime_error (see `cql3::untyped_result_set::one()`) if table not found. | ||
| // NOTE: there's a sibling `cdc_current_generation_timestamp` in `system_distributed_keyspace`, that does the same for tables backed up by vnodes. |
There was a problem hiding this comment.
Minor wording: "backed up by" reads like backup/restore; this seems to mean "backed by" (vnodes). Clarifying this would avoid confusion for future readers.
| // NOTE: there's a sibling `cdc_current_generation_timestamp` in `system_distributed_keyspace`, that does the same for tables backed up by vnodes. | |
| // NOTE: there's a sibling `cdc_current_generation_timestamp` in `system_distributed_keyspace`, that does the same for tables backed by vnodes. |
🟢 CI State: SUCCESS
Build Details:
|
|
@radoslawcybulski needs rebase |
Add a reference to `system_keyspace` object to `executor` object in alternator. The reference is needed, because in future commit we will add there (and use) helper functions that read `cdc_log` tables for tablet based tables similarly to already existing siblings for vnodes living in `system_distributed_keyspace`.
Add helper functions to `system_keyspace` object, that deal with reading cdc content for tablet based table's. `read_cdc_for_tablets_current_generation_timestamp` will read current generation's timestamp. `read_cdc_for_tablets_versioned_streams` will build timestamp -> `cdc::streams_version` map similar to how `system_distributed_keyspace::cdc_get_versioned_streams` works. We're adding those helper functions, because their siblings in `system_distributed_keyspace` work only, when base table is backed up by vnodes. New additions work only, when base table is backed up by tablets.
Add a code, that will handle Streams reading, when table is using tablets underneath. Fixes scylladb#23838
Remove `if` condition, that prevented tables with tablets working with Streams. Remove a test, that verifies, that Alternator will reject tables with tablets underneath working with Streams feature enabled on them. Update few tests, that were expected to fail on tablets to enable their normal execution.
Add tests for Streams, when table uses tablets underneath. One test verifies filtering using CHILD_SHARDS feature. Other one makes sure we get read all data while the table undergoes tablet count change. Add `--tablet-load-stats-refresh-interval-in-seconds=1` to `alternator/run` script, as otherwise newly added tests will fail. The setting changes how often scylla refreshes tablet metadata. This can't be done using `scylla_config_temporary`, as 1) default is 60 seconds 2) scylla will wait full timeout (60s) to read configuration variable again.
|
Patch updated - rebased. |
nyh
left a comment
There was a problem hiding this comment.
Thanks. Looks good to me.
There are some typos that copilot found, and probably a bunch of other stuff, but I'll merge anyway. Feel free to send followup patches to later fix these typos or anything else you want to fix.
| future<std::map<db_clock::time_point, cdc::streams_version>> cdc_get_versioned_streams(db_clock::time_point not_older_than, context); | ||
|
|
||
| // Read current generation timestamp for the given table. Throws runtime_error (see `cql3::untyped_result_set::one()`) if table not found. | ||
| // NOTE: there's a sibling `read_cdc_for_tablets_current_generation_timestamp` in `system_keyspace`, that does the same for tables backed up by tablets. |
| TAGS = [] | ||
| # The following fixture is to ensure that tests in this module will be tested with both vnodes and tablets. | ||
| # This fixture runs automatically for every test in this module. | ||
| # To avoid relying on semantics of a similar fixture used in TTL tests, we define it locally here instead of reusing |
There was a problem hiding this comment.
I would have removed this comment, but never mind now.
| # | ||
| # SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.1 | ||
|
|
||
| # Tests for stream operations' nuanses that are intristic to ScyllaDB (parent-children relationship on stream shards). |
🟢 CI State: SUCCESS
Build Details:
|

Implements neccesary changes for Streams to work with tablet based tables.
system_keyspacethat helps reading cdc content from cdc log tables for tablet based base tables (similar api to ones for vnodes)ifchecks, update tests that fail / skip if tablets are selectedFixes #23838
Fixes SCYLLADB-463