Skip to content

Commit c082184

Browse files
nyhxemul
nyh
authored and
xemul
committed
alternator: document the state of tablet support in Alternator
In commit c24bc3b we decided that creating a new table in Alternator will by default use vnodes - not tablets - because of all the missing features in our tablets implementation that are important for Alternator, namely - LWT, CDC and Alternator TTL. We never documented this, or the fact that we support a tag `experimental:initial_tablets` which allows to override this decision and create an Alternator table using tablets. We also never documented what exactly doesn't work when Alternator uses tablet. This patch adds the missing documentation in docs/alternator/new-apis.md (which is a good place for describing the `experimental:initial_tablets` tag). The patch also adds a new test file, test_tablets.py, which includes tests for all the statements made in the document regarding how `experimental:initial_tablets` works and what works or doesn't work when tablets are enabled. Two existing tests - for TTL and Streams non-support with tablets - are moved to the new test file. When the tablets feature will finally be completed, both the document and the tests will need to be modified (some of the tests should be outright deleted). But it seems this will not happen for at least several months, and that is too long to wait without accurate documentation. Fixes #21629 Signed-off-by: Nadav Har'El <[email protected]> Closes #22462
1 parent 2bb455e commit c082184

4 files changed

Lines changed: 194 additions & 32 deletions

File tree

‎docs/alternator/new-apis.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,46 @@ If a certain data center or rack has no functional nodes, or doesn't even
144144
exist, an empty list (`[]`) is returned by the `/localnodes` request.
145145
A client should be prepared to consider expanding the node search to an
146146
entire data center, or other data centers, in that case.
147+
148+
## Tablets
149+
"Tablets" are ScyllaDB's new approach to replicating data across a cluster.
150+
It replaces the older approach which was named "vnodes". Compared to vnodes,
151+
tablets are smaller pieces of tables that are easier to move between nodes,
152+
and allow for faster growing or shrinking of the cluster when needed.
153+
154+
In this version, tablet support is incomplete and not all of the features
155+
which Alternator needs are supported with tablets. So currently, new
156+
Alternator tables default to using vnodes - not tablets.
157+
158+
However, if you do want to create an Alternator table which uses tablets,
159+
you can do this by specifying the `experimental:initial_tablets` tag in
160+
the CreateTable operation. The value of this tag can be:
161+
162+
* Any valid integer as the value of this tag enables tablets.
163+
Typically the number "0" is used - which tells ScyllaDB to pick a reasonable
164+
number of initial tablets. But any other number can be used, and this
165+
number overrides the default choice of initial number of tablets.
166+
167+
* Any non-integer value - e.g., the string "none" - creates the table
168+
without tablets - i.e., using vnodes.
169+
170+
The `experimental:initial_tablets` tag only has any effect while creating
171+
a new table with CreateTable - changing it later has no effect.
172+
173+
Because the tablets support is incomplete, when tablets are enabled for an
174+
Alternator table, the following features will not work for this table:
175+
176+
* The table must have one of the write isolation modes which does not
177+
not use LWT, because it's not supported with tablets. The allowed write
178+
isolation modes are `forbid_rmw` or `unsafe_rmw`.
179+
Setting the isolation mode to `always_use_lwt` will succeed, but the writes
180+
themselves will fail with an InternalServerError. At that point you can
181+
still change the write isolation mode of the table to a supported mode.
182+
See <https://github.com/scylladb/scylladb/issues/18068>.
183+
184+
* Enabling TTL with UpdateTableToLive doesn't work (results in an error).
185+
See <https://github.com/scylladb/scylla/issues/16567>.
186+
187+
* Enabling Streams with CreateTable or UpdateTable doesn't work
188+
(results in an error).
189+
See <https://github.com/scylladb/scylla/issues/16317>.

‎test/alternator/test_streams.py‎

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -23,26 +23,6 @@
2323
# using the following tags when creating each table below:
2424
TAGS = [{'Key': 'experimental:initial_tablets', 'Value': 'none'}]
2525

26-
# Before Alternator Streams is supported with tablets (#16317), let's verify
27-
# that enabling Streams results in an orderly error. This test should be
28-
# deleted when #16317 is fixed.
29-
def test_streams_enable_error_with_tablets(dynamodb, scylla_only):
30-
# Test attempting to create a table already with streams
31-
with pytest.raises(ClientError, match='ValidationException.*tablets'):
32-
with new_test_table(dynamodb,
33-
Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
34-
StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'},
35-
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
36-
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
37-
pass
38-
# Test attempting to add a stream to an existing table
39-
with new_test_table(dynamodb,
40-
Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
41-
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
42-
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
43-
with pytest.raises(ClientError, match='ValidationException.*tablets'):
44-
table.update(StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'});
45-
4626
stream_types = [ 'OLD_IMAGE', 'NEW_IMAGE', 'KEYS_ONLY', 'NEW_AND_OLD_IMAGES']
4727

4828
def disable_stream(dynamodbstreams, table):

‎test/alternator/test_tablets.py‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Copyright 2024-present ScyllaDB
2+
#
3+
# SPDX-License-Identifier: LicenseRef-ScyllaDB-Source-Available-1.0
4+
5+
# Tests for the Scylla-only "tablets" feature.
6+
#
7+
# Ideally, tablets are just an implementation detail (replacing the
8+
# old vnodes), that the DynamoDB API user would not even be aware
9+
# of. So there should be very few, if any, tests in this file.
10+
# However, temporarily - while the tablets feature is only partially
11+
# working and turned off by default (see issue #21989) - it is useful
12+
# to have here a few tests that clarify the situation and how to
13+
# override it. Most of these tests, or perhaps even this entire file,
14+
# will probably go away eventually.
15+
16+
import pytest
17+
import boto3
18+
from botocore.exceptions import ClientError
19+
20+
from .util import new_test_table
21+
22+
# All tests in this file are scylla-only
23+
@pytest.fixture(scope="function", autouse=True)
24+
def all_tests_are_scylla_only(scylla_only):
25+
pass
26+
27+
# Utility function for checking if a given table is using tablets
28+
# or not. We rely on some knowledge of Alternator internals:
29+
# 1. For table with name X, Scylla creates a keyspace called alternator_X
30+
# 2. We can read a CQL system table using the ".scylla.alternator." prefix.
31+
def uses_tablets(dynamodb, table):
32+
info = dynamodb.Table('.scylla.alternator.system_schema.scylla_keyspaces')
33+
try:
34+
response = info.query(
35+
KeyConditions={'keyspace_name': {
36+
'AttributeValueList': ['alternator_'+table.name],
37+
'ComparisonOperator': 'EQ'}})
38+
except dynamodb.meta.client.exceptions.ResourceNotFoundException:
39+
# The internal Scylla table doesn't even exist, either this isn't
40+
# Scylla or it's older Scylla and doesn't use tablets.
41+
return False
42+
if not 'Items' in response or not response['Items']:
43+
return False
44+
if 'initial_tablets' in response['Items'][0] and response['Items'][0]['initial_tablets']:
45+
return True
46+
return False
47+
48+
# Right now, new Alternator tables are created *without* tablets.
49+
# This test should be changed if this default ever changes.
50+
def test_default_tablets(dynamodb):
51+
schema = {
52+
'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
53+
'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
54+
with new_test_table(dynamodb, **schema) as table:
55+
# Change this assertion if Alternator's default changes!
56+
assert not uses_tablets(dynamodb, table)
57+
58+
# Tests for the initial_tablets tag. Currently, it is considered
59+
# experimental, and named "experimental:initial_tablets", but perhaps
60+
# in the future it will graduate out of experimental status and
61+
# the prefix will be replaced by "system:".
62+
initial_tablets_tag = 'experimental:initial_tablets'
63+
64+
# Check that a table created with a number as initial_tablets will use
65+
# tablets. Different numbers have different meanings (0 asked to use
66+
# default number, any other number overrides the default) but they
67+
# all enable tablets.
68+
def test_initial_tablets_number(dynamodb):
69+
for value in ['0', '4']:
70+
schema = {
71+
'Tags': [{'Key': initial_tablets_tag, 'Value': value}],
72+
'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
73+
'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
74+
with new_test_table(dynamodb, **schema) as table:
75+
assert uses_tablets(dynamodb, table)
76+
77+
# Check that a table created with a non-number (e.g., the string "none")
78+
# as initial_tablets, will not use tablets.
79+
def test_initial_tablets_number(dynamodb):
80+
schema = {
81+
'Tags': [{'Key': initial_tablets_tag, 'Value': 'none'}],
82+
'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
83+
'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
84+
with new_test_table(dynamodb, **schema) as table:
85+
assert not uses_tablets(dynamodb, table)
86+
87+
# Before Alternator TTL is supported with tablets (#16567), let's verify
88+
# that enabling TTL results in an orderly error. This test should be deleted
89+
# when #16567 is fixed.
90+
def test_ttl_enable_error_with_tablets(dynamodb):
91+
with new_test_table(dynamodb,
92+
Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
93+
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
94+
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
95+
with pytest.raises(ClientError, match='ValidationException.*tablets'):
96+
table.meta.client.update_time_to_live(TableName=table.name,
97+
TimeToLiveSpecification={'AttributeName': 'expiration', 'Enabled': True})
98+
99+
# Before Alternator Streams is supported with tablets (#16317), let's verify
100+
# that enabling Streams results in an orderly error. This test should be
101+
# deleted when #16317 is fixed.
102+
def test_streams_enable_error_with_tablets(dynamodb):
103+
# Test attempting to create a table already with streams
104+
with pytest.raises(ClientError, match='ValidationException.*tablets'):
105+
with new_test_table(dynamodb,
106+
Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
107+
StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'},
108+
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
109+
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
110+
pass
111+
# Test attempting to add a stream to an existing table
112+
with new_test_table(dynamodb,
113+
Tags=[{'Key': initial_tablets_tag, 'Value': '4'}],
114+
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
115+
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
116+
with pytest.raises(ClientError, match='ValidationException.*tablets'):
117+
table.update(StreamSpecification={'StreamEnabled': True, 'StreamViewType': 'KEYS_ONLY'});
118+
119+
# Currently, LWT is not supported for tablets because of known bugs
120+
# (see #18068). We still allow creating an Alternator table with
121+
# tablets and using LWT for write isolation (always_use_lwt),
122+
# but the writes themselves will fail. When #18068 is fixed, this
123+
# test should be fixed to expect success - not failure.
124+
def test_alternator_tablets_and_lwt(dynamodb):
125+
schema = {
126+
'Tags': [
127+
{'Key': initial_tablets_tag, 'Value': '0'},
128+
{'Key': 'system:write_isolation', 'Value': 'always_use_lwt'}],
129+
'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
130+
'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
131+
with new_test_table(dynamodb, **schema) as table:
132+
assert uses_tablets(dynamodb, table)
133+
# This put_item should pass after #18068 is fixed:
134+
with pytest.raises(ClientError, match="InternalServerError"):
135+
table.put_item(Item={'p': 'hello'})
136+
assert table.get_item(Key={'p': 'hello'})['Item'] == {'p': 'hello'}
137+
138+
# An Alternator table created tablets and with a write isolation
139+
# mode that doesn't use LWT ("forbid_rmw") works normally, even
140+
# before #18068 is fixed.
141+
def test_alternator_tablets_without_lwt(dynamodb):
142+
schema = {
143+
'Tags': [
144+
{'Key': initial_tablets_tag, 'Value': '0'},
145+
{'Key': 'system:write_isolation', 'Value': 'forbid_rmw'}],
146+
'KeySchema': [ { 'AttributeName': 'p', 'KeyType': 'HASH' } ],
147+
'AttributeDefinitions': [ { 'AttributeName': 'p', 'AttributeType': 'S' }]}
148+
with new_test_table(dynamodb, **schema) as table:
149+
assert uses_tablets(dynamodb, table)
150+
table.put_item(Item={'p': 'hello'})
151+
assert table.get_item(Key={'p': 'hello'})['Item'] == {'p': 'hello'}

‎test/alternator/test_ttl.py‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,18 +23,6 @@
2323
# using the following tags when creating each table below:
2424
TAGS = [{'Key': 'experimental:initial_tablets', 'Value': 'none'}]
2525

26-
# Before Alternator TTL is supported with tablets (#16567), let's verify
27-
# that enabling TTL results in an orderly error. This test should be deleted
28-
# when #16567 is fixed.
29-
def test_ttl_enable_error_with_tablets(dynamodb, scylla_only):
30-
with new_test_table(dynamodb,
31-
Tags=[{'Key': 'experimental:initial_tablets', 'Value': '4'}],
32-
KeySchema=[ { 'AttributeName': 'p', 'KeyType': 'HASH' }, ],
33-
AttributeDefinitions=[ { 'AttributeName': 'p', 'AttributeType': 'S' } ]) as table:
34-
with pytest.raises(ClientError, match='ValidationException.*tablets'):
35-
table.meta.client.update_time_to_live(TableName=table.name,
36-
TimeToLiveSpecification={'AttributeName': 'expiration', 'Enabled': True})
37-
3826
# passes_or_raises() is similar to pytest.raises(), except that while raises()
3927
# expects a certain exception must happen, the new passes_or_raises()
4028
# expects the code to either pass (not raise), or if it throws, it must

0 commit comments

Comments
 (0)