Skip to content

Commit ab79313

Browse files
morningmanclaude
andcommitted
[fix](row-binlog) Keep row binlog tablets on the binlog compaction policy
### What problem does this PR solve? Issue Number: None Related PR: #68033, #65810 Problem Summary: A table with `binlog.format = 'ROW'` has a hidden row binlog tablet for each data tablet. Row binlog tablets are created with the dedicated `binlog` compaction policy (`BinlogCumulativeCompactionPolicy`). #68033 stopped the cloud path from sending compaction policy changes to them; the local (non-cloud) path still sends them. Reproduced on master in local mode: 1. `ALTER TABLE t SET ("compaction_policy" = "time_series")`: `SchemaChangeHandler.updatePartitionProperties` sends the new policy to every visible index, the row binlog index included, and the BE (`update_tablet_meta_callback`) stores it without looking at the tablet role. The row binlog tablet meta then says `time_series`, and its next compaction runs the time series policy instead of the binlog policy. 2. On a `time_series` table, `ALTER TABLE t SET ("time_series_compaction_goal_size_mbytes" = "2048")` fails, because the BE rejects time series settings on a tablet that is not on the time series policy, and the row binlog tablets are on `binlog`: Failed to update partition[t]. tablet meta. Error: ... [INVALID_ARGUMENT]only time series compaction policy support time series config]. This operation maybe partial successfully, You should retry until success. The data tablets are already updated by then. Fix: - FE: `updatePartitionProperties` leaves out the row binlog tablets when the update carries a compaction policy or a time series setting, as #68033 does for cloud. ALTER sets one table property at a time, so such an update has nothing else for them; the other tablet meta updates (in memory, storage policy, binlog config, disable_auto_compaction, ...) still reach them. - BE: `update_tablet_meta_callback` refuses to change the compaction policy of a row binlog tablet, and no longer accepts `binlog` as a new policy: it is only set when a row binlog tablet is created (`CreateReplicaTask`). An older FE, or a future caller, can no longer move a row binlog tablet off its policy without an error. ### Release note Altering `compaction_policy` or a `time_series_compaction_*` property of a table with `binlog.format = 'ROW'` no longer changes the compaction policy of its hidden row binlog tablets, and no longer fails for the time series properties. ### Check List (For Author) - Test: - Build: `build.sh --fe --be` (BE Release; FE with checkstyle). - Regression test: `row_binlog_p0/test_row_binlog_alter_compaction_policy`. Before the fix it fails with `expected: <binlog> but was: <time_series>` (row binlog tablet meta after the policy ALTER); with the fix it passes. All 24 suites of `row_binlog_p0` pass with the fix. - Unit Test: FE `SchemaChangeHandlerTest` (37, including the new `testAlterCompactionSettingSkipsRowBinlogTablets`), all pass. With the FE change reverted, the new test fails. - Manual test: BE with this fix and FE without it. The policy ALTER fails with `can not change compaction policy of row binlog tablet <id>`, and the row binlog tablet meta keeps `binlog`. - Behavior changed: Yes. See Release note. The BE also rejects a compaction policy change on a row binlog tablet, and `binlog` as a new policy. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 15eb3f4 commit ab79313

5 files changed

Lines changed: 212 additions & 5 deletions

File tree

‎be/src/agent/task_worker_pool.cpp‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,6 @@
7373
#include "runtime/snapshot_loader.h"
7474
#include "runtime/user_function_cache.h"
7575
#include "service/backend_options.h"
76-
#include "storage/compaction/cumulative_compaction_binlog_policy.h"
7776
#include "storage/compaction/cumulative_compaction_policy.h"
7877
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
7978
#include "storage/data_dir.h"
@@ -910,12 +909,18 @@ void update_tablet_meta_callback(StorageEngine& engine, const TAgentTaskRequest&
910909
need_to_save = true;
911910
}
912911
if (tablet_meta_info.__isset.compaction_policy) {
912+
// The binlog policy is set when a row binlog tablet is created and never changes.
913+
if (tablet->is_row_binlog_tablet()) {
914+
status = Status::InvalidArgument(
915+
"can not change compaction policy of row binlog tablet {}",
916+
tablet_meta_info.tablet_id);
917+
continue;
918+
}
913919
if (tablet_meta_info.compaction_policy != CUMULATIVE_SIZE_BASED_POLICY &&
914-
tablet_meta_info.compaction_policy != CUMULATIVE_TIME_SERIES_POLICY &&
915-
tablet_meta_info.compaction_policy != CUMULATIVE_BINLOG_POLICY) {
920+
tablet_meta_info.compaction_policy != CUMULATIVE_TIME_SERIES_POLICY) {
916921
status = Status::InvalidArgument(
917922
"invalid compaction policy, only support for size_based or "
918-
"time_series or binlog");
923+
"time_series");
919924
continue;
920925
}
921926
tablet->tablet_meta()->set_compaction_policy(tablet_meta_info.compaction_policy);

‎fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3101,9 +3101,14 @@ protected void updatePartitionProperties(Database db, OlapTable olapTable, Parti
31013101
String partitionName = partition.getName();
31023102
// Use the captured partition identity, including temporary partitions with the same name.
31033103
Map<Long, Set<Pair<Long, Integer>>> beIdToTabletIdWithHash = Maps.newHashMap();
3104+
// Row binlog tablets keep the binlog compaction policy they are created with, so a new compaction
3105+
// policy or time series setting must not reach them. ALTER sets one table property at a time,
3106+
// so such an update has nothing else for them.
3107+
boolean includeRowBinlog = compactionPolicy == null
3108+
&& (timeSeriesCompactionConfig == null || timeSeriesCompactionConfig.isEmpty());
31043109
olapTable.readLock();
31053110
try {
3106-
for (MaterializedIndex index : partition.getMaterializedIndices(IndexExtState.VISIBLE, true)) {
3111+
for (MaterializedIndex index : partition.getMaterializedIndices(IndexExtState.VISIBLE, includeRowBinlog)) {
31073112
int schemaHash = olapTable.getSchemaHashByIndexId(index.getId());
31083113
for (Tablet tablet : index.getTablets()) {
31093114
for (Replica replica : tablet.getReplicas()) {

‎fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,15 +23,19 @@
2323
import org.apache.doris.catalog.Env;
2424
import org.apache.doris.catalog.Index;
2525
import org.apache.doris.catalog.KeysType;
26+
import org.apache.doris.catalog.MaterializedIndex;
27+
import org.apache.doris.catalog.MaterializedIndex.IndexExtState;
2628
import org.apache.doris.catalog.MaterializedIndexMeta;
2729
import org.apache.doris.catalog.OlapTable;
30+
import org.apache.doris.catalog.Partition;
2831
import org.apache.doris.catalog.PrimitiveType;
2932
import org.apache.doris.catalog.ScalarType;
3033
import org.apache.doris.catalog.Table;
3134
import org.apache.doris.catalog.info.ColumnPosition;
3235
import org.apache.doris.catalog.info.IndexType;
3336
import org.apache.doris.common.DdlException;
3437
import org.apache.doris.common.FeConstants;
38+
import org.apache.doris.common.Pair;
3539
import org.apache.doris.common.jmockit.Deencapsulation;
3640
import org.apache.doris.indexpolicy.DropIndexPolicyLog;
3741
import org.apache.doris.indexpolicy.IndexPolicy;
@@ -43,6 +47,7 @@
4347
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
4448
import org.apache.doris.qe.ConnectContext;
4549
import org.apache.doris.qe.StmtExecutor;
50+
import org.apache.doris.task.UpdateTabletMetaInfoTask;
4651
import org.apache.doris.utframe.TestWithFeService;
4752

4853
import com.google.common.collect.Lists;
@@ -52,12 +57,14 @@
5257
import org.apache.logging.log4j.Logger;
5358
import org.junit.jupiter.api.Assertions;
5459
import org.junit.jupiter.api.Test;
60+
import org.mockito.MockedConstruction;
5561
import org.mockito.Mockito;
5662

5763
import java.lang.reflect.Method;
5864
import java.util.LinkedList;
5965
import java.util.List;
6066
import java.util.Map;
67+
import java.util.Set;
6168
import java.util.concurrent.atomic.AtomicInteger;
6269
import java.util.function.IntSupplier;
6370
import java.util.stream.Collectors;
@@ -331,6 +338,59 @@ public void testWithRowBinlogSchemaChangeWithHistoricalValue() throws Exception
331338
Assertions.assertFalse(cols.contains(Column.generateBeforeColName(Column.SEQUENCE_COL)));
332339
}
333340

341+
@Test
342+
@SuppressWarnings("unchecked")
343+
public void testAlterCompactionSettingSkipsRowBinlogTablets() throws Exception {
344+
String tableName = "binlog_compaction_setting";
345+
String create = "CREATE TABLE IF NOT EXISTS test." + tableName + " (\n"
346+
+ "k1 INT NOT NULL,\n"
347+
+ "v1 INT\n"
348+
+ ")\n"
349+
+ "DUPLICATE KEY(k1)\n"
350+
+ "DISTRIBUTED BY HASH(k1) BUCKETS 1\n"
351+
+ "PROPERTIES('replication_num'='1','binlog.enable'='true','binlog.format'='ROW');";
352+
createTableWithRowBinlog(create);
353+
354+
Database db = Env.getCurrentInternalCatalog().getDbOrMetaException("test");
355+
OlapTable tbl = (OlapTable) db.getTableOrMetaException(tableName, Table.TableType.OLAP);
356+
Set<Long> dataTabletIds = Sets.newHashSet();
357+
Set<Long> rowBinlogTabletIds = Sets.newHashSet();
358+
for (Partition partition : tbl.getPartitions()) {
359+
for (MaterializedIndex index : partition.getMaterializedIndices(IndexExtState.VISIBLE, true)) {
360+
(index.isRowBinlog() ? rowBinlogTabletIds : dataTabletIds).addAll(index.getTabletIdsInOrder());
361+
}
362+
}
363+
Assertions.assertFalse(rowBinlogTabletIds.isEmpty());
364+
365+
// Build the update tasks without sending them, and record the tablets they are for.
366+
Set<Long> updatedTabletIds = Sets.newHashSet();
367+
boolean originalRunningUnitTest = FeConstants.runningUnitTest;
368+
FeConstants.runningUnitTest = true;
369+
try (MockedConstruction<UpdateTabletMetaInfoTask> ignored = Mockito.mockConstruction(
370+
UpdateTabletMetaInfoTask.class, (task, context) -> {
371+
for (Pair<Long, Integer> tablet : (Set<Pair<Long, Integer>>) context.arguments().get(1)) {
372+
updatedTabletIds.add(tablet.first);
373+
}
374+
})) {
375+
alterTable("ALTER TABLE test." + tableName + " SET ('compaction_policy' = 'time_series')",
376+
connectContext);
377+
Assertions.assertEquals(dataTabletIds, updatedTabletIds);
378+
379+
updatedTabletIds.clear();
380+
alterTable("ALTER TABLE test." + tableName + " SET ('time_series_compaction_goal_size_mbytes' = '2048')",
381+
connectContext);
382+
Assertions.assertEquals(dataTabletIds, updatedTabletIds);
383+
384+
// Other tablet meta still goes to the row binlog tablets.
385+
updatedTabletIds.clear();
386+
alterTable("ALTER TABLE test." + tableName + " SET ('disable_auto_compaction' = 'true')",
387+
connectContext);
388+
Assertions.assertEquals(Sets.union(dataTabletIds, rowBinlogTabletIds), updatedTabletIds);
389+
} finally {
390+
FeConstants.runningUnitTest = originalRunningUnitTest;
391+
}
392+
}
393+
334394
@Test
335395
public void testModifyTableLightSchemaChangeReplayNormalizesBfColumns() throws Exception {
336396
String tableName = "sc_replay_bf_case";
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !alter_policy --
3+
0 1 10
4+
0 1 11
5+
0 2 20
6+
0 3 30
7+
8+
-- !alter_time_series_setting --
9+
0 1 10
10+
0 1 11
11+
0 2 20
12+
0 3 30
13+
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// Altering a compaction setting of a row binlog table must not reach its hidden row binlog
19+
// tablets: they keep the binlog compaction policy they were created with.
20+
suite("test_row_binlog_alter_compaction_policy") {
21+
if (isCloudMode()) {
22+
return
23+
}
24+
25+
def backendIdToIp = [:]
26+
def backendIdToHttpPort = [:]
27+
getBackendIpHttpPort(backendIdToIp, backendIdToHttpPort)
28+
29+
def getTabletMeta = { tablet ->
30+
def meta = null
31+
httpTest {
32+
endpoint "${backendIdToIp["${tablet.BackendId}"]}:${backendIdToHttpPort["${tablet.BackendId}"]}"
33+
uri "/api/meta/header/${tablet.TabletId}"
34+
op "get"
35+
check { respCode, body ->
36+
assertEquals(200, respCode as int)
37+
meta = parseJson(body.toString())
38+
}
39+
}
40+
return meta
41+
}
42+
43+
// A manual compaction makes a tablet switch to the policy in its tablet meta, so after it
44+
// the tablet status shows the policy the tablet really compacts with.
45+
def checkCompactionPolicy = { String tableName, String dataTabletPolicy ->
46+
trigger_and_wait_compaction(tableName, "cumulative")
47+
def rowBinlogTabletNum = 0
48+
for (def tablet : sql_return_maparray("SHOW TABLETS FROM ${tableName}")) {
49+
def expectedPolicy = dataTabletPolicy
50+
if (tablet.IsRowBinlog == "true") {
51+
expectedPolicy = "binlog"
52+
rowBinlogTabletNum++
53+
}
54+
assertEquals(expectedPolicy, getTabletMeta(tablet).compaction_policy)
55+
def (code, out, err) = be_show_tablet_status(backendIdToIp["${tablet.BackendId}"],
56+
backendIdToHttpPort["${tablet.BackendId}"], tablet.TabletId)
57+
assertEquals(0, code)
58+
assertEquals(expectedPolicy, parseJson(out.trim())["cumulative policy type"])
59+
}
60+
assertTrue(rowBinlogTabletNum > 0)
61+
}
62+
63+
// Case 1: change the compaction policy.
64+
sql "DROP TABLE IF EXISTS test_row_binlog_alter_compaction_policy_dup FORCE"
65+
sql """
66+
CREATE TABLE test_row_binlog_alter_compaction_policy_dup (
67+
k1 INT,
68+
v1 INT
69+
)
70+
DUPLICATE KEY(k1)
71+
DISTRIBUTED BY HASH(k1) BUCKETS 1
72+
PROPERTIES (
73+
"replication_num" = "1",
74+
"disable_auto_compaction" = "true",
75+
"binlog.enable" = "true",
76+
"binlog.format" = "ROW"
77+
)
78+
"""
79+
sql "INSERT INTO test_row_binlog_alter_compaction_policy_dup VALUES (1, 10), (2, 20)"
80+
sql "INSERT INTO test_row_binlog_alter_compaction_policy_dup VALUES (1, 11), (3, 30)"
81+
82+
sql """ALTER TABLE test_row_binlog_alter_compaction_policy_dup SET ("compaction_policy" = "time_series")"""
83+
checkCompactionPolicy("test_row_binlog_alter_compaction_policy_dup", "time_series")
84+
order_qt_alter_policy """
85+
SELECT __DORIS_BINLOG_OP__ AS op, k1, v1
86+
FROM binlog("table" = "test_row_binlog_alter_compaction_policy_dup")
87+
"""
88+
89+
// Case 2: change a time series setting. The row binlog tablets are not on the time_series
90+
// policy, so the setting must not be sent to them.
91+
sql "DROP TABLE IF EXISTS test_row_binlog_alter_compaction_policy_ts FORCE"
92+
sql """
93+
CREATE TABLE test_row_binlog_alter_compaction_policy_ts (
94+
k1 INT,
95+
v1 INT
96+
)
97+
DUPLICATE KEY(k1)
98+
DISTRIBUTED BY HASH(k1) BUCKETS 1
99+
PROPERTIES (
100+
"replication_num" = "1",
101+
"disable_auto_compaction" = "true",
102+
"compaction_policy" = "time_series",
103+
"binlog.enable" = "true",
104+
"binlog.format" = "ROW"
105+
)
106+
"""
107+
sql "INSERT INTO test_row_binlog_alter_compaction_policy_ts VALUES (1, 10), (2, 20)"
108+
sql "INSERT INTO test_row_binlog_alter_compaction_policy_ts VALUES (1, 11), (3, 30)"
109+
110+
sql """
111+
ALTER TABLE test_row_binlog_alter_compaction_policy_ts
112+
SET ("time_series_compaction_goal_size_mbytes" = "2048")
113+
"""
114+
for (def tablet : sql_return_maparray("SHOW TABLETS FROM test_row_binlog_alter_compaction_policy_ts")) {
115+
if (tablet.IsRowBinlog != "true") {
116+
assertEquals(2048L, getTabletMeta(tablet).time_series_compaction_goal_size_mbytes as Long)
117+
}
118+
}
119+
checkCompactionPolicy("test_row_binlog_alter_compaction_policy_ts", "time_series")
120+
order_qt_alter_time_series_setting """
121+
SELECT __DORIS_BINLOG_OP__ AS op, k1, v1
122+
FROM binlog("table" = "test_row_binlog_alter_compaction_policy_ts")
123+
"""
124+
}

0 commit comments

Comments
 (0)