Skip to content

Commit 938f875

Browse files
morningmanclaude
andcommitted
[fix](fe) Reject unknown binlog properties and non-boolean binlog.enable
### What problem does this PR solve? Issue Number: None Related PR: #69311 Problem Summary: **In short.** CCR binlog is configured by four properties: `binlog.enable`, `binlog.ttl_seconds`, `binlog.max_bytes` and `binlog.max_history_nums`. `PropertyAnalyzer.analyzeBinlogConfig`, which every DDL uses to check them, ignored any other key under `binlog.` and read `binlog.enable` with `Boolean.parseBoolean`, so a misspelled key or a value such as `"yes"` was accepted without a word. The worst case: `ALTER TABLE t SET ("binlog.enable" = "yes")` succeeds and turns off the CCR binlog of a table that had it on. This PR rejects both, the way #69311 already made the `changelog.*` properties strict. **Background.** - Every DDL path validates the binlog properties with `analyzeBinlogConfig`: CREATE DATABASE and ALTER DATABASE (`CreateDatabaseCommand` / `AlterDatabasePropertiesCommand.validate`), CREATE TABLE (`InternalCatalog.createOlapTable`) and ALTER TABLE (`ModifyTablePropertiesOp.validate`, then `SchemaChangeHandler.updateBinlogConfig`). Database properties are stored as the user wrote them. - #69311 split changelog (formerly row binlog) from CCR binlog. Its `changelog.*` properties reject unknown keys (`Unknown changelog property: <key>`) and accept only `true` / `false` for `changelog.enable`; the replaced `binlog.format` and `binlog.need_historical_value` are rejected in every DDL. The four CCR keys kept the lenient parsing. **The problem, and what it cost.** On a cluster running master before this change: | Statement | Before | |---|---| | `ALTER TABLE t SET ("binlog.enable" = "yes")`, `t` has binlog on | succeeds; `t`'s binlog is now off | | `CREATE TABLE ... ("binlog.enable" = "yes")` | the table is created with binlog off | | `ALTER TABLE t SET ("binlog.ttl_second" = "3600")` (typo) | `Nothing is changed. please check your alter stmt.` | | `CREATE TABLE ... ("binlog.ttl_second" = "3600")` | `Unknown properties: {binlog.ttl_second=3600}` | | `CREATE DATABASE d PROPERTIES ("binlog.ttl_second" = "3600")` | succeeds; the key is persisted | | `ALTER DATABASE d SET PROPERTIES ("binlog.max_byte" = "1024")` | succeeds; the key is persisted | | `ALTER DATABASE d SET PROPERTIES ("binlog.enable" = "1")` | succeeds; `"1"` is persisted and read as false | **How this PR fixes it.** - `analyzeBinlogConfig` rejects any key under `binlog.` other than the four with `Unknown binlog property: <key>`. The replaced `binlog.format` and `binlog.need_historical_value` are still checked first, so they keep their message that names the replacement. - `binlog.enable` accepts only `true` or `false`, in any case, and otherwise fails with the existing `Invalid binlog enable value: <value>`, which the old `try` / `catch` around `Boolean.parseBoolean` could never reach. - CREATE TABLE runs this check on the properties as written, where it only checked the replaced keys before. Its later call to `analyzeBinlogConfig` sees the effective binlog config (the database default merged with the table properties) that `createOlapTable` puts back over them, and by then `"yes"` has already become `"false"`. - Persisted metadata is not validated again: image load and edit log replay read binlog properties through `BinlogConfig.mergeFromProperties`, which is unchanged, so a database that already stored such a key loads as before. A database's binlog default reaches CREATE TABLE as a `BinlogConfig` object, not as properties, so it does not make CREATE TABLE fail either. **Results.** | Statement | After | |---|---| | `ALTER TABLE t SET ("binlog.enable" = "yes")` | `Invalid binlog enable value: yes`; `t` keeps its binlog setting | | `CREATE TABLE ... ("binlog.enable" = "yes")` | `Invalid binlog enable value: yes` | | `ALTER TABLE` / `CREATE TABLE` / `CREATE DATABASE` with `"binlog.ttl_second"` | `Unknown binlog property: binlog.ttl_second` | | `ALTER DATABASE ... ("binlog.max_byte" = "1024")` | `Unknown binlog property: binlog.max_byte` | | `ALTER DATABASE ... ("binlog.enable" = "1")` | `Invalid binlog enable value: 1` | | `"binlog.enable" = "TRUE"` / `"False"` | accepted, stored as `true` / `false` (unchanged) | ### Release note The `binlog.*` properties of tables and databases are validated strictly: CREATE / ALTER TABLE and CREATE / ALTER DATABASE reject a key under `binlog.` other than `binlog.enable`, `binlog.ttl_seconds`, `binlog.max_bytes` and `binlog.max_history_nums`, and `binlog.enable` accepts only `true` or `false`. Before, such keys were ignored or stored, and any other `binlog.enable` value was read as false. ### Check List (For Author) - Test: Unit Test, Regression test, Manual test - FE UT: `PropertyAnalyzerTest.testAnalyzeBinlogConfig` and `ChangelogConfigSplitTest.testDdlValidatesBinlogProperties` (new; CREATE TABLE and ALTER TABLE), plus the other test classes that create tables or databases with binlog properties. - Regression: `ddl_p0/test_binlog_property_validation` (new) covers each row of the Results table on tables and databases. - Manual: the Before table was taken on a local cluster running master before this change. - Behavior changed: Yes. DDL that set an unknown `binlog.*` key or a non-boolean `binlog.enable` now fails instead of being ignored, stored, or read as false. - Does this need documentation: No. The current docs do not describe other `binlog.*` keys or other `binlog.enable` values. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 351d938 commit 938f875

6 files changed

Lines changed: 174 additions & 7 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1534,17 +1534,22 @@ public static Map<String, String> analyzeBinlogConfig(Map<String, String> proper
15341534
return null;
15351535
}
15361536
checkLegacyBinlogProperties(properties);
1537+
for (String key : properties.keySet()) {
1538+
if (key.startsWith(PROPERTIES_BINLOG_PREFIX) && !key.equals(PROPERTIES_BINLOG_ENABLE)
1539+
&& !key.equals(PROPERTIES_BINLOG_TTL_SECONDS) && !key.equals(PROPERTIES_BINLOG_MAX_BYTES)
1540+
&& !key.equals(PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
1541+
throw new AnalysisException("Unknown binlog property: " + key);
1542+
}
1543+
}
15371544

15381545
Map<String, String> binlogConfigMap = Maps.newHashMap();
15391546
// check PROPERTIES_BINLOG_ENABLE = "binlog.enable";
15401547
if (properties.containsKey(PROPERTIES_BINLOG_ENABLE)) {
1541-
String enable = properties.get(PROPERTIES_BINLOG_ENABLE);
1542-
try {
1543-
binlogConfigMap.put(PROPERTIES_BINLOG_ENABLE, String.valueOf(Boolean.parseBoolean(enable)));
1544-
properties.remove(PROPERTIES_BINLOG_ENABLE);
1545-
} catch (Exception e) {
1548+
String enable = properties.remove(PROPERTIES_BINLOG_ENABLE);
1549+
if (!StringUtils.equalsAnyIgnoreCase(enable, "true", "false")) {
15461550
throw new AnalysisException("Invalid binlog enable value: " + enable);
15471551
}
1552+
binlogConfigMap.put(PROPERTIES_BINLOG_ENABLE, String.valueOf(Boolean.parseBoolean(enable)));
15481553
}
15491554
// check PROPERTIES_BINLOG_TTL_SECONDS = "binlog.ttl_seconds";
15501555
if (properties.containsKey(PROPERTIES_BINLOG_TTL_SECONDS)) {
@@ -1587,7 +1592,7 @@ public static Map<String, String> analyzeBinlogConfig(Map<String, String> proper
15871592
/**
15881593
* Rejects the binlog keys the changelog.* properties replaced, whatever their value is.
15891594
*/
1590-
public static void checkLegacyBinlogProperties(Map<String, String> properties) throws AnalysisException {
1595+
private static void checkLegacyBinlogProperties(Map<String, String> properties) throws AnalysisException {
15911596
if (properties.containsKey(PROPERTIES_LEGACY_BINLOG_FORMAT)) {
15921597
throw new AnalysisException("Property " + PROPERTIES_LEGACY_BINLOG_FORMAT
15931598
+ " is no longer supported, use \"" + PROPERTIES_CHANGELOG_ENABLE

‎fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2495,7 +2495,8 @@ private boolean createOlapTable(Database db, CreateTableInfo createTableInfo) th
24952495
boolean tableHasExist = false;
24962496
ChangelogConfig createTableChangelogConfig;
24972497
try {
2498-
PropertyAnalyzer.checkLegacyBinlogProperties(createTableInfo.getProperties());
2498+
// Check the binlog properties as written: the effective binlog config is put back over them below.
2499+
PropertyAnalyzer.analyzeBinlogConfig(Maps.newHashMap(createTableInfo.getProperties()));
24992500
createTableChangelogConfig = db.getChangelogConfigForCreateTable(createTableInfo.getProperties());
25002501
} catch (AnalysisException e) {
25012502
throw new DdlException(e.getMessage());

‎fe/fe-core/src/test/java/org/apache/doris/catalog/ChangelogConfigSplitTest.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,22 @@ public void testCreateTableValidatesChangelogProperties() {
280280
"Unknown changelog property: changelog.format");
281281
}
282282

283+
@Test
284+
public void testDdlValidatesBinlogProperties() throws Exception {
285+
String dup = "(k INT, v INT) DUPLICATE KEY(k)";
286+
assertCreateFails("binlog_invalid_enable", dup, "'binlog.enable'='yes'", "Invalid binlog enable value: yes");
287+
assertCreateFails("binlog_unknown_key", dup, "'binlog.ttl_second'='10'",
288+
"Unknown binlog property: binlog.ttl_second");
289+
290+
OlapTable table = createDupTable(db, "binlog_alter", "'binlog.enable'='TRUE'");
291+
Assertions.assertTrue(table.getBinlogConfig().isEnableForCCR());
292+
assertAlterFails("ALTER TABLE split.binlog_alter SET ('binlog.enable'='yes')",
293+
"Invalid binlog enable value: yes");
294+
assertAlterFails("ALTER TABLE split.binlog_alter SET ('binlog.ttl_second'='10')",
295+
"Unknown binlog property: binlog.ttl_second");
296+
Assertions.assertTrue(table.getBinlogConfig().isEnableForCCR());
297+
}
298+
283299
@Test
284300
public void testDatabaseChangelogDefaults() throws Exception {
285301
createDatabaseWithSql("CREATE DATABASE split_default_db "

‎fe/fe-core/src/test/java/org/apache/doris/common/PropertyAnalyzerTest.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,37 @@ public void testBinlogTtlRejectsValuesBelowDisabledSentinel() throws AnalysisExc
6767
() -> PropertyAnalyzer.analyzeBinlogConfig(properties));
6868
}
6969

70+
@Test
71+
public void testAnalyzeBinlogConfig() throws AnalysisException {
72+
Map<String, String> properties = new HashMap<>();
73+
properties.put(PropertyAnalyzer.PROPERTIES_BINLOG_ENABLE, "TRUE");
74+
properties.put(PropertyAnalyzer.PROPERTIES_BINLOG_TTL_SECONDS, "60");
75+
properties.put(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES, "1024");
76+
properties.put(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS, "10");
77+
properties.put(PropertyAnalyzer.PROPERTIES_CHANGELOG_ENABLE, "false");
78+
Map<String, String> analyzed = PropertyAnalyzer.analyzeBinlogConfig(properties);
79+
Assertions.assertEquals(Map.of(PropertyAnalyzer.PROPERTIES_BINLOG_ENABLE, "true",
80+
PropertyAnalyzer.PROPERTIES_BINLOG_TTL_SECONDS, "60",
81+
PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES, "1024",
82+
PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS, "10"), analyzed);
83+
Assertions.assertEquals(Map.of(PropertyAnalyzer.PROPERTIES_CHANGELOG_ENABLE, "false"), properties);
84+
85+
properties.put(PropertyAnalyzer.PROPERTIES_BINLOG_ENABLE, "False");
86+
Assertions.assertEquals("false", PropertyAnalyzer.analyzeBinlogConfig(properties)
87+
.get(PropertyAnalyzer.PROPERTIES_BINLOG_ENABLE));
88+
89+
for (Map.Entry<String, String> invalid : Map.of(
90+
PropertyAnalyzer.PROPERTIES_BINLOG_ENABLE, "Invalid binlog enable value: yes",
91+
"binlog.ttl_second", "Unknown binlog property: binlog.ttl_second",
92+
"binlog.compaction", "Unknown binlog property: binlog.compaction").entrySet()) {
93+
Map<String, String> invalidProperties = new HashMap<>();
94+
invalidProperties.put(invalid.getKey(), "yes");
95+
AnalysisException exception = Assertions.assertThrows(AnalysisException.class,
96+
() -> PropertyAnalyzer.analyzeBinlogConfig(invalidProperties));
97+
Assertions.assertTrue(exception.getMessage().contains(invalid.getValue()), exception.getMessage());
98+
}
99+
}
100+
70101
@ParameterizedTest
71102
@ValueSource(strings = {"ROW", "row", "STATEMENT_AND_SNAPSHOT", "statement_and_snapshot", "rows"})
72103
public void testReplacedBinlogKeysAreRejected(String value) {
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !table_binlog_enable --
3+
test_binlog_property_validation_tbl binlog.enable false
4+
5+
-- !database_binlog_properties --
6+
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// The "binlog.*" properties configure CCR binlog. A key under "binlog." other than binlog.enable,
19+
// binlog.ttl_seconds, binlog.max_bytes and binlog.max_history_nums is rejected instead of being
20+
// ignored, and binlog.enable accepts only true or false, in any case.
21+
suite("test_binlog_property_validation") {
22+
sql "DROP TABLE IF EXISTS test_binlog_property_validation_tbl"
23+
24+
test {
25+
sql """
26+
CREATE TABLE test_binlog_property_validation_tbl (
27+
k1 INT NOT NULL,
28+
v1 INT
29+
)
30+
DUPLICATE KEY(k1)
31+
DISTRIBUTED BY HASH(k1) BUCKETS 1
32+
PROPERTIES (
33+
"replication_num" = "1",
34+
"binlog.ttl_second" = "3600"
35+
)
36+
"""
37+
exception "Unknown binlog property: binlog.ttl_second"
38+
}
39+
test {
40+
sql """
41+
CREATE TABLE test_binlog_property_validation_tbl (
42+
k1 INT NOT NULL,
43+
v1 INT
44+
)
45+
DUPLICATE KEY(k1)
46+
DISTRIBUTED BY HASH(k1) BUCKETS 1
47+
PROPERTIES (
48+
"replication_num" = "1",
49+
"binlog.enable" = "yes"
50+
)
51+
"""
52+
exception "Invalid binlog enable value: yes"
53+
}
54+
55+
sql """
56+
CREATE TABLE test_binlog_property_validation_tbl (
57+
k1 INT NOT NULL,
58+
v1 INT
59+
)
60+
DUPLICATE KEY(k1)
61+
DISTRIBUTED BY HASH(k1) BUCKETS 1
62+
PROPERTIES (
63+
"replication_num" = "1",
64+
"binlog.enable" = "False"
65+
)
66+
"""
67+
order_qt_table_binlog_enable """
68+
SELECT TABLE_NAME, PROPERTY_NAME, PROPERTY_VALUE
69+
FROM information_schema.table_properties
70+
WHERE TABLE_CATALOG = 'internal'
71+
AND TABLE_SCHEMA = '${context.dbName}'
72+
AND TABLE_NAME = 'test_binlog_property_validation_tbl'
73+
AND PROPERTY_NAME = 'binlog.enable'
74+
"""
75+
76+
// The unknown key used to fail with "Nothing is changed", and "yes" used to be read as false,
77+
// which turned binlog off on a table that had it on.
78+
test {
79+
sql """ALTER TABLE test_binlog_property_validation_tbl SET ("binlog.ttl_second" = "3600")"""
80+
exception "Unknown binlog property: binlog.ttl_second"
81+
}
82+
test {
83+
sql """ALTER TABLE test_binlog_property_validation_tbl SET ("binlog.enable" = "yes")"""
84+
exception "Invalid binlog enable value: yes"
85+
}
86+
87+
sql "DROP DATABASE IF EXISTS test_binlog_property_validation_db"
88+
test {
89+
sql """CREATE DATABASE test_binlog_property_validation_db PROPERTIES ("binlog.ttl_second" = "3600")"""
90+
exception "Unknown binlog property: binlog.ttl_second"
91+
}
92+
sql "CREATE DATABASE test_binlog_property_validation_db"
93+
// Database properties are stored as given, so these used to be persisted as they were.
94+
test {
95+
sql """ALTER DATABASE test_binlog_property_validation_db SET PROPERTIES ("binlog.max_byte" = "1024")"""
96+
exception "Unknown binlog property: binlog.max_byte"
97+
}
98+
test {
99+
sql """ALTER DATABASE test_binlog_property_validation_db SET PROPERTIES ("binlog.enable" = "1")"""
100+
exception "Invalid binlog enable value: 1"
101+
}
102+
order_qt_database_binlog_properties """
103+
SELECT SCHEMA_NAME, PROPERTY_NAME, PROPERTY_VALUE
104+
FROM information_schema.database_properties
105+
WHERE SCHEMA_NAME = 'test_binlog_property_validation_db'
106+
AND PROPERTY_NAME LIKE 'binlog.%'
107+
"""
108+
}

0 commit comments

Comments
 (0)