Skip to content

Commit 722469c

Browse files
morningmanclaude
andcommitted
[improvement](fe) Validate submitted meta.cache keys of the catalog's own engines
### What problem does this PR solve? Issue Number: None Related PR: #66717, #67726 Problem Summary: branch-4.x (#66717) validates every meta.cache.* catalog property against a whitelist at CREATE / ALTER CATALOG. Master (#67726) rejects only an unknown .max-weight entry that the statement submits and checks the values of a few keys, so a misspelled option (meta.cache.hive.file.max-weigth), enable=1 (read by Boolean.parseBoolean as false, which silently turns the cache off) and ttl-second=-2 are all accepted. Master also accepts a catalog quota above the FE global quota (external_meta_cache_max_weight) and only clamps it at run time, so SHOW CREATE CATALOG promises more memory than the caches get. Every key that a CREATE or ALTER CATALOG submits under meta.cache.<engine>. for an engine the catalog owns must now name a known option (enable, ttl-second, capacity, max-weight) with a value of its type: enable is true or false, ttl-second >= -1, capacity >= 0 and max-weight a positive size. The check sits in CacheSpec.checkWeightProperties, which the hive (with its iceberg sibling), iceberg, paimon, max_compute and adbc connectors already call with the submitted properties, and in a new ExternalCatalog hook that CatalogMgr calls on CREATE and ALTER for the namespace fe-core owns (default, or doris for a remote Doris catalog); replay never runs it. Unknown entries stay accepted, since a newer connector may read an entry this version does not know, and keys an earlier statement stored are not validated again, so a stored key never blocks an unrelated ALTER. A submitted catalog quota, and a submitted entry quota of a catalog without one, may not exceed the FE global quota of the FE running the statement. ### Release note CREATE / ALTER CATALOG now rejects a meta.cache.<engine>.<entry>.<option> property whose option is not enable, ttl-second, capacity or max-weight or whose value has the wrong type, and a meta.cache.max-weight (or an entry max-weight of a catalog without one) above external_meta_cache_max_weight. ### Check List (For Author) - Test: Unit Test, Regression test - CacheSpecTest; HiveCatalogPropertiesTest, IcebergConnectorValidatePropertiesTest, PaimonConnectorValidatePropertiesTest, MaxComputePartitionCacheTest, AdbcCatalogPropertiesTest; ExternalCatalogMetaCacheWeightTest - external_table_p0/test_catalog_ddl - Behavior changed: Yes. DDL that master accepted is now rejected: a misspelled option, a value of the wrong type, and a catalog or entry quota above the FE global quota. Compared with branch-4.x: unknown entries and engines a catalog does not route stay accepted; meta.cache.hive.partition_values.*, meta.cache.iceberg.snapshot.* and meta.cache.paimon.{table,snapshot}.max-weight are not supported on master (a submitted max-weight of them is rejected as an unknown weight entry, their other options and stored values are ignored); an entry max-weight of 0 is rejected instead of disabling the entry. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1bb3537 commit 722469c

12 files changed

Lines changed: 398 additions & 6 deletions

File tree

‎fe/fe-connector/fe-connector-adbc/src/test/java/org/apache/doris/connector/adbc/AdbcCatalogPropertiesTest.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,19 @@ void weightValidationDistinguishesSubmittedAndPersistedUnknownEntries() {
249249
current, Map.of("meta.cache.max-weight", "1MB", "meta.cache.adbc.metadata.max-weight", "2MB")));
250250
}
251251

252+
@Test
253+
void submittedMetaCacheKeysMustNameAKnownOption() {
254+
AdbcConnectorProvider provider = new AdbcConnectorProvider();
255+
Map<String, String> current = minimal();
256+
IllegalArgumentException misspelled = Assertions.assertThrows(IllegalArgumentException.class,
257+
() -> provider.validatePropertiesForUpdate(
258+
current, Map.of("meta.cache.adbc.metadata.ttl-secnod", "60")));
259+
Assertions.assertEquals("Unknown external meta cache property: meta.cache.adbc.metadata.ttl-secnod",
260+
misspelled.getMessage());
261+
current.put("meta.cache.adbc.metadata.ttl-secnod", "60");
262+
Assertions.assertDoesNotThrow(() -> provider.validatePropertiesForUpdate(current, Map.of("comment", "new")));
263+
}
264+
252265
/**
253266
* A cache knob is read through {@code CacheSpec}, which answers an unparseable value with the default
254267
* rather than an error. Silently caching for ten minutes when the operator wrote {@code ttl-second=6O0}

‎fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java‎

Lines changed: 92 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -220,13 +220,26 @@ public static void checkWeightProperties(
220220
checkWeightProperties(properties, Collections.emptyMap(), engine, knownEntries);
221221
}
222222

223-
/** DDL validation: reject unknown entries only when explicitly submitted by this statement. */
223+
/**
224+
* DDL validation: the keys this statement submits for {@code engine} must be well formed (see
225+
* {@link #checkSubmittedEngineProperties}), its submitted weights must not exceed the FE-global limit, and an
226+
* unknown weight entry is rejected only when this statement submits it.
227+
*/
224228
public static void checkWeightProperties(Map<String, String> properties,
225229
Map<String, String> submittedProperties, String engine, String... knownEntries) {
230+
checkWeightProperties(properties, submittedProperties, MetaCacheGovernance.globalMaxWeight(), engine,
231+
knownEntries);
232+
}
233+
234+
/** As {@link #checkWeightProperties(Map, Map, String, String...)}, against the given FE-global limit. */
235+
public static void checkWeightProperties(Map<String, String> properties, Map<String, String> submittedProperties,
236+
OptionalLong globalMaxWeight, String engine, String... knownEntries) {
237+
checkSubmittedEngineProperties(submittedProperties, engine);
226238
OptionalLong catalogMax = checkCatalogWeightProperty(properties);
227239
if (properties == null) {
228240
return;
229241
}
242+
checkSubmittedWeightsAgainstGlobal(properties, submittedProperties, globalMaxWeight, engine);
230243
String prefix = metaCacheKeyPrefix(engine);
231244
Set<String> weightKeys = new HashSet<>();
232245
for (String entry : knownEntries) {
@@ -244,17 +257,79 @@ public static void checkWeightProperties(Map<String, String> properties,
244257
// Like other connector properties, validate consumed values but tolerate unknown keys.
245258
continue;
246259
}
247-
long parsed = parseWeight(property.getValue(), key, false, 0L);
248-
if (parsed <= 0L) {
249-
throw new IllegalArgumentException(key + " must be positive");
250-
}
260+
long parsed = parsePositiveWeight(property.getValue(), key);
251261
if (catalogMax.isPresent() && parsed > catalogMax.getAsLong()) {
252262
throw new IllegalArgumentException(key + " can not exceed "
253263
+ MetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY);
254264
}
255265
}
256266
}
257267

268+
/**
269+
* DDL validation of the {@code meta.cache.<engine>.<entry>.<option>} keys this statement submits: the option must
270+
* be enable, ttl-second, capacity or max-weight, and the value one of its type. Entry names are not checked, so a
271+
* key of an entry that a newer connector reads stays accepted, and keys an earlier statement persisted are not
272+
* inspected, so a stored key never blocks an unrelated ALTER.
273+
*/
274+
public static void checkSubmittedEngineProperties(Map<String, String> submittedProperties, String engine) {
275+
String prefix = metaCacheKeyPrefix(engine);
276+
for (Map.Entry<String, String> property : submittedProperties.entrySet()) {
277+
String key = property.getKey();
278+
if (!key.startsWith(prefix)) {
279+
continue;
280+
}
281+
int optionStart = key.lastIndexOf('.');
282+
if (optionStart <= prefix.length()) {
283+
throw unknownProperty(key);
284+
}
285+
String value = property.getValue();
286+
switch (key.substring(optionStart)) {
287+
case KEY_ENABLE:
288+
checkBooleanProperty(value, key);
289+
break;
290+
case KEY_TTL_SECOND:
291+
checkLongProperty(value, CACHE_NO_TTL, key);
292+
break;
293+
case KEY_CAPACITY:
294+
checkLongProperty(value, 0L, key);
295+
break;
296+
case KEY_MAX_WEIGHT:
297+
parsePositiveWeight(value, key);
298+
break;
299+
default:
300+
throw unknownProperty(key);
301+
}
302+
}
303+
}
304+
305+
/**
306+
* DDL validation against the FE-global limit, the one of the FE running the statement: a submitted catalog
307+
* weight, and a submitted {@code engine} entry weight of a catalog without a catalog weight, must not exceed it.
308+
* Admission clamps every cache to the limit anyway; this keeps SHOW CREATE CATALOG from promising more. Only
309+
* submitted keys are checked, so a limit lowered after a restart never blocks an unrelated ALTER.
310+
*/
311+
public static void checkSubmittedWeightsAgainstGlobal(Map<String, String> effectiveProperties,
312+
Map<String, String> submittedProperties, OptionalLong globalMaxWeight, String engine) {
313+
if (!globalMaxWeight.isPresent()) {
314+
return;
315+
}
316+
boolean catalogLimited = effectiveProperties.containsKey(MetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY);
317+
String prefix = metaCacheKeyPrefix(engine);
318+
for (Map.Entry<String, String> property : submittedProperties.entrySet()) {
319+
String key = property.getKey();
320+
boolean limitedByGlobal = key.equals(MetaCacheBudgetManager.CATALOG_MAX_WEIGHT_PROPERTY)
321+
|| !catalogLimited && key.startsWith(prefix) && key.endsWith(KEY_MAX_WEIGHT);
322+
if (!limitedByGlobal) {
323+
continue;
324+
}
325+
long weight = parsePositiveWeight(property.getValue(), key);
326+
if (weight > globalMaxWeight.getAsLong()) {
327+
throw new IllegalArgumentException(key + " (" + weight + " bytes) can not exceed the FE global max"
328+
+ " weight external_meta_cache_max_weight (" + globalMaxWeight.getAsLong() + " bytes)");
329+
}
330+
}
331+
}
332+
258333
public static boolean isCacheEnabled(boolean enable, long ttlSecond, long capacity) {
259334
return enable && ttlSecond != 0 && capacity != 0;
260335
}
@@ -400,6 +475,18 @@ private static long checkedLong(BigInteger value, String key, String rawValue) {
400475
return value.longValue();
401476
}
402477

478+
private static long parsePositiveWeight(String value, String key) {
479+
long parsed = parseWeight(value, key, false, 0L);
480+
if (parsed <= 0L) {
481+
throw new IllegalArgumentException(key + " must be positive");
482+
}
483+
return parsed;
484+
}
485+
486+
private static IllegalArgumentException unknownProperty(String key) {
487+
return new IllegalArgumentException("Unknown external meta cache property: " + key);
488+
}
489+
403490
private static IllegalArgumentException invalidWeight(String key, String value) {
404491
return new IllegalArgumentException("Invalid cache weight for '" + key + "': " + value);
405492
}

‎fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/CacheSpecTest.java‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,87 @@ public void parsesAndValidatesWeightProperties() {
285285
() -> CacheSpec.checkCatalogWeightProperty(properties));
286286
}
287287

288+
@Test
289+
public void submittedEngineKeysMustNameAKnownOptionWithAValueOfItsType() {
290+
IllegalArgumentException misspelled = Assertions.assertThrows(IllegalArgumentException.class,
291+
() -> CacheSpec.checkSubmittedEngineProperties(
292+
Collections.singletonMap("meta.cache.iceberg.table.max-weigth", "16MB"), "iceberg"));
293+
Assertions.assertEquals("Unknown external meta cache property: meta.cache.iceberg.table.max-weigth",
294+
misspelled.getMessage());
295+
for (String key : new String[] {"meta.cache.iceberg.max-weight", "meta.cache.iceberg.enable",
296+
"meta.cache.iceberg..enable", "meta.cache.iceberg.table."}) {
297+
Assertions.assertThrows(IllegalArgumentException.class, () -> CacheSpec.checkSubmittedEngineProperties(
298+
Collections.singletonMap(key, "1"), "iceberg"), key);
299+
}
300+
IllegalArgumentException enable = Assertions.assertThrows(IllegalArgumentException.class,
301+
() -> CacheSpec.checkSubmittedEngineProperties(
302+
Collections.singletonMap("meta.cache.iceberg.table.enable", "1"), "iceberg"));
303+
Assertions.assertEquals("The parameter meta.cache.iceberg.table.enable is wrong, value is 1",
304+
enable.getMessage());
305+
IllegalArgumentException ttl = Assertions.assertThrows(IllegalArgumentException.class,
306+
() -> CacheSpec.checkSubmittedEngineProperties(
307+
Collections.singletonMap("meta.cache.iceberg.table.ttl-second", "-2"), "iceberg"));
308+
Assertions.assertEquals("The parameter meta.cache.iceberg.table.ttl-second is wrong, value is -2",
309+
ttl.getMessage());
310+
Assertions.assertThrows(IllegalArgumentException.class, () -> CacheSpec.checkSubmittedEngineProperties(
311+
Collections.singletonMap("meta.cache.iceberg.manifest.capacity", "-1"), "iceberg"));
312+
Assertions.assertThrows(IllegalArgumentException.class, () -> CacheSpec.checkSubmittedEngineProperties(
313+
Collections.singletonMap("meta.cache.iceberg.manifest.max-weight", "0"), "iceberg"));
314+
315+
// Entries are not checked: a newer connector may read one this version does not know. Other engines'
316+
// keys, the catalog weight and plain properties belong to other validators.
317+
Map<String, String> accepted = new HashMap<>();
318+
accepted.put("meta.cache.iceberg.schema.ttl-second", "0");
319+
accepted.put("meta.cache.iceberg.future_entry.enable", "TRUE");
320+
accepted.put("meta.cache.iceberg.table.ttl-second", "-1");
321+
accepted.put("meta.cache.iceberg.table.capacity", "0");
322+
accepted.put("meta.cache.iceberg.table.max-weight", "64MB");
323+
accepted.put("meta.cache.paimon.table.enable", "1");
324+
accepted.put("meta.cache.max-weight", "1GB");
325+
accepted.put("comment", "x");
326+
CacheSpec.checkSubmittedEngineProperties(accepted, "iceberg");
327+
}
328+
329+
@Test
330+
public void checkWeightPropertiesValidatesOnlyTheSubmittedEngineKeys() {
331+
Map<String, String> stored = new HashMap<>();
332+
stored.put("meta.cache.iceberg.table.max-weigth", "16MB");
333+
stored.put("meta.cache.iceberg.table.enable", "1");
334+
Assertions.assertDoesNotThrow(() -> CacheSpec.checkWeightProperties(
335+
stored, Collections.singletonMap("comment", "updated"), "iceberg", "table"));
336+
Assertions.assertThrows(IllegalArgumentException.class, () -> CacheSpec.checkWeightProperties(
337+
stored, Collections.singletonMap("meta.cache.iceberg.table.enable", "yes"), "iceberg", "table"));
338+
}
339+
340+
@Test
341+
public void submittedWeightsMayNotExceedTheGlobalLimit() {
342+
OptionalLong global = OptionalLong.of(1024L * 1024 * 1024);
343+
Map<String, String> catalogAbove = Collections.singletonMap("meta.cache.max-weight", "2GB");
344+
IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
345+
() -> CacheSpec.checkWeightProperties(catalogAbove, catalogAbove, global, "iceberg", "table"));
346+
Assertions.assertEquals("meta.cache.max-weight (2147483648 bytes) can not exceed the FE global max weight"
347+
+ " external_meta_cache_max_weight (1073741824 bytes)", e.getMessage());
348+
Map<String, String> catalogAtLimit = Collections.singletonMap("meta.cache.max-weight", "1GB");
349+
CacheSpec.checkWeightProperties(catalogAtLimit, catalogAtLimit, global, "iceberg", "table");
350+
351+
// Without a catalog weight the global limit bounds an entry weight; with one the catalog weight does.
352+
Map<String, String> entryAbove = Collections.singletonMap("meta.cache.iceberg.table.max-weight", "2GB");
353+
Assertions.assertThrows(IllegalArgumentException.class,
354+
() -> CacheSpec.checkWeightProperties(entryAbove, entryAbove, global, "iceberg", "table"));
355+
Map<String, String> limitedCatalog = new HashMap<>(entryAbove);
356+
limitedCatalog.put("meta.cache.max-weight", "1GB");
357+
IllegalArgumentException parent = Assertions.assertThrows(IllegalArgumentException.class,
358+
() -> CacheSpec.checkWeightProperties(limitedCatalog, entryAbove, global, "iceberg", "table"));
359+
Assertions.assertEquals("meta.cache.iceberg.table.max-weight can not exceed meta.cache.max-weight",
360+
parent.getMessage());
361+
362+
// A stored weight is not compared again, e.g. after the limit was lowered, and no limit bounds nothing.
363+
CacheSpec.checkWeightProperties(catalogAbove, Collections.singletonMap("comment", "updated"), global,
364+
"iceberg", "table");
365+
CacheSpec.checkWeightProperties(catalogAbove, catalogAbove, OptionalLong.empty(), "iceberg", "table");
366+
CacheSpec.checkSubmittedWeightsAgainstGlobal(entryAbove, entryAbove, global, "paimon");
367+
}
368+
288369
@Test
289370
public void invalidPersistedEntryWeightFallsBackToUnbounded() {
290371
Map<String, String> properties = new HashMap<>();

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveCatalogPropertiesTest.java‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,24 @@ void icebergSiblingWeightsAreValidatedAtTheHmsCreateTimeDoor() {
245245
HiveTestProperties.mapWith(), unknownEntry));
246246
}
247247

248+
@Test
249+
void submittedMetaCacheKeysOfBothEnginesMustNameAKnownOption() {
250+
HiveConnectorProvider provider = new HiveConnectorProvider();
251+
IllegalArgumentException misspelled = Assertions.assertThrows(IllegalArgumentException.class,
252+
() -> provider.validateProperties(
253+
HiveTestProperties.mapWith("meta.cache.hive.file.max-weigth", "64MB")));
254+
Assertions.assertEquals("Unknown external meta cache property: meta.cache.hive.file.max-weigth",
255+
misspelled.getMessage());
256+
Assertions.assertThrows(IllegalArgumentException.class,
257+
() -> provider.validateProperties(HiveTestProperties.mapWith("meta.cache.hive.file.enable", "1")));
258+
Assertions.assertThrows(IllegalArgumentException.class, () -> provider.validateProperties(
259+
HiveTestProperties.mapWith("meta.cache.iceberg.table.ttl-second", "-2")));
260+
// A key stored by an earlier statement is not validated again by an unrelated ALTER.
261+
Assertions.assertDoesNotThrow(() -> provider.validatePropertiesForUpdate(
262+
HiveTestProperties.mapWith("meta.cache.hive.file.max-weigth", "64MB"),
263+
java.util.Collections.singletonMap("comment", "updated")));
264+
}
265+
248266
@Test
249267
void validCatalogPassesBothDoors() {
250268
Assertions.assertDoesNotThrow(() -> HiveTestProperties.minimal().checkCreateTimeOnlyRules());

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorValidatePropertiesTest.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,17 @@ public void metaCacheKnobsRejectedThroughProvider() {
130130
"meta.cache.iceberg.table.enable", "maybe")).contains("is wrong"));
131131
}
132132

133+
@Test
134+
public void misspelledMetaCacheOptionRejectedThroughProvider() {
135+
// The option must be known; an unknown entry stays accepted because a newer connector may read it.
136+
Assertions.assertEquals(
137+
"Unknown external meta cache property: meta.cache.iceberg.table.max-weigth",
138+
rejectMessage(props("iceberg.catalog.type", "hms", "hive.metastore.uris", "thrift://h",
139+
"meta.cache.iceberg.table.max-weigth", "16MB")));
140+
PROVIDER.validateProperties(props("iceberg.catalog.type", "hms", "hive.metastore.uris", "thrift://h",
141+
"meta.cache.iceberg.schema.ttl-second", "0"));
142+
}
143+
133144
@Test
134145
public void validMetaCacheKnobsAcceptedThroughProvider() {
135146
// ttl-second=-1 (no-expiration sentinel, min is -1), 0 (disable), capacity=0, boolean enable all pass.

‎fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,21 @@ public void weightValidationDistinguishesSubmittedAndPersistedUnknownEntries() {
9191
current, Map.of("meta.cache.max-weight", "1MB", "meta.cache.max_compute.partition.max-weight", "2MB")));
9292
}
9393

94+
@Test
95+
public void submittedMetaCacheKeysMustNameAKnownOption() {
96+
MaxComputeConnectorProvider provider = new MaxComputeConnectorProvider();
97+
Map<String, String> current = MCTestProperties.minimalMap();
98+
IllegalArgumentException misspelled = Assertions.assertThrows(IllegalArgumentException.class,
99+
() -> provider.validatePropertiesForUpdate(
100+
current, Map.of("meta.cache.max_compute.partition.max-weigth", "1MB")));
101+
Assertions.assertEquals("Unknown external meta cache property: meta.cache.max_compute.partition.max-weigth",
102+
misspelled.getMessage());
103+
Assertions.assertThrows(IllegalArgumentException.class, () -> provider.validatePropertiesForUpdate(
104+
current, Map.of("meta.cache.max_compute.partition.enable", "1")));
105+
current.put("meta.cache.max_compute.partition.max-weigth", "1MB");
106+
Assertions.assertDoesNotThrow(() -> provider.validatePropertiesForUpdate(current, Map.of("comment", "new")));
107+
}
108+
94109
@Test
95110
public void managedCacheRetainsOnlySpecsAndHonorsBothLimits() throws Exception {
96111
Odps odps = new Odps(new AliyunAccount("ak", "sk"));

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,21 @@ public void rejectsMalformedMetaCacheKnob() {
9898
"meta.cache.paimon.table.enable", "maybe")));
9999
}
100100

101+
@Test
102+
public void rejectsMisspelledMetaCacheOption() {
103+
IllegalArgumentException misspelled = Assertions.assertThrows(IllegalArgumentException.class,
104+
() -> validate(props(
105+
"paimon.catalog.type", "filesystem", "warehouse", "/wh",
106+
"meta.cache.paimon.partition_view.max-weigth", "16MB")));
107+
Assertions.assertEquals(
108+
"Unknown external meta cache property: meta.cache.paimon.partition_view.max-weigth",
109+
misspelled.getMessage());
110+
Assertions.assertThrows(IllegalArgumentException.class,
111+
() -> validate(props(
112+
"paimon.catalog.type", "filesystem", "warehouse", "/wh",
113+
"meta.cache.paimon.partition_view.enable", "1")));
114+
}
115+
101116
@Test
102117
public void acceptsValidMetaCacheKnobs() {
103118
// Valid values must pass: ttl-second=-1 is the "no expiration" sentinel (min is -1), 0 disables,

0 commit comments

Comments
 (0)