You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 722469c
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](fe) Validate submitted meta.cache keys of the catalog's own engines
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary: branch-4.x (#66717) validates every meta.cache.* catalog
property against a whitelist at CREATE / ALTER CATALOG. Master (#67726)
rejects only an unknown .max-weight entry that the statement submits and
checks the values of a few keys, so a misspelled option
(meta.cache.hive.file.max-weigth), enable=1 (read by Boolean.parseBoolean
as false, which silently turns the cache off) and ttl-second=-2 are all
accepted. Master also accepts a catalog quota above the FE global quota
(external_meta_cache_max_weight) and only clamps it at run time, so SHOW
CREATE CATALOG promises more memory than the caches get.
Every key that a CREATE or ALTER CATALOG submits under meta.cache.<engine>.
for an engine the catalog owns must now name a known option (enable,
ttl-second, capacity, max-weight) with a value of its type: enable is true
or false, ttl-second >= -1, capacity >= 0 and max-weight a positive size.
The check sits in CacheSpec.checkWeightProperties, which the hive (with its
iceberg sibling), iceberg, paimon, max_compute and adbc connectors already
call with the submitted properties, and in a new ExternalCatalog hook that
CatalogMgr calls on CREATE and ALTER for the namespace fe-core owns
(default, or doris for a remote Doris catalog); replay never runs it.
Unknown entries stay accepted, since a newer connector may read an entry
this version does not know, and keys an earlier statement stored are not
validated again, so a stored key never blocks an unrelated ALTER. A
submitted catalog quota, and a submitted entry quota of a catalog without
one, may not exceed the FE global quota of the FE running the statement.
### Release note
CREATE / ALTER CATALOG now rejects a meta.cache.<engine>.<entry>.<option>
property whose option is not enable, ttl-second, capacity or max-weight or
whose value has the wrong type, and a meta.cache.max-weight (or an entry
max-weight of a catalog without one) above external_meta_cache_max_weight.
### Check List (For Author)
- Test: Unit Test, Regression test
- CacheSpecTest; HiveCatalogPropertiesTest, IcebergConnectorValidatePropertiesTest, PaimonConnectorValidatePropertiesTest, MaxComputePartitionCacheTest, AdbcCatalogPropertiesTest; ExternalCatalogMetaCacheWeightTest
- external_table_p0/test_catalog_ddl
- Behavior changed: Yes. DDL that master accepted is now rejected: a misspelled option, a value of the wrong type, and a catalog or entry quota above the FE global quota. Compared with branch-4.x: unknown entries and engines a catalog does not route stay accepted; meta.cache.hive.partition_values.*, meta.cache.iceberg.snapshot.* and meta.cache.paimon.{table,snapshot}.max-weight are not supported on master (a submitted max-weight of them is rejected as an unknown weight entry, their other options and stored values are ignored); an entry max-weight of 0 is rejected instead of disabling the entry.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-adbc/src/test/java/org/apache/doris/connector/adbc/AdbcCatalogPropertiesTest.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveCatalogPropertiesTest.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorValidatePropertiesTest.java
+11Lines changed: 11 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -130,6 +130,17 @@ public void metaCacheKnobsRejectedThroughProvider() {
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java
+15Lines changed: 15 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -91,6 +91,21 @@ public void weightValidationDistinguishesSubmittedAndPersistedUnknownEntries() {
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java
+15Lines changed: 15 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -98,6 +98,21 @@ public void rejectsMalformedMetaCacheKnob() {
0 commit comments