Skip to content

Commit 2358e5e

Browse files
committed
[refactor](lance) Remove the durable Lance index job framework (#66497)
Per the updated direction in #66497 (#66497 (comment)), Lance index mutations move to synchronous SQL with bounded worker execution modeled on Iceberg rewrite_data_files, and a durable asynchronous job framework is no longer part of the initial implementation. This removes it outright rather than leaving it disabled behind a flag or maintaining two execution paths. Removed: - the durable Lance index job state machine, manager, persistent same-name fences, unresolved-job quotas, and job replay infrastructure introduced by #67235 - the job-specific parts of #67630: durable job creation and JobId result sets, SHOW LANCE INDEX JOB(S) with its grammar and plan wiring, the catalog DDL guards tied to unresolved jobs, and the Flight SQL JobId result-schema special case for Lance index DDL - the three unresolved-job quota configuration keys, the job-specific tests, and the now-unreferenced job-locator resolver pair and dataset-locator helper on the Lance catalog and client - stale comments in the retained validation files that still described the removed framework Kept and adapted for the synchronous path: - the privilege, schema, parameter, snapshot, and IF-condition validation (the LanceIndexAdmission preflight, the static validator, the schema-contract builder, and the target-aware catalog guard); a mutation that passes the whole preflight now ends in the shared not-supported rejection, while an IF preflight no-op completes as a plain success - read-only SHOW INDEX, the lance_index_entries TVF, and the statically validated DDL surface; enable_lance_index_mutation remains the gate - the schema-contract and name-normalization helpers, relocated out of the removed job package; the replay-path target guard keeps slimmed direct coverage Compatibility: enable_lance_index_mutation has defaulted to false since #67630 merged and no released build exposes it enabled, so no deployment can hold persisted job records. The OP_LANCE_INDEX_JOB_UPSERT (500) code is kept as a tombstone that is never reused: journals from builds that manually enabled the gate stay readable through a minimal consume-and-discard branch (the payload was Text-delimited JSON), and the image chunk the old manager wrote is skipped through the existing deprecated-module list, so metadata from previous builds still loads.
1 parent 8cd9bf9 commit 2358e5e

62 files changed

Lines changed: 346 additions & 8865 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 6 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -4237,33 +4237,14 @@ public void handle(Field field, String value) throws Exception {
42374237
public static long lance_catalog_arrow_memory_limit_bytes = 256L * 1024 * 1024;
42384238

42394239
@ConfField(mutable = true, masterOnly = true, varType = VariableAnnotation.EXPERIMENTAL, description = {
4240-
"是否启用 Lance 外表索引变更(CREATE/CREATE OR REPLACE/DROP INDEX)的 admission。默认关闭;"
4241-
+ "启用前需确认未决 job 配额均为正值。注意:在 dispatch(后续版本)与 FORCE_RELEASE(后续版本)就绪前"
4242-
+ "开启本开关会产生不可回收的 PENDING job 并冻结对应 catalog 的身份属性变更与 DROP CATALOG。",
4243-
"Enable admission of Lance index mutations (CREATE/CREATE OR REPLACE/DROP INDEX). "
4244-
+ "Disabled by default; unresolved-job quotas must be positive before enabling. "
4245-
+ "WARNING: enabling before dispatch and FORCE_RELEASE land in a later release creates "
4246-
+ "PENDING jobs that cannot be resolved and freezes catalog identity changes and DROP CATALOG."})
4240+
"是否启用 Lance 外表索引变更(CREATE/CREATE OR REPLACE/DROP INDEX)的权威校验路径。默认关闭;"
4241+
+ "在同步执行路径就绪前,开启本开关只会让通过校验的语句收到暂不支持的错误。",
4242+
"Enable the authoritative validation path of Lance index mutations "
4243+
+ "(CREATE/CREATE OR REPLACE/DROP INDEX). Disabled by default; until the synchronous "
4244+
+ "execution path lands, enabling it only makes validated statements fail with the "
4245+
+ "not-supported error."})
42474246
public static boolean enable_lance_index_mutation = false;
42484247

4249-
@ConfField(mutable = true, masterOnly = true,
4250-
callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
4251-
description = {"单个 Lance 数据表(locator 身份)允许的最大未决索引 job 数。",
4252-
"Max unresolved Lance index jobs per table (locator identity)."})
4253-
public static long lance_index_job_max_unresolved_per_table = 8;
4254-
4255-
@ConfField(mutable = true, masterOnly = true,
4256-
callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
4257-
description = {"单个 Lance catalog 允许的最大未决索引 job 数。",
4258-
"Max unresolved Lance index jobs per catalog."})
4259-
public static long lance_index_job_max_unresolved_per_catalog = 64;
4260-
4261-
@ConfField(mutable = true, masterOnly = true,
4262-
callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
4263-
description = {"全部 catalog 合计允许的最大未决 Lance 索引 job 数。",
4264-
"Max unresolved Lance index jobs across all catalogs (global)."})
4265-
public static long lance_index_job_max_unresolved_global = 256;
4266-
42674248
@ConfField(mutable = true, masterOnly = true,
42684249
callback = LanceIndexConfigValidator.PositiveIntConfigHandler.class,
42694250
description = {"Lance IVF_PQ 索引 num_partitions 的静态上限。",

‎fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java‎

Lines changed: 5 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -20,40 +20,19 @@
2020
import java.lang.reflect.Field;
2121

2222
/**
23-
* Validators for the Lance index admission configuration items in {@link Config}. The
24-
* unresolved-job quotas and the static bounds must stay positive (section 9.7 enabling
25-
* precondition), so every handler rejects a zero, negative, or unparsable value before
26-
* assigning. The handlers extend {@link ConfigBase.DefaultConfHandler} and delegate to
23+
* Validators for the Lance index configuration items in {@link Config}. The static bounds
24+
* must stay positive, so the handler rejects a zero, negative, or unparsable value before
25+
* assigning. The handler extends {@link ConfigBase.DefaultConfHandler} and delegates to
2726
* {@code super.handle} after validation: a bare {@code ConfHandler} implementation would
2827
* validate without ever assigning, letting ADMIN SET FRONTEND CONFIG pass with no effect.
2928
* Note the callback only runs on the ADMIN SET path; values loaded from fe.conf bypass it
30-
* ({@code ConfigBase.setFields} assigns directly), so the admission path re-asserts the
31-
* positive invariant where the values are consumed.
29+
* ({@code ConfigBase.setFields} assigns directly), so the consuming validation path
30+
* re-asserts the positive invariant where the values are used.
3231
*/
3332
public final class LanceIndexConfigValidator {
3433
private LanceIndexConfigValidator() {
3534
}
3635

37-
/**
38-
* Accepts only a positive long: validates first, then assigns via the default handler.
39-
*/
40-
public static class PositiveLongConfigHandler extends ConfigBase.DefaultConfHandler {
41-
@Override
42-
public void handle(Field field, String confVal) throws Exception {
43-
String trimmedVal = confVal == null ? "" : confVal.trim();
44-
final long value;
45-
try {
46-
value = Long.parseLong(trimmedVal);
47-
} catch (NumberFormatException e) {
48-
throw new ConfigException(field.getName() + " must be a positive long, but got: " + confVal, e);
49-
}
50-
if (value <= 0) {
51-
throw new ConfigException(field.getName() + " must be a positive long, but got: " + confVal);
52-
}
53-
super.handle(field, trimmedVal);
54-
}
55-
}
56-
5736
/**
5837
* Accepts only a positive int: validates first, then assigns via the default handler.
5938
*/

‎fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java‎

Lines changed: 2 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,8 @@
2525
import java.nio.file.Path;
2626

2727
/**
28-
* Coverage for the Lance index admission configuration items in {@link Config} and the
29-
* positive-value validators in {@link LanceIndexConfigValidator}: reviewed defaults, the
28+
* Coverage for the Lance index configuration items in {@link Config} and the positive-value
29+
* validator in {@link LanceIndexConfigValidator}: reviewed defaults, the
3030
* {@link ConfigBase.ConfField} wiring (mutable/masterOnly/callback/varType), rejection of
3131
* zero, negative and non-numeric values, and that an accepted value is really assigned to
3232
* the field (a bare validating handler that never assigns would let ADMIN SET pass without
@@ -37,21 +37,12 @@ public class LanceIndexConfigValidatorTest {
3737
@Test
3838
public void testReviewedDefaults() {
3939
Assertions.assertFalse(Config.enable_lance_index_mutation);
40-
Assertions.assertEquals(8L, Config.lance_index_job_max_unresolved_per_table);
41-
Assertions.assertEquals(64L, Config.lance_index_job_max_unresolved_per_catalog);
42-
Assertions.assertEquals(256L, Config.lance_index_job_max_unresolved_global);
4340
Assertions.assertEquals(4096, Config.lance_index_max_num_partitions);
4441
Assertions.assertEquals(256, Config.lance_index_max_num_sub_vectors);
4542
}
4643

4744
@Test
4845
public void testConfFieldWiring() throws Exception {
49-
assertCallbackWiring("lance_index_job_max_unresolved_per_table", true,
50-
LanceIndexConfigValidator.PositiveLongConfigHandler.class);
51-
assertCallbackWiring("lance_index_job_max_unresolved_per_catalog", true,
52-
LanceIndexConfigValidator.PositiveLongConfigHandler.class);
53-
assertCallbackWiring("lance_index_job_max_unresolved_global", true,
54-
LanceIndexConfigValidator.PositiveLongConfigHandler.class);
5546
assertCallbackWiring("lance_index_max_num_partitions", true,
5647
LanceIndexConfigValidator.PositiveIntConfigHandler.class);
5748
assertCallbackWiring("lance_index_max_num_sub_vectors", true,
@@ -74,48 +65,6 @@ private static void assertCallbackWiring(String fieldName, boolean masterOnly, C
7465
Assertions.assertEquals(callback, anno.callback(), fieldName);
7566
}
7667

77-
@Test
78-
public void testPositiveLongHandlerAssignsAcceptedValue() throws Exception {
79-
assertLongAssigns("lance_index_job_max_unresolved_per_table");
80-
assertLongAssigns("lance_index_job_max_unresolved_per_catalog");
81-
assertLongAssigns("lance_index_job_max_unresolved_global");
82-
}
83-
84-
private static void assertLongAssigns(String fieldName) throws Exception {
85-
Field field = Config.class.getField(fieldName);
86-
long original = field.getLong(null);
87-
try {
88-
new LanceIndexConfigValidator.PositiveLongConfigHandler().handle(field, " 12345 ");
89-
Assertions.assertEquals(12345L, field.getLong(null), fieldName);
90-
} finally {
91-
field.setLong(null, original);
92-
}
93-
}
94-
95-
@Test
96-
public void testPositiveLongHandlerRejectsInvalidValues() throws Exception {
97-
assertLongRejected("lance_index_job_max_unresolved_per_table", "0");
98-
assertLongRejected("lance_index_job_max_unresolved_per_catalog", "0");
99-
assertLongRejected("lance_index_job_max_unresolved_global", "0");
100-
assertLongRejected("lance_index_job_max_unresolved_per_table", "-8");
101-
assertLongRejected("lance_index_job_max_unresolved_per_table", "not-a-number");
102-
assertLongRejected("lance_index_job_max_unresolved_per_table", "");
103-
}
104-
105-
private static void assertLongRejected(String fieldName, String value) throws Exception {
106-
Field field = Config.class.getField(fieldName);
107-
long original = field.getLong(null);
108-
try {
109-
ConfigException e = Assertions.assertThrows(ConfigException.class,
110-
() -> new LanceIndexConfigValidator.PositiveLongConfigHandler().handle(field, value),
111-
fieldName + " <- \"" + value + "\"");
112-
Assertions.assertTrue(e.getMessage().contains(fieldName), e.getMessage());
113-
Assertions.assertEquals(original, field.getLong(null), fieldName + " must remain unchanged");
114-
} finally {
115-
field.setLong(null, original);
116-
}
117-
}
118-
11968
@Test
12069
public void testPositiveIntHandlerAssignsAcceptedValue() throws Exception {
12170
assertIntAssigns("lance_index_max_num_partitions");
@@ -167,18 +116,9 @@ public void testSetMutableConfigPath() throws Exception {
167116
tempFile.toFile().deleteOnExit();
168117
config.init(tempFile.toAbsolutePath().toString());
169118

170-
long originalQuota = Config.lance_index_job_max_unresolved_per_catalog;
171119
int originalBound = Config.lance_index_max_num_partitions;
172120
boolean originalGate = Config.enable_lance_index_mutation;
173121
try {
174-
ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "96");
175-
Assertions.assertEquals(96L, Config.lance_index_job_max_unresolved_per_catalog);
176-
Assertions.assertThrows(ConfigException.class,
177-
() -> ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "0"));
178-
Assertions.assertThrows(ConfigException.class,
179-
() -> ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "xyz"));
180-
Assertions.assertEquals(96L, Config.lance_index_job_max_unresolved_per_catalog);
181-
182122
ConfigBase.setMutableConfig("lance_index_max_num_partitions", "8192");
183123
Assertions.assertEquals(8192, Config.lance_index_max_num_partitions);
184124
Assertions.assertThrows(ConfigException.class,
@@ -188,7 +128,6 @@ public void testSetMutableConfigPath() throws Exception {
188128
ConfigBase.setMutableConfig("enable_lance_index_mutation", "true");
189129
Assertions.assertTrue(Config.enable_lance_index_mutation);
190130
} finally {
191-
Config.lance_index_job_max_unresolved_per_catalog = originalQuota;
192131
Config.lance_index_max_num_partitions = originalBound;
193132
Config.enable_lance_index_mutation = originalGate;
194133
}

‎fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -333,7 +333,6 @@ KEY: 'KEY';
333333
KEYS: 'KEYS';
334334
KILL: 'KILL';
335335
LABEL: 'LABEL';
336-
LANCE: 'LANCE';
337336
LARGEINT: 'LARGEINT';
338337
LAYOUT: 'LAYOUT';
339338
LAST: 'LAST';

‎fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -497,8 +497,6 @@ supportedShowStatement
497497
| SHOW WARM UP JOB wildWhere? #showWarmUpJob
498498
| SHOW PYTHON VERSIONS #showPythonVersions
499499
| SHOW PYTHON PACKAGES IN STRING_LITERAL #showPythonPackages
500-
| SHOW LANCE INDEX JOBS ((FROM | IN) db=multipartIdentifier)? (WHERE expression)? #showLanceIndexJobs
501-
| SHOW LANCE INDEX JOB jobId=INTEGER_VALUE #showLanceIndexJob
502500
;
503501

504502
supportedLoadStatement
@@ -2204,7 +2202,6 @@ nonReserved
22042202
| JSON
22052203
| JSONB
22062204
| LABEL
2207-
| LANCE
22082205
| LAST
22092206
| LDAP
22102207
| LDAP_ADMIN_PASSWORD

‎fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,6 @@
111111
import org.apache.doris.datasource.iceberg.IcebergExternalTable;
112112
import org.apache.doris.datasource.iceberg.IcebergSysExternalTable;
113113
import org.apache.doris.datasource.jdbc.JdbcExternalTable;
114-
import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
115114
import org.apache.doris.datasource.paimon.PaimonExternalTable;
116115
import org.apache.doris.datasource.paimon.PaimonSysExternalTable;
117116
import org.apache.doris.deploy.DeployManager;
@@ -572,8 +571,6 @@ public class Env {
572571

573572
private InsertOverwriteManager insertOverwriteManager;
574573

575-
private LanceIndexJobManager lanceIndexJobManager;
576-
577574
private DNSCache dnsCache;
578575

579576
private final NereidsSqlCacheManager sqlCacheManager;
@@ -859,7 +856,6 @@ public Env(boolean isCheckpointCatalog) {
859856
this.mtmvService = new MTMVService();
860857
this.eventProcessor = new EventProcessor(mtmvService);
861858
this.insertOverwriteManager = new InsertOverwriteManager();
862-
this.lanceIndexJobManager = new LanceIndexJobManager();
863859
this.dnsCache = new DNSCache();
864860
this.sqlCacheManager = new NereidsSqlCacheManager();
865861
this.sortedPartitionsCacheManager = new NereidsSortedPartitionsCacheManager();
@@ -985,10 +981,6 @@ public InsertOverwriteManager getInsertOverwriteManager() {
985981
return insertOverwriteManager;
986982
}
987983

988-
public LanceIndexJobManager getLanceIndexJobManager() {
989-
return lanceIndexJobManager;
990-
}
991-
992984
public TabletScheduler getTabletScheduler() {
993985
return tabletScheduler;
994986
}
@@ -1830,11 +1822,6 @@ private void transferToMaster() {
18301822

18311823
insertOverwriteManager.allTaskFail();
18321824

1833-
// A durable RUNNING Lance index job at this point may have lost its result with the
1834-
// old master: sweep it to UNKNOWN (and refresh RUNNING back to REQUIRED) before any
1835-
// master-only dispatcher could start.
1836-
lanceIndexJobManager.onTransferToMaster();
1837-
18381825
toMasterProgress = "start daemon threads";
18391826

18401827
// coz current fe was not master fe and didn't get all fes' alive session report before, which cause
@@ -2685,18 +2672,6 @@ public long saveDictionaryManager(CountingDataOutputStream out, long checksum) t
26852672
return checksum;
26862673
}
26872674

2688-
public long loadLanceIndexJobManager(DataInputStream in, long checksum) throws IOException {
2689-
this.lanceIndexJobManager = LanceIndexJobManager.read(in);
2690-
LOG.info("finished replay lance index job manager from image");
2691-
return checksum;
2692-
}
2693-
2694-
public long saveLanceIndexJobManager(CountingDataOutputStream out, long checksum) throws IOException {
2695-
this.lanceIndexJobManager.write(out);
2696-
LOG.info("finished save lance index job manager to image");
2697-
return checksum;
2698-
}
2699-
27002675
// Only called by checkpoint thread
27012676
// return the latest image file's absolute path
27022677
public String saveImage() throws IOException {

‎fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1242,7 +1242,8 @@ public enum ErrorCode {
12421242
ERR_LANCE_INDEX_MUTATION_DISABLED(5102, new byte[]{'4', '2', '0', '0', '0'},
12431243
"%s is disabled for Lance catalog tables (enable_lance_index_mutation = false)"),
12441244

1245-
ERR_LANCE_INDEX_JOB_NOT_FOUND(5103, new byte[]{'4', '2', '0', '0', '0'}, "Lance index job not found: %s"),
1245+
// 5103 (ERR_LANCE_INDEX_JOB_NOT_FOUND) was removed with the durable Lance index job
1246+
// framework; the number stays unused so old client-visible codes keep their meaning.
12461247

12471248
ERR_NOT_CLOUD_MODE(6000, new byte[]{'4', '2', '0', '0', '0'},
12481249
"Command only support in cloud mode.");

0 commit comments

Comments
 (0)