Skip to content

Commit a53722f

Browse files
committed
[fix](lance) Revalidate SHOW targets against the current dataset locator
Tighten the SHOW LANCE INDEX JOBS / SHOW LANCE INDEX JOB authorization boundary in three ways: - Exception-safe target resolution. getDbNullable/getTableNullable can throw for external catalogs (expired credentials, remote provider faults). Any resolution failure is now handled exactly like an unresolvable target under the existing orphan semantics: the detail command answers non-ADMIN callers with the fixed not-found response instead of leaking the provider error, and one bad target no longer aborts the whole listing. The provider error is never surfaced. - Durable-locator revalidation. After a job reaches a terminal state and releases its guard, a legitimate ALTER can repoint the same db.table names at a different dataset. LanceExternalCatalog gains resolveCurrentIndexJobLocator(db, table), which reuses the reader's table-access resolution plus LanceIndexDatasetLocator.normalize and returns null on any failure. Both SHOW commands now treat a target as resolvable only when the names resolve AND the current locator is non-null AND equals the job's persisted normalized locator; otherwise the row follows the orphan (ADMIN-only) rule, so a user holding SHOW on the repointed table cannot read the old dataset's history. The listing resolves each unique (catalogId, db, table) locator once per run through a dedupe cache (failed resolutions are cached too). Non- Lance catalog instances keep the name-resolution-only rule. - FROM db-name normalization. The FROM database filter compared the raw requested string, so under lower_case_database_names = 1/2 a FROM DB1 filter missed jobs persisted under the resolved name db1. The requested name is now resolved through the target catalog (explicit or current) and replaced by the database's full name before the exact comparison; an unresolvable name or a failed resolution keeps the raw string and never fails the statement. Tests: provider failures during resolution are orphan-like for the list (omitted for non-ADMIN, visible to ADMIN) and byte-identical to the missing response for the detail command; locator match keeps ordinary table-level SHOW, while a repointed or unresolvable locator hides the row/returns not-found for non-ADMIN and stays visible to ADMIN; one listing resolves a shared target's locator once; FROM DB1 normalizes to db1 through the catalog and falls back to the raw string when the catalog cannot resolve it.
1 parent d192e4d commit a53722f

5 files changed

Lines changed: 490 additions & 22 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.datasource.ExternalCatalog;
2626
import org.apache.doris.datasource.InitCatalogLog;
2727
import org.apache.doris.datasource.SessionContext;
28+
import org.apache.doris.datasource.lance.job.LanceIndexDatasetLocator;
2829
import org.apache.doris.datasource.property.metastore.AbstractLanceProperties;
2930
import org.apache.doris.datasource.property.metastore.LanceFileSystemMetastoreProperties;
3031
import org.apache.doris.datasource.property.metastore.LanceRestMetastoreProperties;
@@ -105,6 +106,33 @@ public void advanceIndexTargetVersion() {
105106
indexTargetVersion++;
106107
}
107108

109+
/**
110+
* Resolves the dataset the given (db, table) names currently point at, through the
111+
* same table-access resolution the readers use, and returns its durable locator form
112+
* ({@link LanceIndexDatasetLocator#normalize}).
113+
*
114+
* <p>This exists for SHOW-authorization revalidation of persisted Lance index jobs:
115+
* once a job reaches a terminal state and releases its guard, a legitimate catalog
116+
* ALTER can repoint the same db.table names at a different dataset, and the job must
117+
* stop being readable through table-level SHOW on the new target. Any failure - the
118+
* catalog is not initialized, the provider is unreachable, credentials expired, or
119+
* the names no longer resolve - yields {@code null}; callers must treat null as
120+
* "not resolved" (the orphan, ADMIN-only visibility rule), never as an authorization
121+
* grant.
122+
*
123+
* @return the normalized locator of the dataset the names currently point at, or
124+
* null when it cannot be resolved
125+
*/
126+
public String resolveCurrentIndexJobLocator(String dbName, String tableName) {
127+
try {
128+
makeSureInitialized();
129+
ResolvedTableAccess tableAccess = resolveTableAccess(dbName, tableName);
130+
return LanceIndexDatasetLocator.normalize(tableAccess.datasetUri);
131+
} catch (Exception e) {
132+
return null;
133+
}
134+
}
135+
108136
public LanceExternalCatalog(long catalogId, String name, String resource, Map<String, String> props,
109137
String comment) {
110138
super(catalogId, name, InitCatalogLog.Type.LANCE, comment);

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommand.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,10 @@
4747
*
4848
* <p>Non-disclosing job detail (v5.1 section 8): the record is loaded first and authorized
4949
* against its persisted target before any field is returned. Orphan and half-orphan targets
50-
* (catalog gone, or persisted db/table no longer resolvable) require global ADMIN; a
51-
* resolvable target requires table-level SHOW. A missing job and an unauthorized job share
50+
* (catalog gone, persisted db/table no longer resolvable, the resolution failing outright
51+
* because the provider is unreachable, or the table repointed at a different dataset
52+
* locator) require global ADMIN; a resolvable, still-matching target requires table-level
53+
* SHOW. A missing job and an unauthorized job share
5254
* the same fixed ERR_LANCE_INDEX_JOB_NOT_FOUND response that names only the job id, so the
5355
* existence and target of a job are never disclosed. The job locator, provider, normalized
5456
* names, propertiesJson and schema contract contents are never shown.

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommand.java‎

Lines changed: 99 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.apache.doris.common.AnalysisException;
2727
import org.apache.doris.common.util.TimeUtils;
2828
import org.apache.doris.datasource.CatalogIf;
29+
import org.apache.doris.datasource.lance.LanceExternalCatalog;
2930
import org.apache.doris.datasource.lance.job.LanceIndexJob;
3031
import org.apache.doris.datasource.lance.job.LanceIndexJobMutationState;
3132
import org.apache.doris.mysql.privilege.PrivPredicate;
@@ -45,20 +46,24 @@
4546
import com.google.common.collect.ImmutableList;
4647

4748
import java.util.ArrayList;
49+
import java.util.HashMap;
4850
import java.util.HashSet;
4951
import java.util.List;
5052
import java.util.Locale;
53+
import java.util.Map;
5154
import java.util.Set;
5255

5356
/**
5457
* SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "tbl" [AND State = "PENDING"]].
5558
*
5659
* <p>Lists the durable Lance index job records held by the master. Rows whose persisted
57-
* target no longer resolves (the catalog is gone, or the catalog is there but the db or
58-
* table no longer resolves) are visible to global ADMIN only; every other row requires
59-
* table-level SHOW on the persisted (catalog, db, table). Rows that fail the check are
60-
* omitted entirely, so non-ADMIN users see no orphan trace, not even a count. The job
61-
* locator, provider, normalized names, propertiesJson and schema contract are never shown.
60+
* target no longer resolves (the catalog is gone, the catalog is there but the db or
61+
* table no longer resolves, or the resolution itself fails because the provider is
62+
* unreachable), or whose persisted dataset locator no longer matches the catalog's
63+
* current one, are visible to global ADMIN only; every other row requires table-level
64+
* SHOW on the persisted (catalog, db, table). Rows that fail the check are omitted
65+
* entirely, so non-ADMIN users see no orphan trace, not even a count. The job locator,
66+
* provider, normalized names, propertiesJson and schema contract are never shown.
6267
*
6368
* <p>The WHERE clause is deliberately narrowed to EqualTo predicates combined with AND
6469
* over the case-insensitive keys TableName and State (no Like, unlike the SHOW COPY
@@ -126,9 +131,11 @@ private void validate(ConnectContext ctx) throws AnalysisException {
126131
dbName = nameParts.get(0);
127132
CatalogIf currentCatalog = ctx.getCurrentCatalog();
128133
ctlName = currentCatalog == null ? null : currentCatalog.getName();
134+
normalizeDbName(currentCatalog);
129135
} else if (nameParts.size() == 2) {
130136
ctlName = nameParts.get(0);
131137
dbName = nameParts.get(1);
138+
normalizeDbName(Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName));
132139
} else {
133140
throw new AnalysisException(
134141
"Only support SHOW LANCE INDEX JOBS FROM [catalog.]database, but get: " + nameParts);
@@ -137,6 +144,32 @@ private void validate(ConnectContext ctx) throws AnalysisException {
137144
analyzeWhereClause();
138145
}
139146

147+
/**
148+
* Resolves the FROM database name through the target catalog's own naming rules
149+
* before it is compared against the persisted job records: with
150+
* lower_case_database_names = 1/2, {@code FROM DB1} must still match jobs stored
151+
* under the resolved full name {@code db1}. A name the catalog cannot resolve (or a
152+
* resolution that fails outright) keeps the raw requested string, so the filter just
153+
* stays an exact comparison; the resolution itself must never fail the statement.
154+
*/
155+
private void normalizeDbName(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog) {
156+
if (dbName == null || catalog == null) {
157+
return;
158+
}
159+
try {
160+
DatabaseIf<? extends TableIf> db = catalog.getDbNullable(dbName);
161+
if (db == null) {
162+
return;
163+
}
164+
String fullName = db.getFullName();
165+
if (fullName != null && !fullName.isEmpty()) {
166+
dbName = fullName;
167+
}
168+
} catch (Exception e) {
169+
// Keep the raw requested name; the filter remains an exact comparison.
170+
}
171+
}
172+
140173
private void analyzeWhereClause() throws AnalysisException {
141174
if (whereClause == null) {
142175
return;
@@ -191,13 +224,17 @@ private void analyzeSubPredicate(Expression expr) throws AnalysisException {
191224
public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
192225
validate(ctx);
193226
List<List<String>> rows = new ArrayList<>();
227+
// One locator resolution per unique (catalogId, db, table) for the whole listing:
228+
// each resolution is a provider round trip, and failed (null) resolutions are
229+
// cached too so a broken target is probed once, not once per sibling job.
230+
Map<String, String> locatorCache = new HashMap<>();
194231
for (LanceIndexJob job : Env.getCurrentEnv().getLanceIndexJobManager().getAllJobsSnapshot()) {
195232
CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog =
196233
Env.getCurrentEnv().getCatalogMgr().getCatalog(job.getCatalogId());
197234
if (!matchesFilters(catalog, job)) {
198235
continue;
199236
}
200-
if (!isAuthorized(ctx, catalog, job)) {
237+
if (!isAuthorized(ctx, catalog, job, locatorCache)) {
201238
continue;
202239
}
203240
rows.add(renderRow(job, catalog));
@@ -220,14 +257,29 @@ private boolean matchesFilters(CatalogIf<? extends DatabaseIf<? extends TableIf>
220257
}
221258

222259
/**
223-
* Orphan and half-orphan rows (catalog gone, or persisted db/table no longer resolvable)
224-
* are visible to global ADMIN only; every other row needs table-level SHOW on the
225-
* persisted target. The caller omits the row when this returns false, so non-ADMIN users
226-
* see no trace of orphaned jobs, not even a count.
260+
* Orphan and half-orphan rows (catalog gone, persisted db/table no longer resolvable,
261+
* or a resolution that fails outright because the provider is unreachable) are
262+
* visible to global ADMIN only. A Lance catalog target is additionally revalidated
263+
* against the catalog's current durable dataset locator: after a job reaches a
264+
* terminal state and releases its guard, a legitimate ALTER can repoint the same
265+
* db.table names at a different dataset, and SHOW on the new target must not
266+
* disclose the old job. Every other row needs table-level SHOW on the persisted
267+
* target. The caller omits the row when this returns false, so non-ADMIN users see
268+
* no trace of orphaned jobs, not even a count.
227269
*/
228270
static boolean isAuthorized(ConnectContext ctx, CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
229271
LanceIndexJob job) {
230-
if (!targetResolves(catalog, job)) {
272+
return isAuthorized(ctx, catalog, job, null);
273+
}
274+
275+
/**
276+
* Same authorization as {@link #isAuthorized(ConnectContext, CatalogIf, LanceIndexJob)},
277+
* with an optional (catalogId, db, table)-keyed locator cache so a listing resolves
278+
* each unique target's current locator once per run instead of once per job row.
279+
*/
280+
static boolean isAuthorized(ConnectContext ctx, CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
281+
LanceIndexJob job, Map<String, String> locatorCache) {
282+
if (!targetResolves(catalog, job) || !locatorMatches(catalog, job, locatorCache)) {
231283
return Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ctx, PrivPredicate.ADMIN);
232284
}
233285
return Env.getCurrentEnv().getAccessManager().checkTblPriv(ctx, catalog.getName(),
@@ -238,8 +290,42 @@ static boolean targetResolves(CatalogIf<? extends DatabaseIf<? extends TableIf>>
238290
if (catalog == null) {
239291
return false;
240292
}
241-
DatabaseIf<? extends TableIf> db = catalog.getDbNullable(job.getDbName());
242-
return db != null && db.getTableNullable(job.getTableName()) != null;
293+
try {
294+
DatabaseIf<? extends TableIf> db = catalog.getDbNullable(job.getDbName());
295+
return db != null && db.getTableNullable(job.getTableName()) != null;
296+
} catch (Exception e) {
297+
// External metadata resolution can fail outright (credentials expired,
298+
// provider down). That must not leak the provider error or abort the listing:
299+
// a failed resolution is handled exactly like an unresolvable (orphan) target.
300+
return false;
301+
}
302+
}
303+
304+
/**
305+
* Whether the catalog's current durable dataset locator still points at the dataset
306+
* this job was admitted against. Only Lance catalogs carry a durable locator; for
307+
* any other catalog instance (theoretically unreachable, since jobs are only
308+
* admitted for Lance catalogs) the name resolution above is the whole rule. A
309+
* locator that cannot be resolved right now counts as a mismatch, so authorization
310+
* fails closed to the orphan rule instead of granting SHOW through stale names.
311+
*/
312+
private static boolean locatorMatches(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
313+
LanceIndexJob job, Map<String, String> locatorCache) {
314+
if (!(catalog instanceof LanceExternalCatalog)) {
315+
return true;
316+
}
317+
String key = job.getCatalogId() + ":" + job.getDbName() + ":" + job.getTableName();
318+
String currentLocator;
319+
if (locatorCache != null && locatorCache.containsKey(key)) {
320+
currentLocator = locatorCache.get(key);
321+
} else {
322+
currentLocator = ((LanceExternalCatalog) catalog).resolveCurrentIndexJobLocator(
323+
job.getDbName(), job.getTableName());
324+
if (locatorCache != null) {
325+
locatorCache.put(key, currentLocator);
326+
}
327+
}
328+
return currentLocator != null && currentLocator.equals(job.getNormalizedLocator());
243329
}
244330

245331
private static List<String> renderRow(LanceIndexJob job,

0 commit comments

Comments
 (0)