Repository navigation
[fix](arrow-flight) Support table type metadata and filters - #67899
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Exercise SQL-created base tables and views through the Arrow Flight JDBC metadata APIs, covering type enumeration and complete type filters. The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams. ### Release note None ### Check List (For Author) - Test: official regression runner, 1 suite passed against a test FE with real Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests passed. No native BE or data-scan integration was exercised. - Behavior changed: No, test-only addition - Does this need documentation: No
|
I’m syncing this with current master. The conflict is in the schema-helper imports; I’ll keep the newer Arrow type support and rerun the Flight metadata tests before pushing. |
### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Resolve the schema-helper import conflict while retaining master Arrow type support and the table-types metadata fix. ### Release note None; this commit synchronizes the existing PR with master. ### Check List (For Author) - Test: FE Checkstyle passed with zero violations. Official selected FE unit-test command attempted; blocked before execution by the Intel-only grpc-java 1.34.0 generator on this ARM macOS host. - Behavior changed: No additional behavior beyond the existing PR and master. - Does this need documentation: No. Signed-off-by: hutiefang76 <[email protected]>
|
Merged master in 58c16ac and resolved the import conflict, retaining the newer Arrow type support. FE Checkstyle reports zero violations. I attempted the selected tests through run-fe-ut.sh, but the build stops before test execution at the Intel-only grpc-java 1.34.0 generator on this ARM Mac. I have updated the description so the earlier 30-test and SQL/Flight regression results are not mistaken for verification of this merged head. The local toolchain files are not part of the patch. |
Related to apache#67578 (partial A3). Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows. Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends. Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately. - Test: - [x] Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases. - Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering. - Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup. - FE Checkstyle: 0 violations. - No live-cluster or end-to-end ADBC driver test was run. - Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows. - Does this need documentation: No new configuration or API schema. Validation command (JDK 17, matching Thrift 0.24.0 compiler): ```sh bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest ```
### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Exercise SQL-created base tables and views through the Arrow Flight JDBC metadata APIs, covering type enumeration and complete type filters. The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams. ### Release note None ### Check List (For Author) - Test: official regression runner, 1 suite passed against a test FE with real Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests passed. No native BE or data-scan integration was exercised. - Behavior changed: No, test-only addition - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68631 Problem Summary: branch-4.1 apache#68631 (also on branch-4.2) implemented the GetTableTypes data stream and tested it end to end over Flight RPC: a unit test that runs a real FlightServer and FlightSqlClient, and a regression suite that uses a plain Flight SQL client. The tests in this PR call `getStreamTableTypes()` directly with a mocked listener, or go through JDBC `getTableTypes()`, so the GetFlightInfo -> DoGet sequence that a plain Flight SQL client sends was not covered. This commit adds both checks from apache#68631: - `FlightSqlTableTypesTest#tableTypesAreSortedAndStableThroughFlightRpc` calls GetTableTypes three times through a real FlightServer. Each call must return the GetTableTypes schema with one endpoint, and DoGet must return the same sorted types. Only DoGet resolves the session. The 4.x version mocked the ConnectContext and checked that nothing touched it. On master the metadata stream runs as a command of the session (`FlightProtocolAdapter.runCommand`), which reads the context, so the test uses `ConnectContext.forFlight` as the other tests here do. - `test_table_types_over_arrow_flight` sends the same RPCs three times from a plain Flight SQL client, checking the field name, type and nullability, then runs a query on the session afterwards. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - Behavior changed: No - Does this need documentation: No Co-authored-by: Gabriel <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
58c16ac to
2a31c4d
Compare
|
@hutiefang76 I rebased this branch onto the current master and added one commit on top, so it can go through CI and review:
Local results: |
|
run buildall |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Static review of PR 67899 at head 2a31c4d completed in one round and converged. The change implements Arrow Flight SQL GetTableTypes with the authenticated session/command lifecycle already used by the other metadata streams, publishes the three non-null Doris table type values in deterministic order, and applies multi-type GetTables filters after the existing authorization checks. Required-column projection still supplies the mandatory table name/type fields and avoids unrelated status lookups; filtered includeSchema requests describe only the authorized matching tables.
The main checkpoints were all verified: the requested metadata behavior is implemented and covered by the added helper, producer, and SQL regression tests; the diff is focused with no configuration, persistence, thrift/protobuf, or FE-BE protocol changes; session locking, listener error conversion, and Arrow root ownership follow existing paths; no new static-initialization, concurrency, or transaction/data-write risk was found; and the enum-derived advertised types match the values emitted by GetTables across internal and external catalog paths. No additional user focus was provided. The existing inline request for a SQL regression test is already addressed by the new regression suite and is not reposted.
This invocation performed static review only; no builds or tests were run here. No new P0/P1/P2/P3 finding is substantiated, so no inline comments are submitted.
Review converged after 1 round(s).
TPC-H: Total hot run time: 28808 ms |
TPC-DS: Total hot run time: 150855 ms |
ClickBench: Total hot run time: 24.19 s |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
…etTables ### What problem does this PR solve? Issue Number: None Related PR: apache#67899 Problem Summary: GetTables now filters the authorized tables by every requested table type with an exact match against Doris's names (BASE TABLE, VIEW, SYSTEM VIEW). Clients do not send those names. Flight SQL leaves the names to the server but calls TABLE, VIEW and SYSTEM TABLE the commonly supported ones, and that is what clients ask for: the ADBC Flight SQL driver sends "table" (Doris's own ADBC catalog lists tables with getObjects(..., {"table"})), JDBC tools send "TABLE". Before the exact filter, Doris only recognised VIEW and answered every other type with all objects, so these requests worked; with it they match nothing. Reproduction: an ADBC catalog whose source is a Doris Arrow Flight SQL endpoint lists no tables, so SHOW TABLES is empty and every query fails with "Unknown table" (all 18 external_table_p0/adbc suites fail in the External pipeline); JDBC getTables(..., {"TABLE"}) returns no rows and {"TABLE", "VIEW"} returns only views. Fix: map each requested type before matching. The comparison ignores case, TABLE selects BASE TABLE and SYSTEM TABLE selects SYSTEM VIEW; any other name still matches only a type of that name, so UNKNOWN still selects nothing. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - FlightSqlSchemaHelperTableTypesTest: table, TABLE + VIEW, view, SYSTEM TABLE and mixed-case names, with and without schemas - test_table_types_over_arrow_flight: TABLE, table, table + view through Flight SQL JDBC - external_table_p0/adbc suites against a local cluster - Behavior changed: No (restores the listing these clients got before the multi-type filter) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
@hutiefang76 The External pipeline on 2a31c4d failed all 18
Local results on 1258c51: |
|
run buildall |
TPC-H: Total hot run time: 28523 ms |
FE UT Coverage ReportIncrement line coverage |
TPC-DS: Total hot run time: 151889 ms |
ClickBench: Total hot run time: 24.19 s |
|
/review |
There was a problem hiding this comment.
Static review of PR 67899 at head 1258c51 found the GetTableTypes implementation consistent with Doris's advertised BASE TABLE, SYSTEM VIEW, and VIEW mappings. GetTables now requests only name/type status fields, applies case-insensitive aliases and multi-type filtering after catalog, database, and table SHOW authorization, and describes only the authorized matches. The producer reuses the existing session command lock, Flight schema/ticket path, listener error handling, and root allocator cleanup. No separate correctness, authorization, concurrency, lifecycle, protocol, configuration, persistence, data-write, or observability defect was verified.
The existing review thread 3999564249 requested a SQL regression test; the added regression suite addresses that request, so no duplicate blocking comment is carried. The user supplied no additional focus. The new FE and regression tests were inspected but not executed because this contract is static-only.
Verified finding: the raw Flight client in the new regression test authenticates a persistent server-side session but closes only the transport and allocator. Without FlightSqlClient.closeSession, the connection-pool slot and session channel remain until wait_timeout, which can exhaust Flight/user connection quotas or interfere with later suites. Close the session with the bearer token in a finally block before closing the transport. Review convergence was reached in round 2.
Review converged after 2 round(s).
| Integer.parseInt(config.get("extArrowFlightSqlPort"))) | ||
| new RootAllocator().withCloseable { allocator -> | ||
| FlightClient.builder(allocator, location).build().withCloseable { flight -> | ||
| def token = flight.authenticateBasicToken(config.get("extArrowFlightSqlUser"), |
There was a problem hiding this comment.
[P2] Close the raw Flight SQL session before closing the client. authenticateBasicToken opens a Doris session in the shared connection pool, and closing the transport/allocator does not call FlightSqlProducer.closeSession; the session remains registered until CloseSession, KILL CONNECTION, or wait_timeout. This suite therefore leaks one connection slot per run and can exhaust the Flight/user quota or interfere with later suites. Keep the token and call client.closeSession(new CloseSessionRequest(), token) in a finally block before closing flight.
There was a problem hiding this comment.
Fixed in 221a0af. The suite now ends the plain Flight SQL session in a finally block with flight.closeSession(new CloseSessionRequest(), token) before the client and the allocator close, as the other arrow_flight_sql_p0 suites with a plain Flight client do. On a local cluster, the ArrowFlightSQL sessions in SHOW PROCESSLIST grew by two per run of the suite before this change (the framework's Flight SQL JDBC connection and the leaked one) and grow by one after it; the suite passes in both cases.
…types suite ### What problem does this PR solve? Issue Number: None Related PR: apache#67899 Problem Summary: test_table_types_over_arrow_flight calls GetTableTypes through a plain Flight SQL client. authenticateBasicToken opens a Doris session in the Flight connection pool, and the suite then closed only the client and the allocator. Closing the transport does not call CloseSession, so each run left one session of the test user registered until wait_timeout. The other arrow_flight_sql_p0 suites that use a plain Flight client close their sessions. Fix: end the session in a finally block with closeSession(new CloseSessionRequest(), token), the way the drivers do, before the client and the allocator close. On a local cluster, the ArrowFlightSQL sessions in SHOW PROCESSLIST grow by two per run of the suite before this change (the framework's Flight SQL JDBC connection and the leaked one) and by one after it. ### Release note None ### Check List (For Author) - Test: Regression test - test_table_types_over_arrow_flight passes on a local cluster with this PR's FE, before and after the change; SHOW PROCESSLIST as above - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
run buildall |
TPC-H: Total hot run time: 28346 ms |
TPC-DS: Total hot run time: 151417 ms |
ClickBench: Total hot run time: 24.17 s |
|
@hutiefang76 I pushed 221a0af on top of your branch for the round-2 review finding: |
|
/review |
There was a problem hiding this comment.
Reviewed all five changed files at 221a0af5da1921c00a5d419ebfef04c518e4d8ee, with upstream/downstream tracing and two review rounds. One new P2 finding is reported inline: removing the service's VIEW filter exposes VIEW-only discovery on cloud followers to unrelated OLAP version RPC failures, allowing a successful but incomplete result. This should be corrected.
No additional user focus was supplied. The existing request for a SQL/Flight regression is addressed by the added suite. Existing session-leak comment 4429901003 is fixed by the token-preserving CloseSession call in finally; it is not reposted. No existing P0/P1 finding remains to carry forward.
| Critical checkpoint | Conclusion |
|---|---|
| Goal and evidence | The type stream and complete type-set filtering are implemented; the cloud-follower failure case remains incorrect as reported inline. New helper, producer/RPC and JDBC/plain-client tests were inspected. |
| Scope and reuse | Focused production changes reuse TableType.toMysqlType and streamMetadata, with three accompanying test files. |
| Concurrency and locks | Existing bounded session-command locking, thread-context restoration and metadata-lock patterns are retained. New roots and filter state are per request; static type data is safely published. |
| Lifecycle and memory | Request Arrow roots close on success and exceptions; producer/session allocators have separate ownership. Result sets, streams and the raw regression session have cleanup. |
| Configuration | None added. |
| Compatibility | Existing schemas and ticket encoding are retained. The new type stream has the required schema and sorted output described by the Arrow 19 Flight SQL specification. No storage-format change. |
| Thrift/protobuf IDs | No definition changes. |
| Parallel paths | Checked both schema modes, internal/system/external type representations, aliases/case variants and empty/single/multiple/unknown/duplicate filters. The cloud-follower path is the reported exception. |
| Conditional checks | Null filter, set membership and empty-result branches have concrete command semantics; no additional incorrect branch found. |
| Test coverage | Tests cover metadata values, filters, schema alignment, repeated RPCs, failed sends and later query use. They lack a controlled real-service cloud-follower version-failure case, requested inline. |
| Test results | Assertions and decoded schemas inspected; no .out files changed. No test-success claim for this invocation. |
| Observability and errors | Stream errors use existing logging/status conversion. The existing service catch newly allows silent result truncation through this change, as reported inline; no additional telemetry need identified. |
| Transactions and persistence | No production transaction/EditLog/replay modifications. |
| Writes and atomicity | Production metadata paths remain read-only; no new write/crash-recovery protocol. |
| FE/BE variables | None added. |
| Performance | Set matching and schema collection are linear in the retained metadata. Projection removes unused stats/comments, but excluded cloud OLAP tables can still cause unnecessary version work; covered by the same finding. No benchmark run. |
| Other/core invariants | No rowset, committed-data visibility or delete-bitmap changes; no additional substantiated defect after the final recheck. |
All reviewers returned NO_NEW_VALUABLE_FINDINGS in round 2 beyond the accepted finding, and all candidates and initial risks have dispositions. No unresolved static coverage question remains. This was a static-only review: no builds, tests, benchmarks, runtime reproductions or native-BE validation were performed. The added tests' current-head execution remains unverified.
Review converged after 2 round(s).
| String dbName = getDbsResult.getDbs().get(dbIndex); | ||
| String catalogName = getDbsResult.isSetCatalogs() ? getDbsResult.getCatalogs().get(dbIndex) : ""; | ||
| TListTableStatusResult listTableStatusResult = listTableStatus(dbName, catalogName); | ||
| List<TTableStatus> tables = listTableStatus(dbName, catalogName).getTables(); |
There was a problem hiding this comment.
[P2] Keep VIEW-only listings independent of OLAP version RPCs
After removing setType("VIEW"), a GetTables request filtered to VIEW also enumerates matching OLAP tables. On a cloud follower, FrontendServiceImpl.listTableStatus calls CloudPartition.getSnapshotVisibleVersion even for this name/type-only projection. With an expired cache, an MS RPC failure is caught around the entire enumeration and returns only the accumulated prefix. If that base table precedes a view, the view is silently missing although it needs no partition version; the previous VIEW-only path never visited the base table. Skip the version refresh for this projection, or apply the canonical type predicate before that work, and cover this cloud-follower failure case using the real service.
There was a problem hiding this comment.
Fixed in 747a1d4. On a non-master FE in cloud mode, listTableStatus now refreshes an OLAP table's partition versions only when the request needs UPDATE_TIME, the one status column that reads them (OlapTable.getUpdateTime() takes the partitions' visible version time). GetTables asks for TABLE_NAME and TABLE_TYPE only, so no GetTables request, VIEW-only or not, waits on the meta service or is cut short by it. information_schema.tables scans that read UPDATE_TIME (projected or in a predicate) and requests without required_columns still refresh as before (#43451); information_schema.views asks for views only and never reaches the refresh. FrontendServiceImplTest#testListTableStatusRefreshesCloudFollowerVersionsOnlyForUpdateTime runs the service as a cloud follower whose version RPC fails: with the name/type projection it returns both the base table and the view without calling it, and with UPDATE_TIME it still calls it. Without the change the first listing calls it and comes back empty, since the base table's failed refresh ends the enumeration before either object is added. FlightSqlSchemaHelperTableTypesTest now also checks that GetTables sends exactly TABLE_NAME and TABLE_TYPE.
…r UPDATE_TIME ### What problem does this PR solve? Issue Number: None Related PR: apache#67899 Problem Summary: GetTables no longer passes its first requested type to FrontendServiceImpl.listTableStatus, so a request for VIEW only now lists every object in the database and the Flight SQL helper filters the rows. On a non-master FE in cloud mode, listTableStatus refreshes each OLAP table's partition versions from the meta service before it lists the table (apache#43451), whatever columns the caller asked for. GetTables asks for TABLE_NAME and TABLE_TYPE only, and neither reads a partition version; only UPDATE_TIME does (OlapTable.getUpdateTime() takes the partitions' visible version time). So when the cached versions on the follower have expired, every GetTables request waits on meta service RPCs for each OLAP table it lists. When the meta service keeps failing, the RpcException leaves the per-table loop for the catch around the whole enumeration, which logs a warning and returns the rows collected so far: the request succeeds without that table and every object after it, views included. Fix: refresh the partition versions only when the request needs UPDATE_TIME. A request without required_columns (an older BE) still needs every column and still refreshes, and so does an information_schema.tables scan that reads UPDATE_TIME in its projection or in a predicate. ### Release note None ### Check List (For Author) - Test: Unit Test - FrontendServiceImplTest#testListTableStatusRefreshesCloudFollowerVersionsOnlyForUpdateTime: a cloud follower whose version RPC fails lists a base table and a view for TABLE_NAME and TABLE_TYPE without calling it, and still calls it for UPDATE_TIME; without the new condition the first listing calls it and loses rows - FlightSqlSchemaHelperTableTypesTest: GetTables requests exactly TABLE_NAME and TABLE_TYPE - Behavior changed: Yes. On a non-master FE in cloud mode, listing table status fetches partition versions from the meta service only when UPDATE_TIME is requested - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
run buildall |
TPC-H: Total hot run time: 28519 ms |
|
@hutiefang76 I pushed 747a1d4 on top of your branch for the round-3 review finding (thread r4430787448). GetTables no longer passes its first requested type to Local results on 747a1d4: |
|
/review |
TPC-DS: Total hot run time: 150978 ms |
ClickBench: Total hot run time: 24.92 s |
There was a problem hiding this comment.
Static review of PR #67899 at head 747a1d4 converged in round 1.
The change meets its stated goal: GetTableTypes now streams the canonical sorted Doris types, and GetTables normalizes case and TABLE/SYSTEM TABLE aliases before filtering the authorized listing. I traced the full path through FlightInfo/DoGet, session command serialization, Arrow root ownership, FrontendServiceImpl.listTableStatus, describeTables, TableIf.TableType mappings, external catalog propagation, and includeSchema alignment. SHOW authorization is applied before local filtering and rechecked for schema descriptions.
The cloud follower change is projection-safe: names/types-only requests send TABLE_NAME and TABLE_TYPE, requiredColumns normalization prevents UPDATE_TIME refresh for that projection, and legacy/unset projections plus UPDATE_TIME consumers retain the refresh. Status fields and required Thrift fields remain valid. Error/listener handling and allocator cleanup follow the existing metadata streams, including failed sends.
I independently revalidated existing inline context: the VIEW-only cloud version-RPC issue and the raw Flight session leak are fixed at this head, and the requested SQL regression coverage is present. No P0/P1 or other substantiated new finding remains, so there are no inline comments or existing blocking IDs. The user review focus file contains no additional focus. Added unit and regression tests were inspected for coverage, determinism, and cleanup; this invocation did not execute builds or tests under the static-only review contract.
Review converged after 1 round(s).
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
|
run p0 |
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
…Schema (#69867) ### What problem does this PR solve? Issue Number: None Related PR: #68631, #68768 Problem Summary: **In short.** This ports branch-4.1 #68631 (also on branch-4.2) to master. A Flight SQL `CreatePreparedStatement` now returns the analyzed result schema of the statement, `GetSchema` is implemented for both statements and prepared statements, and a prepared execution is checked against the schema that Prepare advertised. Before this PR, Prepare always returned a fixed one-column placeholder, and `GetSchema` answered UNIMPLEMENTED. The analysis runs in the FE only and does not plan or execute the query. Original author: @Gabriel39 **Background.** Flight SQL clients learn the shape of a result before they fetch it. The Arrow Flight SQL JDBC driver prepares every statement and builds `ResultSetMetaData` from the dataset schema that `CreatePreparedStatement` returns. ADBC exposes the same schema as `execute_schema`, and `GetSchema` is the RPC that asks for it without running anything. **The problem, and what it cost.** On master: | | master today | with this PR | |---|---|---| | `CreatePreparedStatement` dataset schema | always `ResultMeta: Utf8`, whatever the statement | the statement's columns, names and Arrow types as BE / FE will send them; statements whose schema cannot be told before they run, and results with an untyped NULL column, keep the placeholder (see below) | | `GetSchema` for a statement or a prepared statement | UNIMPLEMENTED | the analyzed schema, or UNIMPLEMENTED for a statement that keeps the placeholder | | Invalid SQL | fails only at execution | fails at Prepare with INVALID_ARGUMENT | | A query whose privilege check fails | fails only at execution | still fails only at execution: Prepare keeps the placeholder, because only the check on the executed plan decides (see below), and GetSchema answers UNIMPLEMENTED without a schema | | Prepared handle used after the session's catalog / database changed, or after the result schema changed (e.g. another session's DDL) | runs the unqualified SQL in whatever namespace the session has now, with whatever columns come back | refused with NOT_FOUND "prepare again", and the result left by that execution is dropped | | Prepared handle presented by another session | resolved by the identity written in the handle | must start with the caller's own identity (from #68768) | | TIMEV2 / VARBINARY columns in GetTables and Prepare schemas | `Null`, while BE sends `float64` / `binary` | `Float64` / `Binary`, as BE sends them | A client that trusts the advertised schema saw one string column for every query, and could not ask for a schema at all without executing. **How this PR fixes it.** - `arrowflight/FlightSqlQuerySchema` analyzes the statement on a clone of the session variables: a USE / SWITCH prefix is resolved in a temporary namespace, the analysis `StatementContext` is closed on the same thread, and the connector statement scope it may have opened is closed explicitly. Each field gets the same `doris_type` markers and native Variant layout the execution side uses (`FlightSqlSchemaHelper.withDorisTypeMetadata` / `nativeVariantField`), and TIMESTAMPTZ carries the canonical session time zone that BE stamps on its batches. - Privileges are checked on the analyzed plan after the CTEs that execution inlines have been inlined (`PullUpCteAnchor`, `CTEInline`, as the rewriter does before its own check). A CTE used once is therefore checked only on the columns the query reads, and a CTE used more than once on every column it selects, which is also how execution checks them. The rewriter normalizes the plan further before it checks privileges. For example, it drops an ORDER BY under a derived table, so the analyzed plan can read a column that the executed plan does not. When the analyzed plan is denied, the statement keeps the placeholder and execution decides. No schema is disclosed in either case. - FE-side commands whose result header does not depend on running them now report that header without running: HELP, SHOW SNAPSHOT, SHOW CREATE TABLE (one header selector shared with `doRun`, including master's stream, plugin-view and system-table cases), SHOW PROC `/` and `/current_queries`, SHOW CONFIG, and EXPLAIN PLAN PROCESS (Rule / Before / After). - `ShowStoragePolicyCommand.getMetaData()` returned the four-column USING header for every form, while plain SHOW STORAGE POLICY answers the seven columns of `StoragePolicy.STORAGE_META_DATA`. It now returns the header of the form that runs. This also corrects the column definitions that MySQL `COM_STMT_PREPARE` reports for the statement. No other SHOW command declares a header that differs from the one it returns. - WARM UP SELECT keeps the placeholder. It answers OK instead of its statistics whenever planning proves its source empty: a predicate that folds to false, no partition left after pruning, or partitions without data. - A result with an untyped NULL column keeps the placeholder too, at any depth (`SELECT NULL`, `SELECT id, NULL FROM t`, `ARRAY(NULL)`). `DorisArrowTypeMapping` gives NULL_TYPE Arrow `Null`, but BE builds such a column as a `UInt8` flagged as a NULL literal and sends it as nullable `Bool`, so an advertised `Null` would expire the handle at every execution. A typed NULL (`CAST(NULL AS VARCHAR(20))`) is described as its type. - The prepared query kept in `FlightProtocolAdapter` records the SQL, the namespace and the advertised schema. A prepared execution re-analyzes the statement and compares the result with the advertised schema before it runs, and compares the schema it actually produced before it publishes a DoGet ticket. It is the session's next request whether or not these checks pass: before them, it drops what the previous request left, as a statement's execution does, while Prepare and GetSchema are metadata requests and leave it. On a mismatch, the handle is removed, the execution is cancelled, and the deferred coordinator, the channel result and the endpoints are dropped. A handle whose statement can no longer be described when it is re-analyzed, for example after a column grant was revoked, expires the same way, and the statement prepared again keeps the placeholder. A USE / SWITCH inside the statement rebinds only that handle. - `DorisArrowTypeMapping` maps TIMEV2 to `Float64` and VARBINARY to `Binary`, the types BE sends. TIMESTAMPTZ keeps the fixed `"UTC"` in GetTables, and AGG_STATE keeps `Null` there. - `FeServiceClient` rejects a remote Doris metadata timeout whose milliseconds do not fit in an int, instead of letting it wrap around. Where master differs from branch-4.x: - branch-4.x answers UNIMPLEMENTED at Prepare for statements whose schema it cannot tell before they run: DESC, SHOW DATA / PARTITIONS / QUERY STATS / PYTHON PACKAGES, other PROC nodes, COPY INTO, WARM UP CLUSTER, multi-statement requests that do not start with USE / SWITCH, `?` placeholders, and result types the FE cannot describe. Python ADBC falls back to executing without a prepared statement, but the Arrow Flight SQL JDBC driver prepares every statement and has no fallback, so on master these statements would fail over JDBC. `test_arrow_flight_session_lifecycle` runs DESC and `SET @multi = 2; SELECT ...` through JDBC. Here, these statements keep today's placeholder schema, and their handles are not checked at execution. - branch-4.x checks privileges on the analyzed plan as it is, so a CTE used once is checked on every column it selects, and any denial fails Prepare. Here, CTEs are inlined first, and a remaining denial keeps the placeholder. - branch-4.x advertises the statistics header for WARM UP SELECT and then rejects, as a schema change, an execution whose source turned out empty. Here, WARM UP SELECT keeps the placeholder. - branch-4.x advertises Arrow `Null` for an untyped NULL column, which BE sends as `Bool`, so every prepared execution of such a query is rejected as a schema change. Here, such a result keeps the placeholder. - The analysis and the checks run under the session's command lock (`FlightProtocolAdapter.runCommand` / `callCommand`) instead of `synchronized (connectContext)`, and the namespace is saved and restored with `getNameSpaceContext()` / `restoreNameSpaceContext()`. - EXPLAIN PLAN PROCESS has a result over Flight on master, so it gets a static header instead of UNIMPLEMENTED. PLAN REPLAYER and SHOW CREATE PROCEDURE do not exist on master and are not carried over. - SWITCH resolves the default database through the connector provider (`Env.defaultDatabaseOnUse`, now shared with `Env.changeCatalog`) instead of `EsExternalCatalog.DEFAULT_DB`. - TIMESTAMPTZ fields carry the canonical session time zone that BE stamps, where branch-4.x uses the raw session value. - Only the two handle checks from #68768 that do not depend on parameter binding are taken: the caller-prefix check and rebinding only the handle that ran the USE / SWITCH. Parameter binding itself is not part of this PR. - `GetTableTypes`, the other half of #68631, is implemented by #67899. Cost: every Prepare, every prepared execution and every `GetSchema` runs one more FE analysis of the statement. **Results.** On a local cluster built from this branch, every suite under `arrow_flight_sql_p0` passes: the new `test_schema_only` and `test_get_tables_schema`, plus the seven suites that run through Flight JDBC, `test_arrow_flight_session_lifecycle` among them. `datatype_p0/uuid/test_uuid_arrow_flight` passes too. `test_schema_only` checks the advertised schema of each kind of statement against what its execution actually returns. That covers queries, the side-effect-free frontend-side commands executed as prepared statements (SHOW STORAGE POLICY among them), a WARM UP SELECT on an empty table (executed in cloud mode, the only mode that warms up internal tables), and a `group_concat_state` AGG_STATE, which BE sends as Utf8. Co-authored-by: Gabriel <[email protected]>
What problem does this PR solve?
Related to #67578 (partial A3).
Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.
Implement the type stream using the same Doris names as GetTables (BASE TABLE, SYSTEM VIEW and VIEW), sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.
A type is the TABLE_TYPE that information_schema.tables reports for the object: table streams are VIEW, and external catalogs report their tables and views as BASE TABLE. The service lists every authorized object and the filter runs on that listing. Requested names match case-insensitively, and TABLE and SYSTEM TABLE, which Flight SQL calls commonly supported, select BASE TABLE and SYSTEM VIEW: the ADBC Flight SQL driver asks for
table(Doris's own ADBC catalog lists a Doris source withgetObjects(..., {"table"})), and JDBC tools ask forTABLE. Any other name matches only a type of that name, so UNKNOWN selects nothing.Because the filter now runs on the full listing,
FrontendServiceImpl.listTableStatusrefreshes a cloud follower's partition versions from the meta service only when UPDATE_TIME is requested, the one column that reads them. GetTables, which asks for names and types, never waits on those RPCs or is cut short when they fail, and information_schema.tables scans that do not read UPDATE_TIME skip the refresh too.Release note
Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately. Type names match case-insensitively, and TABLE and SYSTEM TABLE are accepted for BASE TABLE and SYSTEM VIEW.
Check List (For Author)
FlightSqlSchemaHelperTableTypesTest(27: every filter with and without schemas, aliases and mixed case, external catalog) andFlightSqlTableTypesTest(3, including GetFlightInfo -> DoGet through a realFlightServer); regressionarrow_flight_sql_p0/test_table_types_over_arrow_flightin P0 and cloud_p0 (Flight SQL JDBCgetTableTypes()/getTables()with the filters above, then the same RPCs from a plain Flight SQL client); the 21external_table_p0/adbcsuites, which list a Doris source through the ADBC Flight SQL driver, in External.FrontendServiceImplTest#testListTableStatusRefreshesCloudFollowerVersionsOnlyForUpdateTime: a cloud follower whose version RPC fails lists a base table and a view for TABLE_NAME and TABLE_TYPE without calling it, and still calls it for UPDATE_TIME.FlightSqlSchemaHelperTableTypesTestalso checks that GetTables requests exactly TABLE_NAME and TABLE_TYPE. The command below passes locally (75 tests) and FE Checkstyle reports 0 violations.Validation command (JDK 17, matching Thrift 0.24.0 compiler):
The branch was rebased onto master 2a4191c (merge commits dropped). 2a31c4d, 1258c51, 221a0af and 747a1d4 were added on top by a maintainer: end-to-end GetTableTypes checks through the Flight RPC, the type-name aliases, closing the test's plain Flight session, and the cloud follower version refresh limited to UPDATE_TIME. No local toolchain configuration or generated sources are included.
AI assistance: DeepSeek and Codex assisted review, conflict resolution, validation and this description.