Repository navigation
[fix](fe) Disable LOCAL TVF (branch-4.2) - #69592
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69527 Problem Summary: Add startup-only enable_local_tvf=false. Reject LOCAL TVF and FILE-local at FE admission before discovery. Explicit opt-in retains the existing storage implementation and requires ADMIN/OPERATOR. Supported explicit LOCAL sinks use the same FE admission check. ### Release note LOCAL TVF and FILE TVF with local storage require enable_local_tvf=true in fe.conf on every query-serving FE and an FE restart. This change provides default-disable mitigation; enabled storage-path sandbox hardening remains separate work. ### Check List (For Author) - Test: Focused FE unit and regression cases added but not executed. Changed Java files pass Checkstyle; diff whitespace checks pass. - Behavior changed: Yes. LOCAL access requires startup opt-in and ADMIN/OPERATOR admission. - Does this need documentation: Yes. Configuration description and release note included; user-documentation follow-up remains pending.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Reviewed all 13 changed files at head 42c3453. One verified P2 finding: the new default-disabled sink gate breaks the existing duplicate-output unit test unless that test explicitly opts in. No P0/P1 findings or existing blocking inline comments were found. The review converged after 2 rounds of static review.
Critical checkpoint conclusions:
- Goal and evidence: LOCAL and FILE-local sources reject before common-property parsing, backend selection, globbing and Lance schema discovery; explicit LOCAL INSERT and analyzed EXPLAIN INSERT reject in BindSink before sink-property validation. The added tests cover these admission branches statically; no execution results were produced.
- Scope: one startup flag and a shared admission helper keep the production change focused. The seven regression deployment opt-ins preserve existing enabled LOCAL coverage.
- Concurrency and locks: request workers attach the authenticated ConnectContext; no new threads or lock nesting are introduced. Existing auth read locks protect privilege queries; the new flag is startup-only.
- Lifecycle and initialization: existing per-expression catalog/table memoization remains; later checkAuth still runs. No new ownership cycle, release path or static initialization dependency was found.
- Configuration: ConfigBase defaults mutable=false and rejects runtime changes. Setting enable_local_tvf in fe.conf and restarting every query-serving FE matches the implementation.
- Compatibility: default-disabled behavior is intentional and requires operator opt-in on every FE. No storage-format, function-symbol or serialized representation changes occur.
- Thrift/protobuf: no definitions changed; cross-release field-ID checks are inapplicable.
- Parallel paths: LOCAL, FILE-local delegation, legacy construction, DESCRIBE, CSV and Lance sources were traced, along with INSERT/EXPLAIN sinks. S3/HDFS/HTTP FILE delegates and remote sinks remain outside this flag. Separate parquet metadata TVFs and general filesystem-provider hardening are outside the expressly named LOCAL/FILE mitigation; their pre-existing literal-local capability is not claimed to be disabled.
- Conditions and errors: the admission-before-discovery purpose is documented; disabled and ADMIN/OPERATOR errors are explicit. Existing resource USAGE checks remain in later query authorization. The built-in operator role includes ADMIN and passes the pre-existing INSERT check; custom identities retain existing write-privilege requirements.
- Tests: new source and sink unit tests cover enabled/disabled, privilege and remote-route branches. Existing enabled sink/write-readback regression coverage is retained by startup opt-ins. The existing isolated duplicate-output test lacks that opt-in (inline finding).
- Expected results: no .out files changed. SHOW CONFIG index 1 is the value; backend_id=-1 yields the regression's expected validation error when enabled, while disabled admission wins otherwise.
- Observability: actionable analysis messages suffice for this FE admission change; no new distributed operation or metrics requirement was identified.
- Transactions and persistence: no EditLog, metadata persistence or transaction lifecycle change.
- Writes and crashes: disabled admission precedes LOCAL sink serialization and execution. Existing enabled filesystem-write/coordinator behavior is unchanged; no new write resource or atomicity mechanism is introduced.
- FE/BE transport: no new transmitted variables or sending paths are required.
- Performance: bounded flag/privilege checks precede expensive discovery; no added input-sized scans or remote operations.
- Other risks: constructor context, FILE routing, memoization/resource authorization, operator privilege composition and configuration semantics were rechecked. No additional substantiated finding remained.
User focus: no additional focus points were supplied. Verification limits: static inspection only; builds, unit tests and regressions were not executed under the supplied contract. The new coverage does not demonstrate actual unprivileged-user behavior or a default-disabled deployment end to end; enabled storage-path sandbox hardening remains separate work. No unresolved candidate or unreviewed changed-file scope remains.
Review converged after 2 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: The existing positive LOCAL sink binding test reaches the new default-off gate before checking duplicate output positions. Save and enable the flag in fixture setup and restore it in teardown. Preserve the original LOCAL target, ROOT context and ExprId assertions. ### Release note None ### Check List (For Author) - Test: Changed-file Checkstyle and diff checks passed. Unit tests were not executed locally. - Behavior changed: No production behavior change; existing test fixture opts in explicitly. - Does this need documentation: No.
|
/review |
There was a problem hiding this comment.
Static review of all 14 changed files at head 018f7940df23329466149623f4c6265a36f5a5fb found no new substantiated defect. The existing P2 thread 4371999865 is fixed: the duplicate-output sink fixture saves, enables, and restores the flag, and its ROOT context has the required privileges. No existing P0/P1 inline finding remains to carry forward.
The read path reaches LocalTableValuedFunction.checkAuth/checkAccess before LOCAL property parsing, backend selection, globbing, or Lance discovery. FILE-local delegates through that constructor and retains later authorization checks. Explicit LOCAL INSERT and ordinary EXPLAIN INSERT both reach BindSink before destination validation and physical sink creation. Prepared execution replans the unbound sink; normal connection entry points install the current user context. Remote FILE delegates and S3/HDFS sinks retain their existing paths and permissions.
Critical checkpoint conclusions:
| Checkpoint | Conclusion |
|---|---|
| Goal and supporting tests | Default-off admission and enabled ADMIN/OPERATOR checks are implemented for the stated LOCAL/FILE-local and explicit LOCAL-sink scope. Added unit tests describe denied/enabled, SELECT/EXPLAIN-related construction, INSERT/EXPLAIN binding, FILE delegation, CSV/Lance, and remote cases; they were inspected, not executed. |
| Scope and clarity | One startup flag and a shared admission method are used at the read and sink boundaries; changes remain focused. |
| Concurrency and lock protection | No new threads, shared mutable runtime policy, or locks. Existing privilege checks use Auth's read-lock path; no new lock order, heavy work under a new lock, or deadlock mechanism was found. |
| Lifecycle and static initialization | Config is loaded before FE services start. The new primitive default introduces no cross-initializer dependency or resource ownership. Test fixtures restore the flag and close scoped static mocks. |
| Configuration and restart behavior | ConfField.mutable defaults to false and ConfigBase.setMutableConfig rejects runtime updates. Each FE must load its own setting at restart, as the description and rejection message state. |
| Compatibility and rolling upgrades | Disabling LOCAL by default is the intentional behavior change. No function signature or storage format changes; admission remains dependent on each FE's installed version and startup setting during rollout. |
| Thrift/protobuf definitions | None changed; no field/enum compatibility comparison is required. |
| Parallel paths | Direct LOCAL, FILE-local, DESCRIBE/factory callers, CSV/Lance, shared/backend-selected reads, explicit sinks, and remote delegates were traced. Local parquet_meta wildcard expansion inherits the gate through its existing LOCAL helper; literal local metadata access remains a separate preexisting path. |
| Special conditions and errors | Disabled admission precedes LOCAL property parsing and privilege calls; the enabled ADMIN-or-OPERATOR predicate matches existing LOCAL checks. Sink matching uses the existing normalized name. Denials propagate as client-visible analysis failures. |
| Functional, negative, and unit coverage | New tests cover malformed properties while disabled, denied privileges before file access, enabled continuation, FILE-local delegation, and remote preservation. The unchanged duplicate-output assertions retain an enabled sink case. Seven regression FE configurations explicitly opt in. |
| Changed test results | No result snapshots changed. New expected-error substrings match the inspected branches; the Groovy suite checks LOCAL/FILE SELECT and EXPLAIN according to the loaded flag. |
| Observability | Disabled errors name the opt-in/restart action; authorization failures name the required privileges. No new distributed operation or service requiring additional metrics was introduced. |
| Transactions, EditLog, and persistence | No transaction metadata, EditLog, replay, or persistence logic changed. No new master-failover state is introduced. |
| Writes, atomicity, and crashes | The LOCAL sink guard runs before writer/coordinator creation. Existing write atomicity, file cleanup, and crash behavior are unchanged; no new resource leak or write-side race was identified. |
| FE/BE variable propagation | The flag is an FE admission policy. No new wire variable or FE/BE send path is required. |
| Performance and memory | Added work is bounded privilege checking and normal revalidation. No new file traversal, unbounded allocation, significant retained memory, or material CPU/copying overhead was identified. |
| Data/version/delete-bitmap invariants and further recheck | No committed-data visibility, partition version, MoW bitmap, or BE memory-management logic changed. Constructor dispatch, parent resource authorization, prepared/cached plans, indirect callers, and unaffected tests were rechecked without another actionable issue. |
No additional user-provided focus was supplied. The initial constructor/context and sink-bypass risks were dismissed with concrete call-chain evidence; no candidate remains unresolved.
Coverage limits: this was static analysis only; no builds, unit/regression runs, live non-admin session, disabled deployment, or multi-FE restart/upgrade exercise was performed. Standard changed regression deployments opt in, so disabled behavior is covered mainly by mocked unit tests unless the conditional Groovy suite is run against a default-config deployment. Enabled storage-path sandbox hardening and broader local access through other preexisting providers/TVFs remain separate work; this review does not establish universal local-filesystem isolation.
Review convergence: converged after 2 rounds, with all three reviewers returning NO_NEW_VALUABLE_FINDINGS on the final frozen payload, complete assigned static coverage, and evidence-backed dispositions for every risk/candidate.
Review converged after 2 round(s).
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69591, apache#69527 Problem Summary: Disable LOCAL and FILE-local admission without an opt-in setting, before file discovery. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same rejection before sink property validation. Retain storage implementation and adapt only directly affected fixtures and suites. ### Release note LOCAL TVF and FILE TVF local storage are unavailable for all users, with no enable setting. LOCAL TVF INSERT is also unavailable. ### Check List (For Author) - Test: 8 targeted FE unit tests passed without failures/skips; changed Java Checkstyle, Groovy syntax and whitespace checks passed. Regression suite bodies not executed. - Behavior changed: Yes. Unconditional LOCAL rejection. - Does this need documentation: Yes. Release note included; operator documentation follow-up.
|
/review |
There was a problem hiding this comment.
Static review of all 36 changed files at head 531cd94 found one new P2: the unchanged Parquet metadata suite still expects LOCAL-backed glob success and the former no-match error. Both now fail at the constructor denial. No new P0/P1 finding was verified.
The concrete flows reviewed were SELECT/EXPLAIN -> LOCAL or FILE-local construction -> rejection before discovery, and INSERT/analyzed EXPLAIN -> UnboundTVFTableSink -> BindSink -> rejection before sink property validation and execution. Remote FILE delegation and S3/HDFS sink paths remain available.
| Required checkpoint | Conclusion |
|---|---|
| Goal and evidence | Explicit LOCAL reads and writes reject unconditionally, including privileged callers. New FE negative tests and the account regression express this behavior; the indirect metadata test omission is reported inline. |
| Scope and clarity | Production changes are confined to the shared denial helper and the independent read/sink admission sites. Test changes adapt consumers of the disabled feature. |
| Concurrency and locks | No new threads, shared mutable state, critical variables, locks, lock-held I/O, or lock-order changes. |
| Lifecycle and static initialization | No new resource ownership or cross-unit initialization dependency; denial precedes backend selection, RPC discovery and sink execution. |
| Configuration | No configuration or dynamic opt-in is added; the rejection has no privilege/configuration race. |
| Compatibility | LOCAL removal is an intentional behavior change. No persisted format, exported function symbol, or rolling-upgrade protocol is modified. |
| Thrift/protobuf | No changed definitions, field IDs, enum values or required fields. |
| Parallel paths | LOCAL expressions, FILE-local delegation, legacy factory, DESCRIBE and independent INSERT/EXPLAIN binding were traced. The additional metadata constructor consumer causes the reported test failure. |
| Special conditions | Case-normalized sink dispatch and the unconditional constructor/helper check match the stated policy. Remote branches are unchanged. |
| Test coverage | Constructor tests cover missing properties, formats, FILE-local and privileged callers; sink tests cover INSERT/analyzed EXPLAIN and remote property validation. The duplicate-output test retains its ExprId assertions using S3. |
| Expected results | Stream Load conversions retain the existing golden counts of 2 and 25000, with final array/map checks. The Parquet metadata local-glob block still expects 3 and must be adapted. No .out files changed. |
| Observability | The explicit analysis error reaches existing query/command error conversion; no new distributed work requires added metrics. |
| Transactions and persistence | No EditLog, commit, visible-version, delete-bitmap, replay or failover changes. |
| Writes and atomicity | LOCAL writes stop before physical sink execution. Remote write machinery and crash cleanup are unchanged. Stream Load replacements use existing success/row-count and publish checks. |
| FE/BE variables | No new transmitted variable or alternate send path to update. |
| Performance | The fixed-cost denial avoids LOCAL discovery/planning work; no new repeated scans, copies or hot-path allocation found. |
| Other concerns | Checked/unchecked exception boundaries and wrapper/lazy construction were inspected; no additional verified production defect found. |
Existing comment 4371999865 no longer applies: BindTVFTableSinkTest now uses S3 while preserving distinct output-position assertions. There are no confirmed existing P0/P1 comments to carry forward. No additional user focus was supplied.
Coverage limits: this was static review only; no builds, unit tests, regression suites or runtime checks were executed. LOCAL success suites and LOCAL export-readback comparisons are intentionally retired. Retained export completion/error checks do not establish exported-content correctness or absence of empty output files. The existing remote UUID S3 branches, HDFS/S3 partition cases, and ORC value/count checks remain active. Independent literal-local metadata and other filesystem APIs are outside this PR's explicit LOCAL/FILE-local admission scope.
Review converged after 2 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69591 Problem Summary: Parquet metadata local wildcard expansion delegates to the disabled LOCAL TVF constructor. Expect the admission error for both the former successful glob and the no-match glob. Preserve all literal-path and remote metadata checks. ### Release note None ### Check List (For Author) - Test: Groovy 4.0.19 syntax parsing and whitespace checks passed; all other suite blocks verified byte-for-byte unchanged. Runtime regression not executed locally. No Java source changed. - Behavior changed: No. Test expectations follow the existing unconditional rejection. - Does this need documentation: No.
|
/review |
There was a problem hiding this comment.
Static review of #69592 at 0bb02dc: no new substantiated findings. Reviewed all 37 changed files, related callers and downstream error/execution paths. Two complete-review reviewers and a separate risk reviewer covered production, Java fixtures, all 32 Groovy changes, exception propagation and remaining LOCAL consumers. Review converged after 3 rounds, including the frozen-payload final sweep; no unresolved candidate or assigned static coverage gap remains.
Critical checkpoint conclusions:
- Goal and proof: LOCAL construction rejects before common-property parsing, backend selection or discovery; explicit LOCAL sinks reject in BindSink before sink-property validation or execution. FILE local delegation and analyzed EXPLAIN reach these gates. Supplied tests exercise these behaviors; this invocation inspected them without executing them.
- Scope: two small production edits share one unconditional gate; the other changes adapt dependent tests.
- Concurrency and locks: no new thread, shared mutable state, lock acquisition or lock-order change. The LOCAL constructor cannot start backend futures, and LOCAL sink rejection precedes coordinator creation.
- Lifecycle and initialization: no ownership, cleanup or static-initialization dependency added; LOCAL read rejection precedes backend resource acquisition.
- Configuration: no flag is added and no opt-in exists, including for ADMIN/OPERATOR; dynamic configuration propagation is inapplicable.
- Compatibility: the intentional SQL behavior change is documented in the PR release note. No storage format or existing function symbol is removed. Older FEs retain their previous admission behavior during rollout; the patch introduces no wire-format change.
- Thrift/protobuf: none changed, so release-branch field/enum comparison is inapplicable.
- Parallel paths: checked direct LOCAL, FILE-local, INSERT, analyzed EXPLAIN, DESC FUNCTION and the metadata-glob delegate. Remote FILE/sinks and literal parquet_meta paths retain their existing routing. Scope is the named LOCAL entry points; underlying storage APIs are retained.
- Conditions and errors: the sink compares its normalized name; the common gate always throws. Local/File expression wrappers and statement/parser boundaries preserve query errors. INSERT's existing privilege check may reject first without file access.
- Tests: five new constructor/delegation tests, two sink-analysis tests, the adapted duplicate-output fixture and the account rejection regression cover malformed properties, formats, privileges, FILE, EXPLAIN and remote validation. The remote FILE fixture mocks construction; actual remote IO and SQL execution were not validated here.
- Expected results: no .out files changed. The ORC Stream Load migrations preserve fixtures, full-row load validation, synchronization and expected counts of 2 and 25000, plus later value/map checks. The framework skips retired output tags. LOCAL-only success suites and export readback assertions are intentionally retired, reducing that provider's coverage while unavailable.
- Observability: the stable LOCAL-disabled analysis error and existing query error reporting suffice for this early rejection; no new distributed path needs metrics.
- Persistence and failover: no EditLog, transaction metadata, visible-version or replay behavior changed.
- Writes and atomicity: LOCAL sink rejection precedes coordinator/file-write creation. Remote write semantics remain unchanged; migrated fixtures use the existing Stream Load validation path.
- FE/BE propagation: no new variable, descriptor or request field requires transmission changes.
- Performance: constant-time admission checks avoid LOCAL parsing/discovery work; no new scan, retry loop or large allocation was introduced.
- Other invariants: data visibility, MoW delete bitmaps, BE memory/null handling and optimizer slot semantics are unchanged. Rechecked lazy LOCAL expressions and unused nested-type LOCAL closures; active nested-type loads use S3.
Existing review context: P2 comment 4371999865 is fixed by the S3 duplicate-output fixture, which preserves distinct ExprId assertions. P2 comment 4373692352 is fixed by both parquet_meta wildcard cases expecting LOCAL-disabled errors while preserving literal-local and remote cases. Neither issue is reposted. No existing P0/P1 finding remains to carry.
User focus: no additional focus was supplied. Limits: static source/control-flow and fixture/output review only; no builds, tests, runtime SQL or regression suites were run. No repository source was modified and no submodules were initialized.
Review converged after 3 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: The optional UUID local-export mode still reloaded through LOCAL after LOCAL TVF was disabled. Use the disabled error expectation where applicable and retain the existing S3 UUID export/reload path, fixtures, export completion checks and golden assertions. ### Release note None ### Check List (For Author) - Test: Groovy 4.0.19 syntax parsing and git diff --check passed; runtime regression not executed locally. - Behavior changed: No (test adaptation only) - Does this need documentation: No
|
/review |
There was a problem hiding this comment.
Reviewed the complete 38-file aggregate diff at a7df11f and the relevant callers, downstream paths, fixtures and regression framework. No new actionable findings were verified. The static review converged after 1 round: both complete-review agents and the separate risk-focused agent returned NO_NEW_VALUABLE_FINDINGS; all candidates and risk questions have evidence-based dispositions.
Existing comments: 4371999865 is addressed by moving the duplicate-output sink fixture to S3 while retaining distinct ExprId assertions. 4373692352 is addressed by expecting LOCAL rejection in both Parquet metadata wildcard cases. Neither is reposted. Both are P2; no existing P0/P1 finding remains to carry. No additional user review focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: LOCAL construction rejects before property parsing, backend selection, globbing or schema discovery; FILE with LocalProperties delegates to that constructor. Explicit LOCAL INSERT and analyzed EXPLAIN reach the sink gate before sink property validation. Added FE and account tests assert these boundaries by inspection; they were not executed in this review.
- Scope: the two small production changes share one rejection method; the remaining 36 changed files adapt dependent tests.
- Concurrency and locks: no shared mutable state, worker, lock acquisition or lock-order change is introduced. Local discovery and sink execution are not reached through these rejected paths.
- Lifecycle: no new owned resource, reference cycle or static-initialization dependency. LOCAL sinks fail before file deletion or coordinator creation; active export tests retain completion/error waits and cleanup.
- Configuration: no flag or runtime opt-in is added; ADMIN/OPERATOR cannot bypass the constructor rejection.
- Compatibility: disabling LOCAL is an intentional SQL behavior change documented in the PR release note. Storage implementation and wire/storage formats are unchanged; mixed FE versions can retain different admission behavior until upgraded.
- Shared definitions: no thrift/protobuf changes, reused IDs, enum changes or required fields.
- Parallel paths: catalog/Nereids LOCAL, FILE-local, metadata local glob, INSERT and EXPLAIN were traced. Remote FILE/S3/HDFS paths remain independent. Literal-local Parquet metadata and other independent local-storage facilities predate this PR and remain available; this review does not claim all local filesystem access is removed.
- Conditions: the sink checks its normalized function name and reuses the constructor admission helper. No format, privilege or configuration exception bypasses that gate.
- Test coverage: malformed properties, CSV/JSON/Lance, LOCAL/FILE, privilege states and sink/EXPLAIN are represented in the new FE fixtures; the account regression covers SELECT/EXPLAIN. Retained remote UUID, S3 export/view and HDFS/S3 partition checks were inspected. Eighteen LOCAL-dependent suites now skip; LOCAL export content/readback and export-consistency coverage is reduced.
- Expected results: no .out files change. The two Stream Load conversions preserve the recorded counts (2 and 25000), tag order and fixture schemas; the array value comparison remains. Framework tag lookup safely skips retired uniquely tagged output blocks. Static ORC footer inspection confirms schema and row counts, not runtime loading or decoded values.
- Observability: the disabled analysis error states the reason; no new distributed operation needs additional metrics or tracing.
- Persistence and transactions: no EditLog, persistent metadata, publish/version visibility or MoW delete-bitmap production logic changes.
- Writes and atomicity: LOCAL sinks fail during planning; remote write implementations are unchanged. Replacement test loads use existing Stream Load success/row-total validation followed by sync and count assertions.
- FE/BE propagation: no new transmitted variable, function type or serialization path.
- Performance and memory: constant-time rejection prevents local discovery IO; no new significant allocation, repeated scan, critical section or BE memory ownership.
- Other checks: all changed paths and transitive LOCAL consumers were rechecked; dormant nested-type LOCAL closures have no active callers. Exceptions propagate to query errors. No additional verified issue or unresolved static coverage question remains.
Validation limits: static review only; no builds, unit tests or regression suites were run, and no submodules were initialized. Runtime, cloud and rolling-upgrade behavior were not exercised. Intentional test skips and retired local readbacks reduce ongoing regression protection as described above.
Review converged after 1 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: The ARRAY and MAP regression fixtures were migrated from LOCAL TVF to ORC Stream Load, but the load scanner rejects complex destination columns. Upload the original fixtures to unique temporary S3 objects and use INSERT SELECT from S3 TVF, cleaning up each object after the insert. Preserve table definitions, synchronization and all existing count/value checks. ### Release note None ### Check List (For Author) - Test: Groovy 4.0.19 syntax parsing, ORC fixture schema/row-count/content inspection and git diff --check passed. Runtime regression was not executed locally; matching FE/BE artifacts are unavailable in these task worktrees. - Behavior changed: No (test data-loading adaptation only) - Does this need documentation: No
|
/review |
There was a problem hiding this comment.
Static review of all 38 changed files at bad5e20 found no new actionable defects. The two complete-review reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS; the review converged in 1 round.
The earlier duplicate-output sink fixture finding (4371999865) is fixed by analyzing S3 while retaining distinct ExprId assertions. The Parquet metadata glob finding (4373692352) is fixed by adapting both matching and no-match LOCAL glob expectations. Neither is reposted, and no existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal: LOCAL and FILE-with-local-storage reads reach the unconditional constructor rejection; explicit LOCAL INSERT and analyzed EXPLAIN INSERT reach the same rejection through BindSink before sink property validation. ADMIN/OPERATOR cannot opt in. Added FE and regression cases cover the intended rejection and remote controls; they were inspected, not run.
- Scope: two small production gates; remaining changes adapt dependent tests or retire LOCAL coverage. The underlying storage implementation remains.
- Concurrency: no new shared mutable state, threads, locks, atomics, or lock-order changes.
- Lifecycle: rejection precedes backend selection/discovery and sink execution; no new resource ownership or static initialization dependency.
- Configuration: none added; no dynamic or startup opt-in exists.
- Compatibility: intentional SQL behavior removal; no changed function signatures, storage encoding, or serialized metadata. Older FEs retain their previous admission behavior until upgraded.
- Thrift/protobuf: untouched; cross-release field/enum comparison is not applicable.
- Parallel paths: factory, Nereids LOCAL, FILE-local delegation, and LOCAL sink analysis are covered. Remote delegates and S3/HDFS sinks retain existing routing. Literal-local Parquet metadata remains an independent supported path; local glob expansion reaches the disabled constructor.
- Conditions/errors: the explicit LOCAL-name condition shares the documented gate; surrounding Nereids and statement-execution boundaries translate rejection into analysis failure.
- Test coverage: rejection, malformed properties, formats, privilege independence, INSERT/EXPLAIN, remote validation/dispatch, and dependent fixtures were inspected. UUID remote scanner variants, HDFS/S3 partition cases, and surviving export completion/error checks remain active.
- Expected results: unchanged output references match the migrated ORC count queries (2 and 25000). Static ORC footer inspection confirms the named field projections; remaining array/map assertions survive. The framework skips retired reference tags correctly.
- Observability: the deterministic disabled-feature error is sufficient for this admission failure; no new distributed runtime path requires metrics or logging.
- Persistence/transactions: no EditLog, commit/publish, visible-version, delete-bitmap, or failover mechanism changes.
- Data writes/atomicity: the gate prevents LOCAL sink execution before coordinator creation; existing remote write semantics are unchanged.
- FE/BE variables: no new transmitted variables or sender/receiver changes.
- Performance/memory: immediate rejection removes LOCAL discovery work; the sink adds one name comparison, with no new growing allocation, loop, or heavy locked operation.
- Further sweep: checked indirect constructor users, exception boundaries, lazy/mocked fixtures, remote alternatives, surviving output tags, and dependent tests; no unresolved defect candidate or unreviewed changed path remains.
Limits: this was a static-only review. No builds, unit tests, SQL regression suites, or remote IO were executed. Eighteen suites now return without assertions, and LOCAL export-readback correctness coverage is retired; this reduces coverage of those behaviors. Remote S3 availability and runtime integration remain unvalidated here. The review does not establish a broader Hadoop/JNI filesystem security model beyond this PR's named LOCAL admission paths.
Review converged after 1 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: The migrated ARRAY and MAP fixtures always used an AWS client even when the regression provider was Azure. Select the existing Azure Blob SDK for Azure upload/cleanup and pass the selected provider and endpoint to S3 TVF. Resolve the framework Azure endpoint placeholder while preserving explicit endpoints. Keep S3 uploads, original fixtures and all existing assertions. ### Release note None ### Check List (For Author) - Test: Eight scripts compiled with the existing SDK dependencies; 12 loopback SDK upload/cleanup scenarios passed, including mixed-case Azure, default/explicit endpoints and injected SQL failure. SQL was intercepted in this probe; no Doris runtime regression or live cloud validation was performed. git diff --check passed. - Behavior changed: No (test data-loading adaptation only) - Does this need documentation: No
|
/review |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: Removing the disabled LOCAL fallback makes three UUID fixture suites use an AWS upload even for Azure. Use the already-declared Azure Blob SDK for Azure uploads and pass the matching provider/endpoint to TVF. Preserve explicit endpoints, the existing S3 upload path, original fixtures and all schema/value/profile assertions. This change is limited to these three existing suites. ### Release note None ### Check List (For Author) - Test: Six scripts compiled with existing SDK dependencies. Twelve loopback SDK scenarios verified the three exact UUID setup/upload blocks, Azure default/explicit endpoints, S3 uploads, byte-identical fixtures, matching TVF parameters and fixture availability for later queries. No SQL execution, Doris runtime regression or live cloud validation was performed. git diff --check passed. - Behavior changed: No (test data-loading adaptation only) - Does this need documentation: No
|
/review |
There was a problem hiding this comment.
Reviewed #69592 at b3f5fb8 using the supplied aggregate diff and changed-file list. No new actionable findings. All 38 changed files and their relevant callers, error boundaries, regression harness, and fixture/golden dependencies were covered. All three reviewers returned NO_NEW_VALUABLE_FINDINGS in round 1; both initial risk focuses are dismissed with concrete evidence. The review converged.
Critical checkpoints:
- Goal and evidence: the unconditional LocalTableValuedFunction constructor check rejects LOCAL and FILE-local before common-property parsing, backend selection, file discovery, or Lance schema access. BindSink rejects explicit LOCAL INSERT and analyzed EXPLAIN INSERT before sink-property validation and execution. Unit and SQL regression assertions cover these intended errors; they were inspected, not executed.
- Scope and clarity: two production call sites share one rejection method; the remaining changes adapt affected tests.
- Concurrency and locks: no new threads, shared mutable state, locks, or lock-order changes. Rejected requests never enter LOCAL discovery or sink execution.
- Lifecycle and initialization: no new resource ownership or cross-static initialization dependency.
- Configuration: no enable flag exists in this change; rejection is unconditional for every user.
- Compatibility: disabling LOCAL is the intentional SQL behavior change documented in the PR release note. No storage-format or serialized-function change is introduced.
- Thrift/protobuf: untouched; cross-release field/enum comparison is inapplicable.
- Parallel paths: registry, Nereids LOCAL, FILE-local, describe/explain, and explicit LOCAL sink paths reach the gates. Remote FILE dispatch and S3/HDFS sink validation retain their existing paths. Literal local Parquet metadata is a separate retained feature; its LOCAL-based wildcard expansion now has matching rejection assertions.
- Conditional checks and errors: the sink checks the normalized LOCAL name; constructor rejection has no privilege/format/property bypass. Nereids wrappers and statement execution propagate the failure to the SQL client.
- Test coverage: negative cases cover missing properties, csv/json/lance, FILE-local, privileged contexts, absence of backend calls, and INSERT/EXPLAIN analysis. Remote FILE coverage mocks provider construction; remote sink checks cover validation. The new account regression covers SELECT/EXPLAIN. These tests do not establish live remote IO or full command execution.
- Expected results: no .out file changes. The original ORC fixtures match the new named projections and count assertions (2 array rows; 25000 map rows). The regression harness skips unused removed tags, preserving later expectations. Retained deterministic value checks remain.
- Observability: the deterministic disabled-feature analysis error is sufficient for this rejection path; no new distributed operation needs logs or metrics.
- Transactions and persistence: no EditLog, persisted metadata, visible-version, or failover protocol changes.
- Data writes and atomicity: rejected LOCAL sinks cannot reach coordinator execution or FE deletion. Existing remote write flows are unchanged.
- FE/BE state: no new transmitted variable or wire definition.
- Performance and memory: constant-time rejection avoids LOCAL discovery work; no new scan, cache, allocation owner, or hot-path loop. Migrated fixtures use existing remote helpers; array/map uploads are removed after synchronous INSERT.
- Additional sweep: checked indirect metadata consumers, lazy Local optimizer test fixtures, uncalled nested-type LOCAL closures, remote provider routing, retained export completion/error checks, and setup dependencies. No additional substantiated issue remained.
Existing review context: comment 4371999865 is resolved by switching the duplicate-output binding fixture to S3 while retaining the ExprId assertions. Comment 4373692352 is resolved by adapting both local Parquet metadata wildcard cases to the disabled error. Neither is reposted; no existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
Limits: this was a static-only review. No builds, tests, or runtime storage checks were performed. LOCAL-only suites and LOCAL export readback comparisons are intentionally retired, so local-export content coverage is reduced; retained export completion/error and remote checks were inspected. No unresolved static candidate or uncovered changed file remains.
Review converged after 1 round(s).
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69592 Problem Summary: UUID export/readback forwards the Azure endpoint placeholder without its provider after the LOCAL fallback is removed. Resolve the Azure endpoint and include provider in the existing shared properties used by both EXPORT and the readback TVF. Keep all fixture data, export state/error handling and ordered result assertions unchanged. ### Release note None ### Check List (For Author) - Test: Both Groovy scripts compiled. Eight intercepted-SQL scenarios verified shared export/readback properties for Azure default/explicit endpoints and OSS, retained export-state handling and ordered golden-check invocations. No Doris SQL execution or live storage validation was performed. git diff --check passed. - Behavior changed: No (test data-loading adaptation only) - Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of all 38 changed files at 0f6e0e07e8d84329151a2b07a93b670f317b249c found no new actionable defect. The complete FE reviewer, complete regression reviewer and separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS; the review converged after 3 rounds, including the final frozen-payload sweep. All candidate investigations and initial risks have evidence-backed dispositions.
Existing findings were independently revalidated: 4371999865 is addressed by using S3 in the duplicate-output fixture while retaining the output-count and distinct-ExprId assertions; 4373692352 is addressed by expecting LOCAL denial in both Parquet metadata wildcard cases. Neither is reposted. No existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
The reviewed flows are SELECT/EXPLAIN -> Nereids or factory LOCAL construction (FILE-local delegates here) -> unconditional rejection, and INSERT/analyzed EXPLAIN -> UnboundTVFTableSink -> BindSink -> the same rejection before sink property validation, physical sink creation or execution.
| Critical checkpoint | Conclusion |
|---|---|
| 1. Goal and supporting tests | Named LOCAL and FILE-local reads reject before LOCAL property parsing, backend selection, discovery or Lance schema access. Explicit LOCAL sinks reject during analysis. Added unit/account assertions cover these boundaries by inspection; they were not executed. |
| 2. Scope and clarity | Two small production call sites share one denial method; remaining changes adapt dependent tests. |
| 3. Concurrency and locks | No new threads, shared mutable admission state, locks, lock ordering or lock-held IO. |
| 4. Lifecycle and initialization | No new owned resource, reference cycle or static initialization dependency. Lazy TVF suppliers cannot publish a successfully constructed LOCAL instance. |
| 5. Configuration | No configuration or opt-in is added; ADMIN/OPERATOR cannot bypass the unconditional rejection. |
| 6. Compatibility | Disabling LOCAL is the intentional SQL behavior change documented in the PR release note. Wire/storage formats are unchanged. Older FEs retain their prior admission behavior until upgraded. |
| 7. Thrift/protobuf | No definitions changed; release-branch field/enum comparison is inapplicable. |
| 8. Parallel paths | Nereids, legacy factory, DESCRIBE, FILE-local, metadata wildcard expansion, INSERT and ordinary EXPLAIN paths were traced. Remote FILE delegates and S3/HDFS sinks remain independent. Literal-local Parquet metadata and underlying filesystem implementations remain separate retained features. |
| 9. Conditions and errors | The sink checks its normalized name and shares the documented admission helper. Constructor denial has no privilege/format/property bypass. Planner/executor exception boundaries preserve the disabled diagnostic and report failure. |
| 10. Test coverage | New fixtures cover malformed properties, CSV/JSON/Lance, LOCAL/FILE, privilege independence, absence of backend calls, INSERT/EXPLAIN analysis and remote controls. Retained UUID scanner/pruning/strict variants, remote partition cases and export completion/error checks were inspected. |
| 11. Expected results | No .out files changed. Static ORC footer inspection confirms the migrated named projections and row counts: 2 array rows and 25000 map rows, matching unchanged golden blocks. Negative diagnostic substrings match the rejection. Framework tag lookup skips retired unique output blocks; retained value assertions remain. |
| 12. Observability | The explicit disabled-feature analysis error suffices for this admission failure. No new distributed operation requires logs or metrics. |
| 13. Transactions and persistence | No EditLog, persisted metadata, commit/publish, visible-version, delete-bitmap, replay or failover implementation changes. |
| 14. Writes and atomicity | LOCAL sinks fail before FE deletion, coordinator creation or writing. Existing remote write implementations are unchanged. Array/map fixture deletion follows synchronous INSERT in finally blocks. |
| 15. FE/BE variables | No new transmitted variable or sending/receiving path. Existing provider properties are reused for migrated remote fixtures. |
| 16. Performance and memory | Constant-time denial avoids LOCAL discovery work; no new growing allocation, repeated scan, hot loop or significant memory owner. Remote fixture setup reuses existing provider clients. |
| 17. Further recheck and core invariants | Rechecked indirect constructors, caches, lazy optimizer fixtures, unused nested-type LOCAL closures, provider routing, SDK availability, output tags and surviving setup/cleanup dependencies. No additional verified defect, unresolved candidate or uncovered changed path remains. No BE nullable, memory-accounting, committed-data visibility or MoW invariant is changed. |
Validation limits: static review only; no builds, unit tests, regression suites or live remote IO were run, and no submodules were initialized. ORC footer inspection establishes schema/count metadata, not decoded values or successful loading. The sink unit tests analyze the explain plan rather than executing the command; the FILE-S3 control mocks delegate construction. Cloud, runtime and rolling-upgrade behavior were not exercised. Eighteen LOCAL-dependent suites now return without assertions, and LOCAL export content/readback comparisons are retired, reducing ongoing regression protection. Retained remote checks depend on configured storage access. This review covers the named LOCAL admission change and does not establish universal local-filesystem isolation.
Review converged after 3 round(s).
There was a problem hiding this comment.
Static review of all 38 changed files at 0f6e0e07e8d84329151a2b07a93b670f317b249c: no new actionable findings. Converged after 2 review rounds, with two complete-scope reviewers and a separate risk reviewer returning NO_NEW_VALUABLE_FINDINGS on the unchanged final payload.
Both existing P2 findings are addressed on this head: 4371999865 by using S3 for the duplicate-output sink fixture while retaining the distinct ExprId assertions; 4373692352 by expecting disabled admission for both local metadata glob cases. No existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and tests: LOCAL construction rejects before common-property parsing, backend selection and file discovery; explicit LOCAL INSERT and analyzed EXPLAIN INSERT reject before sink-property validation. Inspected rejection tests cover FILE local delegation, malformed inputs, formats, privileged users and remote validation parity.
- Scope: production changes are limited to the common rejection helper and two admission sites; regression changes adapt affected consumers.
- Concurrency and locks: no new threads, shared mutable state, locks or lock-order changes; rejection precedes LOCAL runtime work.
- Lifecycle: no new production ownership or static-initialization dependency. Test mocks close, and migrated ORC objects are deleted in finally after synchronous insertion.
- Configuration: none added; there is no re-enable option or privilege exception.
- Compatibility: LOCAL SQL behavior intentionally changes as described in the release note; no stored format, function symbol or wire decoder changes.
- Thrift/protobuf: no changed definitions; cross-release field-ID comparison is not applicable.
- Parallel paths: direct LOCAL, FILE(local), factory/describe and INSERT/EXPLAIN routes were traced. Remote FILE/S3/HDFS paths remain separate. Literal-local Parquet metadata remains independent; local metadata glob expansion reaches the disabled constructor.
- Conditions: sink admission checks the normalized LOCAL name; the shared unconditional rejection documents its timing.
- Coverage: negative unit/SQL cases, retained export completion/error checks, remote UUID cases, HDFS/S3 partition cases and migrated ORC value checks were inspected. LOCAL-only success suites and local export value readbacks are intentionally retired.
- Expected results: no .out files changed. ORC COUNT replacements match existing first result cells (2 and 25000), unchanged fixture schemas and row counts; the output iterator permits omitted LOCAL tags without losing later remote assertions.
- Observability: the disabled analysis error propagates through existing statement error handling; retained export cancellation/errors remain visible. No new distributed operation requires metrics.
- Transactions and persistence: no EditLog, transaction, version-visibility, delete-bitmap or failover implementation changes.
- Writes and crashes: LOCAL sinks fail before physical writes/coordinator creation; remote write implementation is unchanged. No new production resource-leak or atomicity mechanism was identified.
- FE/BE variables: none added or serialized; existing provider/endpoint properties are retained in remote upload/read and export/reload paths.
- Performance and memory: admission is constant-time and avoids LOCAL discovery/schema work; no new large production allocation or BE memory/null/COW path.
- Other risks: exception boundaries, fixture/provider identity, skipped-suite contents, remaining LOCAL consumers and result-tag ordering were rechecked. All source questions were dismissed with evidence; no unresolved candidate or assigned source coverage gap remains.
Verification limits: this was static-only. No build, unit/regression execution, live S3/Azure upload, credential/network validation or submodule initialization was performed. Runtime behavior and the intentionally retired LOCAL success coverage are not claimed as tested.
Review converged after 2 round(s).
FE Regression Coverage ReportIncrement line coverage |
What problem does this PR solve?
Issue Number: N/A
Related PR: #69591, #69527
Problem Summary:
LOCAL TVF can discover files on a backend during analysis. Reject LOCAL unconditionally before property parsing, backend selection and file discovery. FILE TVF with local storage delegates to the same entry point. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same rejection before sink property validation. No configuration option enables these paths, including for ADMIN/OPERATOR. The underlying storage implementation is retained.
Tests assert rejection before backend access and preserve remote FILE/sink behavior where supported. Existing LOCAL-dependent success suites explicitly return while the feature is unavailable. Export completion/error checks remain active; LOCAL export-readback comparisons are retired. Remote HDFS/S3 partition cases, the existing UUID S3 fixture branch, and ORC value checks through S3 TVF using the original uploaded fixtures remain active where present on this branch.
Release note
LOCAL TVF and FILE TVF with local storage are disabled for all users. There is no configuration option to re-enable them. Explicit LOCAL TVF INSERT is also disabled.
Check List (For Author)
Validation: Checkstyle passed for all 5 changed Java files. Changed Groovy suites passed syntax parsing with Groovy 4.0.19. Whitespace checks passed. Runtime regression and formal local code review have not been performed.
Check List (For Reviewer who merge this PR)