Skip to content

[fix](tvf) Secure LOCAL TVF access (4.1) - #69528

Closed
OIiveirra wants to merge 24 commits into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-112-4.1
Closed

OIiveirra wants to merge 24 commits into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-112-4.1

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A

Related PR: #69526

Problem Summary:

LOCAL/FILE-local discovery could escape its secure root, while alternate writers and delegated Hadoop routes could bypass the LOCAL setting or access BE-local files through a remote TVF.

  • Add startup-only enable_local_tvf=false. Require ADMIN/OPERATOR before local discovery and during authorization; gate local writers and cleanup before side effects. Enabled reads require canonical containment under user_files_secure_path, including expanded paths and symlinks; rejection errors do not disclose resolved paths.

  • Validate Hadoop TVF routes in FE and at the actual consumer using one shared route interpreter. Check the branch's actual normalized Globber probes before discovery, then validate returned and supplied file paths literally in Java and native consumers. Freeze merged runtime defaults/resources, retain each native connection's context, and check actual paths on cache hits. Native writes use their create-parent route; cleanup retains literal semantics. Cycle detection tracks the resolved table and path, with a separate depth bound, so finite remote ViewFS aliases can revisit a table. Avro schema/data keep the same guard; data reads reuse the validated filesystem and preserve projection and split behavior.

  • Isolate guarded native caches by exact connection/authority/configuration identity. Guarded Java ViewFS requests own and close their mount clients, including partial initialization and close failures. Disable ViewFS's authority-folding inner cache only for guarded configurations. Preserve unmarked catalog behavior and known remote providers, including secure WASBS, while rejecting local overrides and unknown providers.

  • Admit REMOTE Lance storage bases once before publishing an opened handle. Full dataset-key comparisons still reject changed URI/version/options/policy; LOCAL handles continue rechecking mutable filesystem paths and bases before use.

  • For LOCAL Lance reads, validate SDK-decoded local paths and all manifest storage bases before schema/scanner use. Resolve every non-bare LOCAL URI through the existing metadata-only SDK parser, preserving URL and whole-string fallback identity before canonical containment. Preserve bare-path secure-root anchoring and local-provider aliases. Bind dataset cache identity to the policy. Require matching patched SDK headers/archive; keep a pair-level repair marker until Lance and Paimon both finish with the same selected Rust toolchain.

  • Guarded Hadoop TVF reads and native writes with LOCAL disabled reject JuiceFS: mutable namespace metadata can select or switch to an authoritative local backend. Class and scheme names cannot establish a remote-only contract across reloads. Ordinary unmarked/catalog routing retains its existing policy. Native TVF output paths use their dedicated Thrift field. Case variants and aliases after the actual native NUL-prefix/Configuration-trim projection are removed from backend properties, so output prefixes do not partition client profiles.

Release note

LOCAL TVF and FILE-local are disabled by default. Enable them with enable_local_tvf=true in fe.conf on every serving FE and restart. Enabled local reads still require ADMIN/OPERATOR and a nonempty BE user_files_secure_path. Hadoop TVF reads require proven remote storage even when LOCAL is enabled; use constrained local() for local reads.

Guarded TVFs also reject remote-backed JuiceFS namespaces until an enforceable remote-only backend contract is available. This is an intentional security restriction; ordinary unmarked catalog routing is unchanged.

Upgrade every serving FE/BE with the matching patched Hadoop runtime for full enforcement. Mixed-version enforcement is not guaranteed. Cross-mount Hadoop globs still require a proven remote route, such as an explicit remote fallback.

Rebuild the matching patched Lance headers/archive and Paimon with one Rust toolchain. Unmarked positive-version catalogs keep their existing warehouse policy. Local tree preflight is not an atomic filesystem snapshot.

Check List (For Author)

  • Test
    • Regression test (coverage added; full local execution incomplete)
    • Unit Test
    • Manual test
    • No need to test or manual test. Explain why:

Validation:

  • Prior source-bound JDK17 FE wrapper (unchanged Java inputs): 129 selected tests passed with fresh reports for every required suite, zero failures/errors/skips and frozen source inputs. Coverage includes real Hadoop dispatch through finite same-table aliases, native retained-context and literal cleanup admission, local-target/cycle rejection and the 32-step delegation bound; actual sender/profile/client controls for ordered mixed-case and NUL/whitespace path-key aliases, real Hadoop Globber probes with escaped literal metacharacters, literal returned/supplied paths, native route controls and request-owned ViewFS lifecycle. The Avro module ran explicitly with the BE's existing JDK17 module opens. It covers real rows, projection, nonzero splits, repeated data opens, production scanner-open failure and exceptional cleanup without a caller closing the failed scanner.

  • Prior source-bound Clang21/ASAN HdfsMgrTest: 11 selected tests passed for the native guard changes. Native Hadoop guard source and test bytes remain identical to that validated run.

  • Standard Clang21/ASAN filesystem and Lance selection: 31 tests passed, zero failures/errors/skips with fresh XML and frozen source. A pinned-SDK two-dataset collision reproduces the old wrong schema/row 7; the repaired schema, cache key and real scan select row 42. Out-of-root fallback rejects, and valid URI/relative-path controls pass. The SDK-invalid encoded-parent case is a rejection control. Coverage includes reused LOCAL registered-base admission after narrowing the secure root, initial REMOTE rejection before handle publication, snapshot/key checks, multiple fragment splits and row-id materialization. No live S3 throughput benchmark is claimed.

  • Prior source-bound Checkstyle inspected all 26 Doris Java source/test files in this PR; zero errors. Diff checks passed. The upstream Hadoop shadow source uses the repository's existing Checkstyle exclusion.

  • Existing native/SDK validation remains applicable to the unchanged paths: selected BE filesystem/Lance tests, SDK parser Rust/C API tests, wrapper/ABI checks and Lance/Paimon build controls. This follow-up changes only reused Lance admission and its existing test file. Java, native Hadoop guard and SDK inputs are unchanged from their recorded runs; the affected Lance behavior is covered by the fresh BE selection. Optional file+uring scans were not executed successfully because this host denies io_uring setup.

  • Targeted clang-format16 and Clang21 analysis were performed for the earlier native changes. The scoped-lock checker crash and existing unmatched NOLINTEND limit the static analysis; complete clang-tidy is not claimed.

  • Full SQL/Groovy regression, complete community CI and mixed-version execution were not performed locally.

  • Behavior changed:

    • No.
    • Yes. Default-off local TVF access, canonical containment and effective filesystem route enforcement.
  • Does this need documentation?

    • No.
    • Yes. Startup setting and upgrade guidance need documentation; documentation PR pending.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: Add the startup-only enable_local_tvf setting, defaulting to false. Reject LOCAL TVF and FILE TVF with local storage before property parsing or backend access. Explicit opt-in restores each branch's existing behavior. Dedicated regression deployments opt in to retain LOCAL coverage.

### Release note

LOCAL TVF and FILE TVF with local storage are disabled by default. Set enable_local_tvf=true in fe.conf on every query-serving FE and restart those FEs to restore the existing behavior.

### Check List (For Author)

- Test: 7 focused FE unit tests passed with zero failures or skips; focused FE Checkstyle passed. SQL regression coverage added; runtime validation recorded separately.
- Behavior changed: Yes. Local TVF access requires explicit startup opt-in.
- Does this need documentation: Yes. Release note documents the default and opt-in.
Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:45
@OIiveirra
OIiveirra requested a review from yiguolei as a code owner October 23, 2026 09:45
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Validate canonical roots and resolved paths, check ADMIN/OPERATOR before file access, validate LOCAL Lance schema/scan paths, and preserve LOCAL/S3 range types. Limit Lance validation to the existing LOCAL TVF version-zero contract so fixed-version local catalogs retain their warehouse paths. Reject LocalProperties selected through S3 Lance before dataset opening.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The default-disabled startup gate is retained.

### Check List (For Author)

- Test: 26 focused FE tests, Java Checkstyle and changed-line clang-format 16 passed. BE execution is blocked by the installed Lance-C SDK missing the existing lance_scanner_set_distance_range API; no BE runtime pass is claimed.
- Behavior changed: Yes, enforce secure local paths and authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](fe) Disable LOCAL TVF by default (branch-4.1) [fix](tvf) Restrict local paths and require explicit startup opt-in Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](tvf) Restrict local paths and require explicit startup opt-in [fix](tvf) Secure LOCAL TVF access (4.1) Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary:
S3 Lance could open a local dataset through BrokerProperties because the
entry point rejected only LocalProperties. Validate the final dataset URI
before loading metadata. Allow the pinned SDK's remote providers and reject
local aliases and malformed authorities that could fall back to local paths.
Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS
HTTP locations working without changing catalog or non-Lance paths.

### Release note

S3 Lance rejects local dataset URIs and malformed remote URI authorities
before opening a dataset, regardless of storage-property selection.

### Check List (For Author)

- Test: 31 focused FE unit tests passed on this branch; changed Java
  files pass Checkstyle and git diff --check passes.
- Behavior changed: Yes. S3 Lance requires a valid supported remote URI.
- Does this need documentation: No additional documentation for this
  security validation; the parent PR's configuration documentation is pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two verified findings require changes: [P1] signed-zero port syntax still lets S3 Lance bypass the new LOCAL admission policy through the SDK's local-path fallback; [P2] LOCAL Lance file-URI canonicalization changes percent-encoded dataset identity. Both are reported inline. There were no existing inline threads or blocking comment IDs, and no additional user-provided focus points.

All 20 changed files were reviewed, with FE, BE and independent boundary-risk coverage. The complete LOCAL/FILE/S3 entrypoint -> property normalization -> schema/metadata discovery -> scan-range -> dataset-open/cache chain was traced against this head and the pinned SDK sources.

Individual critical-checkpoint conclusions:

  1. Goal and proof: default-disable, early/repeated privileges and canonical glob containment are implemented, but the two findings leave security and URI compatibility incomplete. Added tests exercise the intended paths; they miss signed-zero ports and escaped file-URI collisions.
  2. Scope: production edits are focused on admission, canonicalization and Lance identity, with corresponding tests and regression opt-ins.
  3. Concurrency and locks: no new threads, shared mutable state or lock order; existing heavy-work RPC/scanner contexts perform filesystem work without newly introduced critical sections.
  4. Lifecycle/static initialization: existing RAII, close paths, per-reader dataset ownership and lazy Session initialization remain intact; no new cross-TU static dependency or ownership cycle.
  5. Configuration: enable_local_tvf defaults false and is startup-only because ConfField.mutable defaults false and setMutableConfig rejects it. Every serving FE must opt in and restart, as described.
  6. Compatibility: no storage/wire format change; legacy scan-level file-type fallback and positive-version catalog behavior remain. Encoded file-URI compatibility regresses as reported.
  7. Thrift/protobuf: no definitions changed, so release-branch ID comparison is not applicable.
  8. Parallel paths: LOCAL and FILE-local share constructor/authorization guards; schema and execution use FILE_LOCAL/version zero; S3 and catalog snapshots use fixed positive versions. The S3 parser bypass remains actionable.
  9. Conditions: range-type precedence and version-zero exceptions are documented and backed by actual producer invariants; no user TVF option supplies a positive local version.
  10. Test coverage: inspected FE positive/negative admission and URI tests, BE glob/root/symlink/schema/relative-scan tests, and SELECT/EXPLAIN Groovy gate checks. Add cases for both inline findings. No build or test was run under this static-only contract.
  11. Expected results: no generated output snapshots changed; the new four-row expectation matches the existing Lance fixture usage. The Groovy suite checks gate/backend rejection, not a successful end-to-end read.
  12. Observability: existing profile/error paths remain; generic containment errors intentionally redact expanded/resolved target names. No new critical metrics requirement identified for admission-only work.
  13. Transactions/persistence: no EditLog, transaction, visible-version, delete-bitmap or master-failover behavior changes.
  14. Writes/atomicity/crashes: production changes are read-only; no new data-write transaction or persistent cleanup state.
  15. FE/BE variables: an existing file_type field is now retained on Lance TVF execution ranges; schema already tags it and legacy scan-level handling is preserved.
  16. Performance/memory: validation is per TVF/schema/split rather than per row; no material new allocation, cache or hot-loop regression substantiated. Glob root recanonicalization is bounded per match.
  17. Further recheck: candidate triggers, upstream guarantees, downstream SDK dispatch, tests and all changed paths were rechecked; no other introduced actionable defect was verified.

Review limits: static source review only; author-reported runs were not independently executed, and actual remote reads, Groovy integration and mixed-version runtime behavior remain untested here. Pre-existing descendant-symlink/manifest child confinement and filesystem-mutation races were examined and excluded from new findings because this PR did not introduce or expand them. No unresolved review candidate or uncovered changed-file scope remains.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 33.33% (9/27) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (40/40) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.04% (32028/42684)
Line Coverage 59.41% (359163/604593)
Region Coverage 56.07% (299258/533763)
Branch Coverage 56.98% (135883/238489)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69528

Problem Summary: Apply the startup-only LOCAL gate to INSERT and EXPLAIN INSERT. Escape the canonical secure root as a literal glob prefix. Validate raw S3 Lance ports and IPv6 before SDK access; decode file URI paths once after removing query and fragment while preserving bare paths.

### Release note

LOCAL TVF INSERT and EXPLAIN INSERT honor enable_local_tvf. Secure directories containing glob metacharacters remain usable.
LOCAL Lance preserves percent-encoded file URI identity; malformed remote Lance authorities are rejected before SDK access.

### Check List (For Author)

- Test: Unit Test: 35 FE and 11 BE selected tests passed; relevant Java Checkstyle, clang-format 16, and git diff --check passed. Regression assertions added, full Groovy not run.
- Behavior changed: Yes, enforce the existing LOCAL startup setting at the write entry and preserve valid local path identity.
- Does this need documentation: Yes, the original PR release note describes the startup opt-in requirement; documentation PR remains pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

1 similar comment
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One verified P2 correctness regression remains: a LOCAL Lance file URI containing a symlink followed by .. can select a different dataset inside the secure root. The inline finding gives the exact path, SDK behavior, and fix direction. No additional P0/P1 finding was verified.

Reviewed head 398acc158bbb9e2a9525a0960a9186f351d73fc6 across all 22 changed paths and related callers. Read flow: LOCAL/FILE -> opt-in and read privileges -> properties/backend -> glob or schema RPC -> tagged Lance range -> secure resolver -> dataset open. Write flow: INSERT -> existing write authorization -> sink binding/LOCAL opt-in -> execution; EXPLAIN shares the sink-binding gate.

Critical checkpoint conclusions:

  • Goal and data correctness: default opt-in, early access checks, glob containment, and remote Lance admission are implemented; the inline URI-normalization defect prevents complete path-identity preservation.
  • Scope and clarity: changes stay within admission, path handling, range identity, tests, and regression configuration; no unrelated rewrite was identified.
  • Concurrency and locking: no new threads, production shared mutable state, locks, or lock order. Glob retains its existing heavy-work-pool execution.
  • Lifecycle and initialization: validation precedes schema/cached opens; existing dataset/session ownership remains intact, with no new cross-translation-unit static dependency.
  • Configuration: enable_local_tvf defaults false and is intentionally immutable at runtime. Each query-serving FE must be configured and restarted. Seven regression deployments explicitly opt in.
  • Compatibility: default disablement is an intentional documented change. No storage format or function signature changes; fixed-version catalog warehouses retain their routing. File-URI compatibility has the reported regression.
  • Thrift/protobuf definitions: none changed, so cross-release ID/enum comparison is inapplicable.
  • Parallel paths: checked LOCAL and FILE-local, SELECT and EXPLAIN, INSERT and EXPLAIN INSERT, remote sinks, schema and scan, shared storage, legacy scan tags, and fixed-version catalogs. Dependent metadata glob discovery inherits LOCAL admission; the flag is not a blanket policy for every local-file feature.
  • Special conditions: version-zero LOCAL ranges receive containment checks; positive catalog snapshots retain operator-managed locations; per-range file type overrides legacy scan-level type. Remote scheme/port conditions have explicit rationale.
  • Test coverage: inspected enabled/disabled admission, privilege revocation, FILE delegation, sink binding, remote normalization/Broker rejection, glob traversal/symlink/root variants, escaped URI identity, schema/scan, and catalog exceptions. The symlink-plus-URI-dot-segment case is missing. Real remote reads and full integration execution are not proven by mocked unit tests.
  • Expected results: no golden output files changed; no incorrect added expected result was identified statically.
  • Observability: admission errors identify configuration or privilege requirements; containment errors intentionally hide expanded names. Existing Lance URI/version profiles and error propagation remain sufficient for these setup paths.
  • Transactions and persistence: no transaction, EditLog, persisted metadata, visibility-version, or delete-bitmap logic changed.
  • Data writes and crash behavior: write changes add an admission check before execution; existing atomicity, cleanup, and crash behavior are not altered.
  • FE/BE propagation: no new wire variable; scan ranges now preserve the existing file_type field, as schema requests already do. Both local schema and execution use the same resolver.
  • Performance and memory: new work occurs at glob/dataset setup rather than per row; no new large allocation or ownership path was identified.
  • Other issues and user focus: no additional user focus was supplied; complete and risk-focused rechecks found no further verified defect.

Existing findings were independently revalidated: signed-zero ports, 4361447979 are now rejected by raw ASCII-digit validation, and escaped file URIs, 4361447989 now decode once with collision coverage. Neither original instance still applies; neither is reposted, and there are no confirmed existing P0/P1 blockers.

Static review only: no builds, tests, source modifications, or submodule initialization were performed. All assigned scopes and candidates have explicit dispositions. The review converged in round 2 after all three reviewers rechecked this exact final payload and returned NO_NEW_VALUABLE_FINDINGS. Runtime validation and the missing regression case remain testing limits, not unresolved review candidates.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #69528 at 398acc1. No new actionable findings were verified. The review converged after one round of complete BE/FE review and a separate cross-layer risk review.

Existing review disposition:

  • The dot-segment/symlink dataset-identity issue in comment 4365938520 still applies: file-URI path segments reach filesystem canonicalization before URL lexical normalization. It is already reported and is not duplicated here.
  • The percent-decoding issue in comment 4361447989 is fixed for the reported escaped-name collision: actual file: paths are decoded once, bare percent paths stay literal, and both schema discovery and scanning use the canonical result.
  • The signed-zero port P1 in comment 4361447979 is fixed by checking the original port characters before metadata loading. No previously reported P0/P1 was confirmed to remain applicable, so existing_blocking_comment_ids is empty.

Critical checkpoints:

  1. Goal and behavior: the default-off LOCAL switch guards LOCAL/FILE-local reads before property parsing or backend access and independently guards INSERT/EXPLAIN sink binding. Canonical containment covers glob matches and LOCAL Lance schema/scan opens; the existing URI normalization P2 remains as noted above.
  2. Scope and clarity: changes are confined to admission, URI/path validation, existing range-type propagation, focused tests, and pipeline opt-ins.
  3. Concurrency and locks: no new threads or lock acquisition/order. New path values are request-local; shared Lance session/cache synchronization is unchanged.
  4. Lifecycle/static initialization: existing schema/dataset ownership and cleanup remain; no new cross-translation-unit initializer dependency or ownership cycle.
  5. Configuration: enable_local_tvf is immutable and false by default; the error documents restarting every FE. Cloud and non-cloud FE use the same check, and seven test pipeline configs explicitly opt in.
  6. Compatibility: no wire or storage-format definition changes. The existing optional range file_type is now set explicitly; the BE retains scan-level fallback. Positive-version catalog scans preserve their configured URI. Older FEs do not gain the new admission policy during a mixed-version rollout.
  7. Thrift/proto: none modified; cross-release field/enum comparison is inapplicable.
  8. Parallel paths: checked LOCAL/FILE aliases, regular glob/direct Lance reads, schema/scan/cached opens, INSERT/EXPLAIN, catalog/TVF, and remote/local range precedence.
  9. Conditions: comments explain explicit-range precedence and the version-zero LOCAL versus positive-version catalog distinction. Final remote Lance validation runs after storage normalization, including Broker passthrough.
  10. Test coverage: added FE tests cover gate/auth ordering, aliases, sink binding, storage types and malformed/valid remote URI cases. Added BE tests cover canonical containment, glob-root literals, symlink escape rejection, relative Lance reads and escaped dataset-name collisions. The regression suite checks admission in both configuration states. The existing dot-segment collision lacks a regression case in this patch.
  11. Expected results: inspected assertions check canonical keys, schema discovery, scan row IDs, explicit failures and absence of premature backend/metadata calls; no golden output files changed. Tests were not executed.
  12. Observability/errors: validation failures propagate explicitly; expanded out-of-root names are hidden. Existing Lance open timers, URI/version profiles and session metrics remain. No new distributed operation needs separate instrumentation.
  13. Transactions/persistence: no EditLog, transaction, visibility or failover-persistence logic changed.
  14. Writes/atomicity: the patch changes sink admission; existing ADMIN/global-LOAD execution authorization and writer/commit behavior remain. No new crash-cleanup path is introduced.
  15. FE/BE propagation: each Lance TVF execution range carries the TVF's existing storage type, while schema requests already carry it; no new transmitted variable.
  16. Performance: path resolution occurs per path/split/open, outside row/batch decoding. No new per-row allocation, lock-heavy work, or change to pruning/materialization was found.
  17. Other core invariants: no changed MoW/delete-bitmap, nullable-column, decoder, or committed-data visibility behavior. Further parser/cache/admission and SDK-provider rechecks produced no additional verified issue.

There was no additional user-provided focus. All changed paths and the three initial risk areas were covered and given a disposition; no unresolved candidate or uncovered changed scope remains. This was a static-only review: no builds, tests, runtime parser experiments, source changes, or submodule initialization were performed. Convergence means the review found no additional valuable issue after rechecking; it does not resolve the pre-existing P2 thread above.

Review converged after 1 round(s).

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69528

Problem Summary: LOCAL Lance canonicalization followed symlinks before URL dot-segment normalization, so a valid file URI could open a different in-root dataset. Uppercase file schemes were rejected. Normalize the actual file URI scheme, separators and raw dot segments before one-pass decoding and filesystem containment; preserve bare path behavior.

### Release note

LOCAL Lance preserves file URI dataset identity for dot segments, case-insensitive schemes and raw URI separators while retaining secure-root containment.

### Check List (For Author)

- Test: 13 selected BE filesystem/Lance tests passed with fresh XML, zero failures/skips and unchanged sources. New regressions fail before the fix, including actual schema and row-value mismatch. Changed-line clang-format16 and git diff --check passed. FE source is unchanged.
- Behavior changed: Yes, restore valid file URI identity without changing bare paths, remote TVFs or fixed-version catalogs.
- Does this need documentation: The original PR documents the LOCAL startup opt-in; no additional public configuration or syntax is introduced.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 22 changed paths at 71be77f48222334cf3311ed71ba9f4b6142043e4: one new P2 finding, inline below. A file: URI with raw trailing whitespace can select the wrong in-root Lance dataset. No new P0/P1 finding was verified.

The existing signed-zero-port finding (4361447979) and percent-decoding finding (4361447989) are fixed for their reported scenarios. The reported file-URI example in dot/symlink thread 4365938520 is fixed, but its bare absolute-path variant remains: the SDK lexically removes jump/.. before following symlinks, while the new canonicalizer follows the symlink first. This is the same already-reported mechanism/location and is not reposted. The pinned path_abs implementation establishes that bare-path baseline. No existing P0/P1 remains applicable. There were no additional user-provided focus points.

Individual critical-checkpoint conclusions:

  • Goal and proof: LOCAL/local FILE reads and LOCAL sinks now require opt-in; read authorization precedes backend access. Added tests cover admission and secure-path behavior, but omit the whitespace form in the new finding and its rootless-URI metadata variant.
  • Scope and clarity: the change is focused on admission, remote-URI validation, Lance range identity, canonical containment, and supporting tests/configuration.
  • Concurrency and locks: statement-local guards and path buffers add no shared mutable state or locks. Glob/schema work uses the existing worker pool; no new lock-order or heavy-under-lock issue was found.
  • Lifecycle and initialization: schema handles retain RAII cleanup; reader-owned datasets and split-owned scanners keep their existing close order. No new cycle, escaping reference, or cross-unit static dependency was found.
  • Configuration: enable_local_tvf defaults false and is nonmutable; the documented every-FE restart matches ConfigBase enforcement. Seven regression pipelines explicitly opt in.
  • Compatibility: no storage-format or function-signature change. Existing range tags and legacy scan-level fallback are handled; positive catalog snapshots retain their operator-configured URI. The new finding and the already-known bare-dot variant are verified input-compatibility defects. A deployed rolling-upgrade matrix was not executed.
  • Thrift/protobuf: no definition, ID, enum, or required-field change; release-branch ID comparison is inapplicable.
  • Parallel paths: direct LOCAL, delegated FILE, INSERT/EXPLAIN, schema/scan/cache opens, remote providers, and fixed-version local catalogs were traced. The sink retains its existing ADMIN/LOAD execution check.
  • Special conditions: comments and producers support the LOCAL-plus-version-zero TVF distinction and per-range precedence. Remote validation checks the final normalized URI, including Broker passthrough.
  • Test coverage: FE tests inspect rejection before I/O, privilege changes, sink analysis and range tags; BE tests cover canonical roots, glob escapes, URI decoding/dots and real schema/row expectations. The account regression covers configured admission. Whitespace-URI scan collisions and rootless-URI DESCRIBE collisions are missing; bare dot/symlink controls need a differential check against the SDK baseline.
  • Expected results: no .out artifact changed; new assertions match the inspected branches, but bare dot/symlink expectations also need alignment with the SDK baseline noted above. Tests were inspected, not run.
  • Observability: admission errors name the control/privilege; failed glob containment hides expanded external names; existing dataset/version errors, timers and profile strings remain available.
  • Transactions and persistence: no EditLog, commit/publish, visible-version, delete-bitmap, persisted-metadata, or failover-state change.
  • Writes and crash behavior: the added sink gate runs before backend scheduling. No writer, atomicity, transaction or durable-resource lifecycle change was introduced.
  • FE-to-BE passing: no new variable; existing file_type is now explicit for Lance scan ranges as it already was for schema requests. Both producers and legacy fallback were checked.
  • Performance and memory: added URI work is bounded by path length and occurs outside row loops. No significant retained allocation or material hot-path regression was identified; no benchmark was run.
  • Other invariants: no nullable-column/materialization change or new version/delete-bitmap handling; failures propagate before opening. The final missed-path sweep found no additional independent issue beyond the new parser finding and deduplicated historical variant.

Coverage limit: this was a static-only review, as required. No build, test, sanitizer, live-storage or performance execution was attempted. All changed source/test/configuration paths and identified risk questions have evidence-based dispositions; no unresolved review candidate or source-coverage gap remains.

Review converged after 3 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69528

Problem Summary: LOCAL Lance file URLs without a slash were resolved relative to the secure root, URI whitespace was treated as literal data, and bare dot segments followed symlinks before SDK lexical normalization. Match file URI preprocessing and absolute-root semantics, and normalize bare paths only at the LOCAL Lance entry before canonical containment. Preserve literal bare whitespace, encoded spaces, glob behavior and existing remote/catalog routing.

### Release note

LOCAL Lance preserves dataset identity for rootless file URLs, raw URI whitespace and bare dot segments while retaining secure-root containment. This does not add FE execution support for rootless Hadoop paths.

### Check List (For Author)

- Test: 13 selected BE filesystem/Lance tests passed with fresh XML, zero failures/skips and unchanged sources. New regressions fail before the fix, including actual schema and row-value mismatch. Changed-line clang-format16 and git diff --check passed. FE source is unchanged.
- Behavior changed: Yes, restore valid file URI identity without changing bare paths, remote TVFs or fixed-version catalogs.
- Does this need documentation: The original PR documents the LOCAL startup opt-in; no additional public configuration or syntax is introduced.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 0c4fa140e43bd3be71e097713f0cbd3f25d38c8c against the supplied aggregate diff and all 22 changed files. Two new verified P2 compatibility findings are inline: bare Lance home aliases and initial file-URI drive segments can select a different dataset or reject a valid in-root read. No new P0/P1 was verified.

Existing review context was independently revalidated: signed-zero/IPv6 P1 comment 4361447979 and percent/dot/whitespace P2 comments 4361447989, 4365938520 (including the bare-dot reply), and 4366084340 are fixed for their reported triggers. None is reposted; no existing P0/P1 remains applicable. No additional user-provided focus was supplied.

Critical checkpoints (static conclusions):

Checkpoint Conclusion
Goal and proof Default-off LOCAL/FILE-local reads, early ADMIN/OPERATOR checks, LOCAL sink admission and secure-root enforcement are implemented. The two inline cases leave dataset identity incomplete. Added tests cover the reported fixes but omit these two inputs.
Scope and clarity Changes stay within admission, path resolution, range identity, tests and deployment opt-in. Parser compatibility is the remaining correctness concern.
Concurrency and locks Query analysis and BE schema/scanner setup use call-local normalization state and startup configuration. No new thread, shared mutation, lock, heavy locked operation or lock-order change. Glob uses its existing worker pool; session initialization keeps call_once.
Lifecycle and static initialization Validation precedes schema/open/cache acquisition. Dataset/scanner RAII, shared-session ownership, split cleanup and key equality checks remain intact; no new global initializer or ownership cycle.
Configuration enable_local_tvf defaults false and is startup-only; ConfigBase rejects dynamic mutation. Every query-serving FE must be configured and restarted. Seven regression deployments explicitly opt in.
Compatibility and rolling upgrade No new wire field, function signature or storage format. Existing optional range tags and legacy scan tags are handled; positive-version catalog warehouses retain their URI. Old processes retain their old policy until upgraded/restarted.
Thrift/proto definitions None changed; cross-branch field/enum allocation comparison is inapplicable.
Parallel paths Checked LOCAL/FILE delegation and reauthorization, ordinary glob versus Lance schema/scan/cache, INSERT and EXPLAIN sink binding, remote S3 metadata/fragments, and fixed-version catalogs. No additional scoped omission verified.
Special conditions Range type overrides legacy tags; version zero identifies the LOCAL whole-dataset path. URI-only preprocessing/decode and bare lexical normalization have matching callers. Empty/NUL inputs and invalid roots fail before opening.
Test coverage Read all changed BE/FE tests and Groovy assertions: disabled/enabled policies, privileges/revocation, remote noninterference, tags/catalogs, symlink/glob escapes, literal roots, path collisions and explicit errors. New tilde/drive cases need coverage. No builds or tests were run.
Changed expected results No generated expected-output file changed. Inspected schema/key/row assertions and expected exceptions match their traced branches.
Observability Admission errors explain the policy; path denial suppresses expanded filenames. Existing schema errors and dataset/version profile context remain; no new distributed metric need identified.
Transactions and persistence No EditLog, rowset/version publication, delete bitmap, transaction state or persisted metadata is changed; master-failover semantics remain unchanged.
Writes and crash behavior The sink change adds pre-execution admission. Existing ADMIN/LOAD authorization and writer atomicity, ownership and cleanup paths remain unchanged.
FE/BE variables No new variable. TVF execution now explicitly serializes LOCAL/S3 range identity; schema RPC already tags it. Generic range construction occurs before the TVF override.
Performance and memory Normalization is setup/listing work bounded by path/result sizes, with no per-row allocation or new retained cache. No changed decoder, mapper, predicate/delete or nullable-column behavior. New Status errors propagate before data publication.
Further issue sweep Rechecked SDK pins/provider selection, remote parser fallback, URI reparsing, FILE wrappers, sink entry points and catalog exceptions. The complete verified new finding set is the two inline P2s.

Static-review coverage includes all changed paths and the affected caller/downstream mechanisms. Runtime parser fuzzing, builds, test execution, full Groovy regression and actual remote-provider reads were outside this review's static-only contract. These limits are not claims of executed validation. Review convergence is assessed separately from the compatibility findings and delivery.

Review converged after 2 round(s).

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
Comment thread be/src/io/fs/local_file_system.cpp Resolved
…ntity

    ### What problem does this PR solve?

    Issue Number: N/A

    Related PR: apache#69528

    Problem Summary: Remote-named TVF sinks could select the built-in local JNI writer while LOCAL was disabled. Apply the shared LOCAL admission check before sink analysis and final sink binding, matching the loader's class-name representation. LOCAL Lance also needs to preserve the pinned SDK's home aliases and file-URL drive components before canonical containment; add distinguishable schema and scan fixtures without changing ordinary glob or remote/catalog routing.

    ### Release note

    The LOCAL startup setting covers sinks using the built-in local JNI writer under S3/HDFS names. LOCAL Lance preserves supported home aliases and file URL drive identity inside the secure directory.

    ### Check List (For Author)

    - Test: 39 selected FE tests and 15 selected BE filesystem/Lance tests passed through standard wrappers with fresh XML, zero failures/errors/skips and unchanged sources. Java Checkstyle, changed-line clang-format16 where applicable and git diff --check passed.
    - Behavior changed: Yes, enforce the existing default-disable policy and retain valid local dataset identity.
    - Does this need documentation: The original PR documents the startup opt-in; this supplement adds no new configuration or syntax.
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 98.67% (148/150) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.48% (26987/43896)
Line Coverage 46.17% (281419/609592)
Region Coverage 41.79% (222672/532863)
Branch Coverage 43.35% (103586/238933)

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: one new P1 finding. The new LOCAL disable switch still admits a native HDFS sink configured with Hadoop's local filesystem, allowing a global LOAD user to create BE-local files while enable_local_tvf=false. The inline finding includes the SQL trigger and verified planner-to-native-writer path.

Reviewed head 66d2b918a4cbcc4a5dca2271396700da91bbe365 against the supplied aggregate diff. All 23 changed files and affected read/write/schema/scan paths were covered. Review convergence is separate from correctness: the review converged after two rounds with this P1 outstanding.

The existing signed-zero/IPv6 P1 is fixed by original-port ASCII validation and IPv6 checks before metadata loading; no existing P0/P1 remains applicable. Existing percent-decoding, URI/bare dot-segment, whitespace and home-alias findings are addressed. The earlier exact triple-slash drive claim is contradicted by the pinned parser; related single-slash/rootless/authority-drive forms are now handled. These issues were independently rechecked and were not reposted. No additional user-provided focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: direct LOCAL/FILE reads and local/JNI writes are gated; native Hadoop local writes remain a gap. New tests exercise direct admission and path identity but omit this native sink case.
  2. Scope and clarity: changes are focused on configuration, admission, range tagging and local path resolution, with matching test/config updates.
  3. Concurrency: no new threads, locks or shared mutable query state; existing glob/schema workers and sequential reader ownership remain. No new lock-order or heavy-under-lock problem found.
  4. Lifecycle and memory: resolver temporaries, the thread-safe passwd fallback and schema dataset ownership introduce no identified leak, ownership cycle or cross-TU initialization dependency; reader/session cleanup remains unchanged.
  5. Configuration: enable_local_tvf is intentionally startup-only and defaults off; messages require every-FE restart. Seven pipeline configurations explicitly opt in.
  6. Compatibility: default-off and privileged local-read admission are intentional behavior changes. Existing per-range tags and legacy scan-level fallback are compatible; pinned catalog snapshots retain their routing. LOCAL latest admission covers bare paths and file URIs; other SDK-local aliases were reviewed as outside that interface.
  7. Protocol definitions: no Thrift/protobuf definition, field ID, enum or storage format changed.
  8. Parallel paths: LOCAL/FILE, SELECT/EXPLAIN, INSERT/EXPLAIN, direct sink binding, JNI class aliases, ordinary glob and Lance schema/scan paths were traced; the verified native-Hadoop omission is the finding.
  9. Conditional checks: FILE_LOCAL/version-zero selects LOCAL latest datasets; positive-version catalogs and remote ranges preserve their URI/options. Comments explain normalization order, legacy tags and JNI slash/NUL handling.
  10. Test coverage: FE and BE assertions cover admission, authorization, backend routing, empty/NUL/remote/escape inputs and path-identity collisions with distinguishable schemas/rows. Native Hadoop local-provider rejection lacks coverage.
  11. Expected results: no golden output file changed. Reviewed new schema/row/error assertions and cleanup/config restoration; no incorrect expected result identified.
  12. Observability: admission errors identify configuration/privilege failures; containment errors hide expanded external names. Existing dataset error/profile context remains, with no added per-row logging.
  13. Transactions/persistence: no visibility, delete-bitmap, EditLog/replay, checkpoint or master-failover protocol changed.
  14. Writes and crashes: existing nontransactional TVF writer/cleanup behavior is unchanged; the new admission guarantee is incomplete as reported. No separate introduced atomicity or crash-lifecycle defect found.
  15. FE/BE propagation: Lance ranges now carry the TVF's LOCAL/S3 identity; schema requests already do so. Both schema open and scan dataset-key creation use the same validated local resolver.
  16. Performance and further risks: parsing/canonicalization runs at file/dataset setup, not row decoding. No substantiated additional performance or materialization defect found after the missed-issue sweep.

Validation limits: static review only; no builds, tests, runtime parser comparisons or deployment checks were executed. Author-reported test runs were treated as context, not reproduced results. Unchanged SDK external-manifest-base semantics were inspected and excluded as pre-existing. No unresolved PR candidate or assigned coverage gap remains.

Review converged after 2 round(s).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of #69528 at fb92e7d found one new P2: guarded ViewFS listings retain distinct S3A child clients across paths without bounded cleanup. The inline comment describes the reachable configuration and ownership chain.

All 64 changed paths and their affected callers were covered. Existing P1 comments 4366428146 (native Hadoop writes), 4366880439 (Hadoop reads), 4366880442 (Lance manifest bases), and 4361447979 (signed-zero ports) were independently revalidated. Their original and recorded follow-up variants are addressed on this head; none was confirmed still applicable, so no existing blocking IDs are retained and no known finding is reposted.

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and proof Default-off LOCAL admission and effective Hadoop/Lance storage checks are present; positive/negative test sources cover the main paths. The new child-client leak still needs correction.
Scope and reuse The broad security change uses shared FE/runtime/native policy helpers and SDK metadata APIs; no unrelated change identified.
Concurrency and locks HdfsMgr protects cache operations; connection/JNI/path work occurs outside its mutex on OS threads. No new lock-order defect found.
Lifecycle and static initialization Dataset/JNI temporary ownership and failure cleanup were checked. The inline finding identifies ViewFS child ownership loss. No new static initialization cycle found.
Configuration enable_local_tvf intentionally defaults false and requires restarting every FE; regression pipelines explicitly opt in.
Symbol/storage compatibility SDK APIs are additive; matching headers/defined exports and paired Rust publication are gated. No persisted format change. Older serving binaries do not enforce the new admission markers.
Thrift compatibility Compared TLanceFileDesc on HEAD, master and branches 3.1/4.0/4.1/4.2: optional id7 is free and existing field meanings agree.
Parallel paths LOCAL/FILE/S3/HDFS reads, INSERT/EXPLAIN/cleanup, inherited providers, native schema/data/cache, Avro and Lance scanner/index/row-id paths were traced.
Special conditions Legacy version-zero LOCAL requests, marked fixed snapshots, URI/plain-path distinctions, local aliases and enabled-write opt-in have explicit handling.
Tests Inspected privilege, route, parser, fixture, cache identity and SDK/build failure controls. Request-scoped ViewFS child cleanup is missing coverage.
Changed results No .out result changed; binary manifest provenance/generator and expected error cases were inspected without executing them.
Observability Existing errors and profile/log context remain; no new per-row logging issue found. Client-cache cardinality is relevant to the inline defect.
Transactions and persistence No Doris EditLog, failover, visibility or delete-bitmap state changed.
Writes and crash behavior Admission precedes existing cleanup/writer operations; SDK interrupted publication is fenced by persistent repair markers. Existing write protocols are retained.
FE/BE variables is_tvf and Hadoop markers are propagated through the applicable schema, scan and Avro channels; user policy aliases are removed before internal injection.
Performance No new per-row work identified. LOCAL tree checks run during admission; the verified cross-request client retention is reported inline.
Other concerns No additional user focus was supplied. Remaining parser/symlink/cache suspicions were accepted, deduplicated or dismissed with recorded evidence.

Validation limits: static source review only. No builds, tests, runtime probes, benchmarks, submodule initialization or repository source changes were performed. Optional remote-service/io_uring controls and binary fixture behavior were not executed. These limits do not leave an unresolved source-review candidate.

Review converged after 2 round(s).

OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection. Guarded Avro data readers reuse the already validated owned filesystem, retaining projection and split behavior; reader cleanup releases every stream and filesystem after failed opens and close errors.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy. Release request-owned Avro filesystem clients on normal and failed reads.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics. Preserve guarded Avro filesystem ownership and exceptional cleanup.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection. Guarded Avro data readers reuse the already validated owned filesystem, retaining projection and split behavior; reader cleanup releases every stream and filesystem after failed opens and close errors.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy. Release request-owned Avro filesystem clients on normal and failed reads.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics. Preserve guarded Avro filesystem ownership and exceptional cleanup.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527, apache#69528

Problem Summary: Keep GlobFilter pattern identity separate from unescaped literal path components. Encode actual probes as exact literal paths or literal subtree prefixes, retain earlier wildcard parents across dot normalization, and honor each rule mode in both FE and runtime routing. This admits valid escaped-star/question/bracket/brace/backslash paths without weakening real local-parent checks.

### Release note

Preserve valid escaped literal filenames and listing prefixes under the remote-only Hadoop TVF policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 FE wrapper, 110 selected tests passed, zero failures/errors/skips, every required report fresh and source frozen. Actual Hadoop Globber and FE/runtime controls included. All 15 Doris Java PR files passed Checkstyle; diff check passed. Complete SQL/Groovy and community CI not claimed.
- Behavior changed: Yes. Preserve literal component identity in Hadoop discovery admission.
- Does this need documentation: No new setting or interface; existing startup and upgrade guidance applies.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69528
commit: f293dcebdba35652d127a186ef304d66936a8b23
base: 00a9e1cb6dd99355f1b385e3c3ec5c23ae5a83cf
reviewed_at: 2026-10-25T17:46+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java:171 — Rechecked failed production open cleanup, preassignment schema input and original/suppressed failures; previous runtime-03 is fixed.
  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:3695 — Literal metacharacters and later real wildcard prefixes retain their identity; previous runtime-04 is fixed while actual-local negatives remain.
  • Full aggregate review covers 65 files and converged in round 1 with zero findings. Source-bound validation records 124 selected FE tests and 26 applicable Java Checkstyle files passing; full community CI and mixed-version behavior were not run in this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of f293dce: one new P2 finding is included inline. A malformed file-URI authority can silently select a different LOCAL Lance dataset. No additional user focus was supplied.

Existing findings: independently revalidated P1 comments 4361447979 (signed-zero port), 4366428146 (native local sinks), 4366880439 (effective Hadoop read routes), and 4366880442 (manifest storage bases); their reported scenarios are addressed at this head, so existing_blocking_comment_ids is empty. The related P2 ViewFS child-lifetime concern in comment 4369693987 still has a native BrokerProperties-passthrough variant: per-prefix file_path values enter the frozen configuration profile while default-cached child clients can outlive the outer native handler. This is recorded as already covered, with no duplicate inline.

Critical-checkpoint conclusions:

  • Goal: the default-off LOCAL gate, early read privileges, canonical confinement and effective Hadoop consumer checks implement the intended access boundary for the reviewed routes; the inline URI-identity regression needs correction.
  • Scope and reuse: the changes are broad but connected to this boundary; the shared Java route interpreter is reused by FE projections and runtime checks, and existing factories/readers remain the IO owners.
  • Concurrency: native handler lookup/insertion uses the manager mutex and exact guarded identities; connection creation and JNI path checks occur outside that mutex. The bthread-to-pthread bridge waits for completion; Java request-owned mount clients use synchronized ownership and idempotent close. No new lock-order defect was substantiated.
  • Lifecycle/static initialization: request-owned Java ViewFS children and Avro streams/readers are cleaned up on partial opens and exceptional closes. Native global child caching has the already-covered residual above. No new cross-translation-unit initialization dependency was found.
  • Configuration: enable_local_tvf defaults to false and is startup-only; every serving FE must restart after opt-in. Regression pipeline configurations explicitly opt in.
  • Compatibility: TLanceFileDesc field 7 is optional is_tvf; HEAD and current master/4.2/4.1/4.0/3.1 definitions show no conflicting ID. Policy markers and the patched Hadoop/Lance ABI require matching serving components; mixed-version enforcement is not guaranteed, as disclosed by the PR.
  • Parallel paths: checked LOCAL/FILE discovery and authorization, native INSERT/EXPLAIN and FE cleanup, HDFS and S3-to-Hadoop routes, declared-schema/provided-file shortcuts, Avro schema/data readers, and Lance schema/scanner/index/row-id paths. The existing local JNI writer is gated; unrelated writers with incompatible constructor interfaces cannot reach a local TVF write.
  • Conditional semantics: selected ViewFS mounts, create-parent versus literal cleanup, scalar versus multi-target comma parsing, runtime defaults/provider overrides, exact authority identity, and URL versus literal filesystem paths were traced against consumers. The new inline addresses an uncovered SDK URL-parse-failure case.
  • Functional/negative tests: inspected gate/auth, containment/symlink, path-identity, manifest-base, real-row, configuration/provider, cache-collision, glob, exceptional-close and paired-installation tests. They do not cover the malformed file-authority collision in the inline.
  • Expected results: no changed regression .out files; the generator uses SDK APIs for the new binary manifests and the tests expect the existing fixture rows. This review did not regenerate fixtures or execute tests.
  • Observability: rejection paths avoid exposing expanded external names; existing reader/IO profiles remain in use. No separate logging or metrics defect was substantiated.
  • Transactions/persistence: no EditLog, transaction-version, delete-bitmap or internal persistence protocol changes; master-failover and internal committed-data visibility contracts are unaffected by these changes.
  • Writes/crashes: reviewed admission before filesystem creation/cleanup and recoverable individual/pair SDK installation markers. External sink atomicity remains its existing contract; no new crash-recovery defect was substantiated.
  • FE-to-BE propagation: native policies reach actual range/schema consumers, including HDFS parameters and Avro's separately created Java configuration; Lance TVF identity reaches schema and execution ranges, while unmarked positive-version catalogs keep their existing policy.
  • Performance: direct remote cached native checks avoid repeated JNI thread dispatch; LOCAL Lance uses one whole-dataset split and tree preflight is linear in visited directories/files. No additional actionable performance issue beyond the known native client retention was verified.
  • Other issues and limits: every changed path and initial risk was reviewed and dispositioned. This was static-only: no builds, tests, live storage IO or submodule initialization. Local tree preflight is not an atomic filesystem snapshot; optional io_uring/remote-service execution and mixed-version runtime behavior were not validated here.

All new verified findings are included in the inline set; previously reported concerns are handled as described above.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.81% (229/270) 🎉
Increment coverage report
Complete coverage report

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69528
commit: 95b42fa44940649bc4492d5201ca79f3778154d4
base: 00a9e1cb6dd99355f1b385e3c3ec5c23ae5a83cf
reviewed_at: 2026-10-25T21:49+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/fe-core/src/main/java/org/apache/doris/planner/TVFTableSink.java:141 — native output-path aliases are removed using first-NUL prefix, Configuration trim and case-insensitive field identity; dedicated output and real configuration/profile isolation remain. Actual ordered sender/profile/client gate observes 1,1,2 before cleanup.
  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:4101 — guarded JuiceFS denial, including remote-backed namespaces, is an explicitly declared security limitation; mutable authoritative backend/reloads cannot be proved remote by class/scheme. Ordinary unmarked/catalog behavior remains.
  • be/src/format_v2/table/lance_reader.cpp:220 — every non-bare LOCAL URI uses the pinned SDK actual local_path, including whole-string malformed-file fallback, before canonical/tree/base policy. Distinct real schema/key/row identity and narrow-root rejection cover the repair.
  • Review covers all65 files, converged in round1 with zero findings. Parent source-bound selected evidence is125 FE and20 BE tests per release; this review ran no builds/tests and does not claim complete CI/SQL/mixed-version or successful optional file+uring execution.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of head 95b42fa. One new P2 compatibility regression is reported inline: finite remote ViewFS aliases can be rejected as cycles. No additional user focus was supplied.

Critical checkpoints:

  • Goal and correctness: default-off LOCAL admission, early ADMIN/OPERATOR read checks, canonical containment, effective Hadoop runtime routing and Lance manifest-base checks are implemented. The inline finding identifies a valid remote route rejected by the new guard.
  • Scope and clarity: the changes span FE, BE, the pinned SDK and build integration because all participate in storage admission; no unrelated correctness issue was verified.
  • Concurrency: cache lookups/insertion/removal are locked; native creation and route checks run outside that mutex. Java request-child ownership is synchronized and isolated between requests; no new lock-order defect was verified.
  • Lifecycle and initialization: checked dataset RAII, failed-open cleanup, Avro reader-before-filesystem close, owned ViewFS peers, JNI global-reference release on pthreads and configuration freezing after provider resource registration. No new static initialization dependency was identified.
  • Configuration: enable_local_tvf is intentionally startup-only, false by default, with restart instructions; regression profiles explicitly opt in.
  • Compatibility: the C metadata APIs are additive. TLanceFileDesc field 7 is optional and does not collide in the inspected master/4.2/4.1/4.0/3.1 definitions. Full enforcement requires upgraded serving FE/BE and the matching patched Hadoop/Lance artifacts; old peers cannot enforce the new marker.
  • Parallel paths: reviewed LOCAL/FILE/S3/HDFS, listing and returned paths, schema and data, native and Avro, writes and cleanup, cached handles and unmarked catalog controls.
  • Special conditions: checked URI/C-string projections, provider selection, ViewFS mounts/globs, aliases and SDK fallback identity. The reported table-only recursion condition is too broad; local targets and real cycles still need rejection.
  • Test coverage: inspected positive/negative admission, parser collision, real Lance row/schema, manifest-base, cache isolation, lifecycle, projection/split and interrupted-installation test source. Builds and tests were not run under this static-only contract; remote-service and opt-in io_uring execution are unverified.
  • Test results: no expected .out files were changed; no runtime pass or full-CI claim is made.
  • Observability: existing IO profiles remain; new admission failures explain the required route/opt-in while containment errors avoid exposing expanded external paths. No new per-row logging was introduced.
  • Persistence and failover: no new EditLog, transaction, visible-version or delete-bitmap state is introduced.
  • Writes and atomicity: admission precedes the affected filesystem side effects; external sink transactionality and cleanup semantics remain those of the existing implementation. No new atomic-commit mechanism is claimed.
  • FE/BE propagation: checked native properties/HDFS schema parameters, Avro JNI markers, Lance range type/TVF marker, guarded cache identities and removal of user-supplied internal-policy aliases.
  • Performance and memory: checked configuration-profile reuse, exact connection/cache isolation, request-child cleanup and recoverable paired SDK publication. LOCAL Lance uses one whole-dataset split; no per-fragment tree-traversal regression was established. No additional verified hot-row or allocation defect was found.
  • Additional risk sweep: the HDFS cross-filesystem symlink hypothesis is unreachable with the reviewed production-global symlink gate disabled; SDK parser/base identity and cached-context paths were rechecked without another verified finding.

Existing P1 comments 4361447979 (signed ports), 4366428146 (native sinks), 4366880439 (Hadoop reads) and 4366880442 (manifest bases) were independently revalidated against this head and their described defects are addressed. They are not reposted and no existing P0/P1 remains independently confirmed as applicable.

Converged in two rounds after complete assigned static coverage and an unchanged-payload final recheck. No unresolved candidate or static coverage gap remains; runtime validation and mixed-version enforcement remain the limits stated above.

Review converged after 2 round(s).

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69528
commit: b1fb0a3fe44334d62eacb8590dbae3aa1ac49e72
base: 00a9e1cb6dd99355f1b385e3c3ec5c23ae5a83cf
reviewed_at: 2026-10-25T23:31+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:4265 — finite same-table ViewFS aliases now use unambiguous resolved table/path identity. Actual pinned Hadoop3.4.2 SINGLE lazy initialization and real unguarded/guarded dispatch independently establish the known route.
  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:4373 — successful state insertion has matching finally removal; repeated states, growing-path cycles, local targets and the independent32-state bound remain rejected as appropriate.32 mount states may terminate remotely;33 rejects.
  • fe/fe-core/src/test/java/org/apache/doris/tablefunction/HadoopTvfRuntimeAdmissionTest.java:391 — new controls cover actual ViewFS stat/literal-delete paths, native retained-Context admission, both LOCAL settings and static32/33 boundary. Only recognized remote leaf storage is mocked; real native writer IO or32-hop network execution is not claimed.
  • Full65-file review converged in round1 with zero findings. Parent source-bound129 FE tests/nine fresh required XML and26-file style pass; review ran no tests/builds. Unchanged C++/SDK/Thrift evidence is reused by bytes, without claiming full CI/SQL/mixed-version execution. Master and4.2 are excluded.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #69528 at b1fb0a3. One new P2 finding: unchanged remote Lance snapshots repeat full storage-base inspection for every fragment split, creating multiplicative setup work for many-base datasets. See the inline comment.

Critical checkpoints:

  • Goal and proof: default-off LOCAL admission, authorization before discovery, canonical containment, and actual Hadoop/Lance consumer checks are present in the inspected read/write flows. Focused positive/negative tests are included; this invocation performed source inspection only.
  • Scope: changes stay within TVF admission, storage/path/cache identity, supporting SDK/build controls, and tests. Shared routing logic is used by FE and JVM consumers.
  • Concurrency and locks: guarded HDFS cache identity/selection and cleanup use the manager mutex and shared ownership; expensive creation and JNI checks occur outside that lock. Request-owned ViewFS children synchronize creation/close. No additional lock-order defect was verified.
  • Lifecycle and initialization: SDK strings/temporary handles use RAII; failed validation does not publish a dataset. Avro closes partial opens, and ViewFS owns its child clients. No new cross-translation-unit static initialization dependency was found.
  • Configuration: enable_local_tvf defaults to false and is startup-only; the description correctly requires configuration and restart on every serving FE.
  • Compatibility: supported URI/provider identities and unmarked catalog routing were traced. Full enforcement requires upgraded serving FE/BE and matching patched runtimes; old BEs ignore the new policy marker.
  • Shared definitions: the optional TLanceFileDesc field 7 was compared with available branch-3.1/4.0/4.1/4.2/master definitions; no new collision, reused ID, or required field was found.
  • Parallel paths: LOCAL/FILE, S3/FILE, HDFS/FILE, INSERT/EXPLAIN/direct binding, FE cleanup, schema RPCs, native/Avro data reads, and fresh/cached Lance handles were checked.
  • Special conditions: native string projection, create-parent routing, literal cleanup versus glob discovery, ViewFS aliases/cycles/depth, schema-skipping CSV, and SDK local fallback rules have explicit handling and focused tests.
  • Test coverage: changed FE/BE/Avro/SDK/build-control/regression tests were inspected, including identity collisions, inherited providers, manifest bases, ownership failures, and interrupted installation. Remote-service and io_uring controls remain opt-in.
  • Results: no new expected-output files were added; expected error paths were inspected. No builds, tests, storage data IO, or submodule initialization were performed by this review. Binary fixture base references/footer were checked statically; row semantics were not executed.
  • Observability: admission failures return explicit errors and containment messages avoid expanded outside names. Existing IO/profile instrumentation is retained.
  • Persistence and transactions: Doris visible-version, EditLog, transaction, and MoW protocols are not modified.
  • Writes and crash behavior: admission precedes consumer IO; existing file-write/delete atomicity semantics are retained. SDK header/archive and paired Rust repair use persistent incomplete markers.
  • FE/BE propagation: is_tvf is set for schema and execution ranges; native read/write policy reaches actual consumers, including Avro's independent configuration.
  • Performance: the inline P2 covers repeated immutable remote-base reconstruction. The initially suspected LOCAL per-fragment tree-walk issue was dismissed because LOCAL uses one whole-dataset split.
  • Other concerns and user focus: no additional user focus was supplied. The HDFS symlink hypothesis was dismissed because the required global symlink-enablement path is absent in the reviewed production sources.

Existing P1 threads 4361447979 (signed-zero URI fallback), 4366428146 (native Hadoop sinks), 4366880439 (Hadoop reads), and 4366880442 (Lance manifest-selected bases) were independently revalidated against this head. Their reported scenarios are addressed; no existing blocking ID is carried and no duplicate comment is posted.

Known limits: cross-mount Hadoop globs retain the explicitly disclosed requirement for a proven route such as a remote fallback; local tree preflight is not an atomic filesystem snapshot; mixed-version enforcement and runtime/performance behavior were not executed here. No unresolved source-review candidate or assigned coverage gap remains.

Review converged after 2 round(s).

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Admit remote storage bases once before publishing an opened Lance dataset. A reused handle must still match the entire dataset key; LOCAL paths and registered bases remain revalidated for filesystem changes. This removes repeated metadata parsing for fragment splits without adding cache state or changing SDK interfaces.

### Release note

Avoid repeated remote storage-base inspection for splits of an unchanged Lance TVF snapshot.

### Check List (For Author)

- Test: 31 selected Clang21/ASAN filesystem and Lance tests passed with fresh XML, zero failures/errors/skips and frozen source; changed C++ ranges passed clang-format16.
- Behavior changed: Preserve storage admission rules and reuse successful remote snapshot admission.
- Does this need documentation: No additional documentation beyond existing security and upgrade guidance.
@OIiveirra

OIiveirra commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Closing this PR because the implementation scope is being reduced to a focused FE change: disable LOCAL TVF by default, retain explicit startup opt-in, and preserve the existing implementation. Replacement work will start from the latest target branch. The broader storage/runtime hardening is outside that replacement scope. The original branch, local changes and review history are preserved.

Replacement draft PR: #69593

@OIiveirra OIiveirra closed this Oct 25, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.98% (232/273) 🎉
Increment coverage report
Complete coverage report

@OIiveirra OIiveirra mentioned this pull request Oct 25, 2026
4 of 12 tasks
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 26, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69591, apache#69528

Problem Summary: Disable LOCAL and FILE-local admission without an opt-in setting, before file discovery. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same rejection before sink property validation. Retain storage implementation and adapt only directly affected fixtures and suites.

### Release note

LOCAL TVF and FILE TVF local storage are unavailable for all users, with no enable setting. LOCAL TVF INSERT is also unavailable.

### Check List (For Author)

- Test: 8 targeted FE unit tests passed without failures/skips; changed Java Checkstyle, Groovy syntax and whitespace checks passed. Regression suite bodies not executed.
- Behavior changed: Yes. Unconditional LOCAL rejection.
- Does this need documentation: Yes. Release note included; operator documentation follow-up.
yiguolei pushed a commit that referenced this pull request Oct 27, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #69591, #69528

Problem Summary:

LOCAL TVF can discover files on a backend during analysis. Reject LOCAL
unconditionally before property parsing, backend selection and file
discovery. FILE TVF with local storage delegates to the same entry
point. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same
rejection before sink property validation. No configuration option
enables these paths, including for ADMIN/OPERATOR. The underlying
storage implementation is retained.

Tests assert rejection before backend access and preserve remote
FILE/sink behavior where supported. Existing LOCAL-dependent success
suites explicitly return while the feature is unavailable. Export
completion/error checks remain active; LOCAL export-readback comparisons
are retired. Remote HDFS/S3 partition cases, the existing UUID S3
fixture branch, and ORC value checks through S3 TVF using the original
uploaded fixtures remain active where present on this branch.

### Release note

LOCAL TVF and FILE TVF with local storage are disabled for all users.
There is no configuration option to re-enable them. Explicit LOCAL TVF
INSERT is also disabled.

### Check List (For Author)

- Test
- [x] Unit Test — 8 targeted FE tests passed; zero failures, errors or
skips.
- [x] Regression test — rejection coverage and affected-suite
adaptations included; suite bodies have not been executed locally.
    - [ ] Manual test
    - [ ] No need to test

Validation: Checkstyle passed for all 5 changed Java files. Changed
Groovy suites passed syntax parsing with Groovy 4.0.19. Whitespace
checks passed. Runtime regression and formal local code review have not
been performed.

- Behavior changed:
    - [ ] No.
- [x] Yes. LOCAL admission always fails; no startup opt-in is available.
- Does this need documentation?
    - [ ] No.
- [x] Yes. Release note included; operator documentation should reflect
disabled LOCAL support.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants