Skip to content

[fix](tvf) Secure LOCAL TVF access (4.2) - #69527

Closed
OIiveirra wants to merge 23 commits into
apache:branch-4.2from
OIiveirra:fix/jira-sdbsec-112-4.2
Closed

OIiveirra wants to merge 23 commits into
apache:branch-4.2from
OIiveirra:fix/jira-sdbsec-112-4.2

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A

Related PR: #69526

Problem Summary:

LOCAL/FILE-local discovery could escape its secure root, while alternate writers and delegated Hadoop routes could bypass the LOCAL setting or access BE-local files through a remote TVF.

  • Add startup-only enable_local_tvf=false. Require ADMIN/OPERATOR before local discovery and during authorization; gate local writers and cleanup before side effects. Enabled reads require canonical containment under user_files_secure_path, including expanded paths and symlinks; rejection errors do not disclose resolved paths.

  • Validate Hadoop TVF routes in FE and at the actual consumer using one shared route interpreter. Check the branch's actual normalized Globber probes before discovery, then validate returned and supplied file paths literally in Java and native consumers. Freeze merged runtime defaults/resources, retain each native connection's context, and check actual paths on cache hits. Native writes use their create-parent route; cleanup retains literal semantics. Avro schema/data keep the same guard; data reads reuse the validated filesystem and preserve projection and split behavior.

  • Isolate guarded native caches by exact connection/authority/configuration identity. Guarded Java ViewFS requests own and close their mount clients, including partial initialization and close failures. Disable ViewFS's authority-folding inner cache only for guarded configurations. Preserve unmarked catalog behavior and known remote providers, including secure WASBS, while rejecting local overrides and unknown providers.

  • For LOCAL Lance reads, validate SDK-decoded local paths and all manifest storage bases before schema/scanner use. Resolve every non-bare LOCAL URI through the existing metadata-only SDK parser, preserving URL and whole-string fallback identity before canonical containment. Preserve bare-path secure-root anchoring and local-provider aliases. Bind dataset cache identity to the policy. Require matching patched SDK headers/archive; keep a pair-level repair marker until Lance and Paimon both finish with the same selected Rust toolchain.

  • Guarded Hadoop TVF reads and native writes with LOCAL disabled reject JuiceFS: mutable namespace metadata can select or switch to an authoritative local backend. Class and scheme names cannot establish a remote-only contract across reloads. Ordinary unmarked/catalog routing retains its existing policy. Native TVF output paths use their dedicated Thrift field. Case variants and aliases after the actual native NUL-prefix/Configuration-trim projection are removed from backend properties, so output prefixes do not partition client profiles.

Release note

LOCAL TVF and FILE-local are disabled by default. Enable them with enable_local_tvf=true in fe.conf on every serving FE and restart. Enabled local reads still require ADMIN/OPERATOR and a nonempty BE user_files_secure_path. Hadoop TVF reads require proven remote storage even when LOCAL is enabled; use constrained local() for local reads.

Guarded TVFs also reject remote-backed JuiceFS namespaces until an enforceable remote-only backend contract is available. This is an intentional security restriction; ordinary unmarked catalog routing is unchanged.

Upgrade every serving FE/BE with the matching patched Hadoop runtime for full enforcement. Mixed-version enforcement is not guaranteed. Cross-mount Hadoop globs still require a proven remote route, such as an explicit remote fallback.

Rebuild the matching patched Lance headers/archive and Paimon with one Rust toolchain. Unmarked positive-version catalogs keep their existing warehouse policy. Local tree preflight is not an atomic filesystem snapshot.

Check List (For Author)

  • Test
    • Regression test (coverage added; full local execution incomplete)
    • Unit Test
    • Manual test
    • No need to test or manual test. Explain why:

Validation:

  • Standard JDK17 FE wrapper: 125 selected tests passed with fresh reports for every required suite, zero failures/errors/skips and frozen source inputs. Coverage includes actual sender/profile/client controls for ordered mixed-case and NUL/whitespace path-key aliases, real Hadoop Globber probes with escaped literal metacharacters, literal returned/supplied paths, native route controls and request-owned ViewFS lifecycle. The Avro module ran explicitly with the BE's existing JDK17 module opens. It covers real rows, projection, nonzero splits, repeated data opens, production scanner-open failure and exceptional cleanup without a caller closing the failed scanner.

  • Standard Clang21/ASAN HdfsMgrTest: 11 selected tests passed for the native guard changes. Native Hadoop guard source and test bytes remain identical to that validated run.

  • Standard Clang21/ASAN filesystem and Lance selection: 20 tests passed, zero failures/errors/skips with fresh XML and frozen source. A pinned-SDK two-dataset collision reproduces the old wrong schema/row 7; the repaired schema, cache key and real scan select row 42. Out-of-root fallback rejects, and valid URI/relative-path controls pass. The SDK-invalid encoded-parent case is a rejection control.

  • Checkstyle inspected all 26 Doris Java source/test files in this PR; zero errors. Diff checks passed. The upstream Hadoop shadow source uses the repository's existing Checkstyle exclusion.

  • Existing native/SDK validation remains applicable to the unchanged paths: selected BE filesystem/Lance tests, SDK parser Rust/C API tests, wrapper/ABI checks and Lance/Paimon build controls. The SDK and native Hadoop inputs remain unchanged; the Lance URI resolver is covered by the fresh BE selection above. Optional file+uring scans were not executed successfully because this host denies io_uring setup.

  • Targeted clang-format16 and Clang21 analysis were performed for the earlier native changes. The scoped-lock checker crash and existing unmatched NOLINTEND limit the static analysis; complete clang-tidy is not claimed.

  • Full SQL/Groovy regression, complete community CI and mixed-version execution were not performed locally.

  • Behavior changed:

    • No.
    • Yes. Default-off local TVF access, canonical containment and effective filesystem route enforcement.
  • Does this need documentation?

    • No.
    • Yes. Startup setting and upgrade guidance need documentation; documentation PR pending.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: Add the startup-only enable_local_tvf setting, defaulting to false. Reject LOCAL TVF and FILE TVF with local storage before property parsing or backend access. Explicit opt-in restores each branch's existing behavior. Dedicated regression deployments opt in to retain LOCAL coverage.

### Release note

LOCAL TVF and FILE TVF with local storage are disabled by default. Set enable_local_tvf=true in fe.conf on every query-serving FE and restart those FEs to restore the existing behavior.

### Check List (For Author)

- Test: 7 focused FE unit tests passed with zero failures or skips; focused FE Checkstyle passed. SQL regression coverage added; runtime validation recorded separately.
- Behavior changed: Yes. Local TVF access requires explicit startup opt-in.
- Does this need documentation: Yes. Release note documents the default and opt-in.
@OIiveirra
OIiveirra requested a review from yiguolei as a code owner October 23, 2026 09:45
Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:45
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Check the canonical root and resolved paths, enforce ADMIN/OPERATOR before file access, validate Lance schema and scan paths, and preserve LOCAL/S3 range types. Reject local storage selected through S3 Lance before dataset opening.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The existing default-disabled startup gate is retained.

### Check List (For Author)

- Test: 26 focused FE tests, Checkstyle and changed-line clang-format 16 passed. BE tests could not build because the local dependency package lacks libgoogle_cloud_cpp_oauth2.a.
- Behavior changed: Yes, enforce secure local path containment and authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](fe) Disable LOCAL TVF by default (branch-4.2) [fix](tvf) Restrict local paths and require explicit startup opt-in Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: FILE_LOCAL also identifies existing local Lance catalogs. Apply the new secure-root check only to the documented LOCAL TVF version-zero ranges, retaining fixed-version catalog warehouse paths. Add catalog compatibility coverage, qualify the nested DatasetKey type in the new tests, and bind the existing glob fixture to its own root across build profiles.

### Release note

Fixed-version local Lance catalog reads keep their existing warehouse paths.

### Check List (For Author)

- Test: changed-line clang-format 16 passed. Local BE execution remains blocked by missing GCS dependencies; targeted compilation diagnostics also identify an existing Lance-C SDK API mismatch. Existing final FE changes have 26 passing tests.
- Behavior changed: Scope the new path check to LOCAL TVF.
- Does this need documentation: Covered by the PR release note.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](tvf) Restrict local paths and require explicit startup opt-in [fix](tvf) Secure LOCAL TVF access (4.2) Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary:
S3 Lance could open a local dataset through BrokerProperties because the
entry point rejected only LocalProperties. Validate the final dataset URI
before loading metadata. Allow the pinned SDK's remote providers and reject
local aliases and malformed authorities that could fall back to local paths.
Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS
HTTP locations working without changing catalog or non-Lance paths.

### Release note

S3 Lance rejects local dataset URIs and malformed remote URI authorities
before opening a dataset, regardless of storage-property selection.

### Check List (For Author)

- Test: 31 focused FE unit tests passed on this branch; changed Java
  files pass Checkstyle and git diff --check passes.
- Behavior changed: Yes. S3 Lance requires a valid supported remote URI.
- Does this need documentation: No additional documentation for this
  security validation; the parent PR's configuration documentation is pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two verified findings are attached inline: a P1 remote-URI validation bypass that reaches local Lance storage, and a P2 regression for valid percent-encoded LOCAL Lance file URIs. There were no existing inline threads or additional user focus points.

Reviewed all 20 changed files at a7a0f19, with independent BE, FE/configuration, and URI/path risk reviews. The policy goal is only partly achieved: direct LOCAL/FILE-local authorization and top-level glob/dataset containment improve, but the S3 URI check still permits local fallback.

Critical checkpoints:

  1. Goal and proof: traced constructor authorization, file discovery, schema RPC, range construction, SDK metadata open, dataset/cache key, scan and close. The two inline cases remain incorrect; added tests omit both.
  2. Scope and clarity: changes are focused on startup policy, path/URI validation, range identity and supporting tests/configuration.
  3. Concurrency and locks: no new thread, lock, lock order or shared production mutation; existing query/scanner and RPC contexts remain in use.
  4. Lifecycle/static initialization: local URI values, existing dataset RAII and session ownership remain intact; no new cross-TU initializer dependency or cleanup defect found.
  5. Configuration: enable_local_tvf is false by default and nonmutable; ConfigBase rejects runtime updates. Enabling requires configuration and restart on every query-serving FE. Seven regression deployments explicitly opt in.
  6. Compatibility: default-off and ADMIN/OPERATOR requirements are intentional behavior changes. Existing fixed-version catalogs retain their warehouse paths; the encoded file-URI regression needs correction. Mixed deployments require upgrading the nodes that enforce the policy.
  7. Protocol definitions: no Thrift/protobuf IDs, required fields, function symbols or persistent format definitions changed.
  8. Parallel paths: checked LOCAL, FILE delegation, non-Lance glob, Lance schema/latest scan/cache reuse, remote fixed snapshots and catalog/search/row-ID paths. HDFS/HTTP TVFs reject Lance in common parsing.
  9. Conditional logic: explicit per-range LOCAL/S3 identity and scan-level fallback agree with their producers; positive catalog versions intentionally bypass LOCAL-TVF containment. The URI predicate has the documented P1 exception.
  10. Test coverage: inspected BE containment/symlink/root/schema/scan tests, FE gate/privilege/revocation/URI/range tests, and LOCAL/FILE SELECT/EXPLAIN regression checks. Mocked loader acceptance does not prove Rust provider dispatch; real remote reads and full Groovy execution were not verified here.
  11. Expected results: no golden result files changed; added row IDs and file sizes agree with the committed fixture and literals.
  12. Observability/errors: new Status failures propagate through existing schema/RPC/scan boundaries; rejected expanded targets use generic errors. Existing open/version/URI profile counters remain. No new execution stage requiring new metrics was introduced.
  13. Transactions/persistence/failover: not affected; no EditLog, publish, delete-bitmap or visible-version change.
  14. Data writes/atomicity/crashes: production changes are read-only; no new transaction, write rollback or crash-recovery protocol.
  15. FE/BE propagation: schema already tags file_type; execution now preserves the TVF identity in the existing optional range field. Both LOCAL schema and execution use the resolver before opening/cache access.
  16. Performance/memory: added validation runs at construction or dataset/split setup, outside row loops; no significant new allocation, unbounded cache or read amplification identified.
  17. Other checks: decoding, NULLs, row alignment, predicates and EOF behavior are unchanged. Top-level directory containment does not provide an SDK-wide sandbox for preexisting dataset-internal symlinks; no introduced defect was established for that separate limitation.

Static source review only: no builds or tests were run and no product files were changed. No unresolved review candidate or uncovered changed file remains. Operator documentation is still pending according to the PR description; the release note describes the intended startup migration.

Review converged after 3 round(s).

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (40/40) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.28% (29908/46527)
Line Coverage 49.13% (315490/642201)
Region Coverage 44.98% (254167/565092)
Branch Coverage 46.62% (119950/257319)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (40/40) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.38% (34510/45184)
Line Coverage 61.19% (389644/636822)
Region Coverage 57.98% (328173/566003)
Branch Coverage 58.98% (151590/257027)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527

Problem Summary: Apply the startup-only LOCAL gate to INSERT and EXPLAIN INSERT. Escape the canonical secure root as a literal glob prefix. Validate raw S3 Lance ports and IPv6 before SDK access; decode file URI paths once after removing query and fragment while preserving bare paths.

### Release note

LOCAL TVF INSERT and EXPLAIN INSERT honor enable_local_tvf. Secure directories containing glob metacharacters remain usable.
LOCAL Lance preserves percent-encoded file URI identity; malformed remote Lance authorities are rejected before SDK access.

### Check List (For Author)

- Test: Unit Test: 35 FE and 11 BE selected tests passed; relevant Java Checkstyle, clang-format 16, and git diff --check passed. Regression assertions added, full Groovy not run.
- Behavior changed: Yes, enforce the existing LOCAL startup setting at the write entry and preserve valid local path identity.
- Does this need documentation: Yes, the original PR release note describes the startup opt-in requirement; documentation PR remains pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

1 similar comment
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of 90a3a93 found one new P2: LOCAL Lance file URIs with parent segments can select a different in-root dataset because filesystem symlinks are resolved before URL dot-segment normalization. The inline comment contains the concrete case and fix direction.

Existing threads were independently revalidated: P1 4361485535 is fixed by original-port digit validation and IPv6 validation; P2 4361485545 is fixed for its escaped-versus-literal path cases by one-pass file-URI decoding. Neither is reposted or carried as a current blocker. The new finding concerns URL dot-segment ordering in the newly added filesystem helper, independently of the earlier percent-decoding issue. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: startup opt-in, early read authorization and pre-open containment are implemented; the new P2 is the remaining identified correctness defect. Focused tests exist but omit its dot-segment/symlink interaction.
  2. Scope: changes are focused on LOCAL admission, secure paths, Lance URI identity and regression opt-ins.
  3. Concurrency and locks: validation uses existing planning/RPC/scanner contexts and request-local state; no new threads, locks, lock order or lock-held heavy work.
  4. Lifecycle and static initialization: dataset ownership/cleanup is unchanged; no new resource cycle or cross-translation-unit initializer dependency.
  5. Configuration: enable_local_tvf defaults false and is immutable at runtime. Enabling it requires configuration and restart on every query-serving FE, as described in the PR.
  6. Compatibility: the default-disable behavior is intentional; positive-version catalog routing is preserved. File-URI compatibility needs the inline correction. No wire/storage format or function-signature change.
  7. Shared definitions: no Thrift/protobuf definitions changed; cross-release field-ID checks are not applicable.
  8. Parallel paths: LOCAL/FILE reads, INSERT/EXPLAIN, schema/execution, shared-storage/single-BE, cloud/shared-nothing selection, S3/Broker normalization and fixed-version catalogs were traced.
  9. Special conditions: range-type precedence, version-zero LOCAL confinement, catalog exceptions, literal glob roots, provider allowlist, signed ports and IPv6 checks match their producers and pinned SDK behavior, apart from the inline URI-normalization defect.
  10. Tests: inspected all changed FE/BE tests and the Groovy suite. Positive/negative admission, privilege ordering, aliases/authorities, range identity, symlink/glob roots and escaped URI cases are represented; dot segments with symlinks are missing. Opted-in pipeline configurations normally exercise the enabled Groovy branch.
  11. Results: no expected-result files changed; new expectations agree with inspected inputs. Author-reported test runs were not independently executed.
  12. Observability: admission errors are explicit; generic containment errors avoid exposing expanded target names. Existing reader profiles and failure propagation remain in place.
  13. Persistence and transactions: no EditLog, transaction publication, visible-version, delete-bitmap or persisted metadata changes.
  14. Data writes and crashes: only sink admission changes; external-file write atomicity and crash/cleanup behavior are otherwise unchanged.
  15. FE-to-BE transport: existing optional file_type is now explicit on Lance TVF execution ranges; schema requests were already tagged, and catalog/remote snapshot versions remain fixed.
  16. Performance and memory: bounded planning/per-open path validation introduces no per-row allocation, retained buffer or cache amplification; per-match canonicalization is required for glob confinement.
  17. Further review: rechecked parser/provider dispatch, cached dataset opens, bare percent/query/fragment identity, legacy tags, authorization initialization and symlink resolution. No additional substantiated defect or unresolved code-analysis question remains.

Coverage: all 22 changed paths were reviewed, with two complete-scope reviewers and a separate remote-URI risk reviewer. Static inspection only: no Doris builds, test suites or live SDK/data scans were run; isolated filesystem path demonstrations supported the inline finding. Source files and submodules were unchanged. Runtime interoperability and full regression execution remain validation limits, not claimed results.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 22 changed paths at 90a3a93aae68c021024842b1f19e32965e730508, with independent BE, FE, and URI-risk coverage.

One new P2 finding is inline: valid uppercase/mixed-case file: schemes now fail LOCAL Lance schema discovery. The existing dot-segment/symlink finding still applies and was not reposted. The earlier signed-port/IPv6 bypass and percent-decoding finding are fixed for their reported cases. No existing P0/P1 was independently confirmed to remain applicable, so existing_blocking_comment_ids is empty. Review opinion: URI compatibility still needs correction.

Individual critical-checkpoint conclusions:

Checkpoint Conclusion
Goal and functional proof Default-off LOCAL/FILE-local reads and LOCAL writes, early read authorization, canonical containment, and remote/local Lance routing are implemented on traced paths; new and existing URI compatibility defects remain as identified above. Tests are present, but were not executed here.
Scope and clarity Production changes focus on admission, path resolution, and range identity, with matching tests/configuration opt-ins.
Concurrency and locking New state is request-local or immutable after startup. Existing heavy-pool/scanner tasks remain; no new threads, locks, lock order, or lock-held I/O was introduced.
Lifecycle and static initialization Schema dataset RAII and scan/session ownership remain intact; no new ownership cycle, cache handle, or cross-TU static dependency found.
Configuration enable_local_tvf=false is startup-only because ConfField defaults to nonmutable. Every serving FE must opt in and restart; seven regression deployment configs opt in.
Compatibility and upgrades Default-disable is intentional and documented. Existing optional range tags are reused; legacy scan-level fallback, positive-version catalog warehouses, and remote options remain. File-scheme compatibility has the new inline regression. No storage-format/symbol change.
Thrift/protobuf definitions None changed; cross-release ID/enum comparison is not applicable.
Parallel paths Checked LOCAL, FILE delegation, SELECT/EXPLAIN, INSERT/EXPLAIN, schema/scan/cache opens, non-Lance glob, remote range override, catalog routing, and the existing LOCAL-backed metadata-glob caller.
Special conditions Comments explain version-zero LOCAL routing, per-range type precedence, Broker passthrough, GooseFS authority exception, and port/IPv6 validation. The reported URI defects identify remaining incorrect conditions.
Test coverage FE tests cover gates/auth/delegation/sinks/range tags/remote URI rejection; BE tests cover glob classes, symlinks, literal roots, NUL, relative/escaped datasets, schema/scan identity, and catalog/remote routing. Case-insensitive scheme and known dot-segment identity cases are missing. The Groovy disabled branch requires a separately disabled deployment; unit tests cover disabled admission.
Changed expected results No .out files or data fixtures changed. Added row/error assertions agree with inspected control flow; execution was not verified.
Observability Errors propagate through exceptions/Status; glob RPC publishes entries only on overall success. Existing schema warnings and dataset URI/version profiles remain; no new metrics-critical operation introduced.
Transactions and persistence No EditLog, internal visible-version, delete-bitmap, transaction, or failover protocol change.
Writes and crash behavior LOCAL sink gate runs before writer creation; existing external-write/cleanup and nontransactional semantics are unchanged. No new crash-cleanup path found.
FE/BE propagation Lance TVF ranges explicitly carry existing FILE_LOCAL/FILE_S3 tags; schema ranges already did. URI/version/fragments/options and consumer precedence were inspected. No new transmission field.
Performance and memory Validation is linear in URI/path or match count at planning/discovery/open, outside row/batch loops. No new significant untracked buffer, unbounded cache, or repeated per-row work found.
Other concerns and user focus No additional user focus was supplied. Every candidate/risk was accepted, deduplicated, or dismissed with evidence; no concrete unresolved static-review coverage gap remains.

Validation limits: static inspection only, including pinned SDK/parser primary sources. No builds, tests, runtime reproduction, deployment restart, real remote-store reads, or exhaustive parser fuzzing were performed. Author-reported test results were treated as context, not independent validation.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527

Problem Summary: LOCAL Lance canonicalization followed symlinks before URL dot-segment normalization, so a valid file URI could open a different in-root dataset. Uppercase file schemes were rejected. Normalize the actual file URI scheme, separators and raw dot segments before one-pass decoding and filesystem containment; preserve bare path behavior.

### Release note

LOCAL Lance preserves file URI dataset identity for dot segments, case-insensitive schemes and raw URI separators while retaining secure-root containment.

### Check List (For Author)

- Test: 13 selected BE filesystem/Lance tests passed with fresh XML, zero failures/skips and unchanged sources. New regressions fail before the fix, including actual schema and row-value mismatch. Changed-line clang-format16 and git diff --check passed. FE source is unchanged.
- Behavior changed: Yes, restore valid file URI identity without changing bare paths, remote TVFs or fixed-version catalogs.
- Does this need documentation: The original PR documents the LOCAL startup opt-in; no additional public configuration or syntax is introduced.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of #69527 at 33cbacc reports two new P2 dataset-identity regressions in LOCAL Lance file-URI resolution. Both can silently select another existing dataset inside the allowed root. The inline comments provide independent triggering inputs and fix directions.

The four existing threads were independently revalidated: signed ports/malformed IPv6 (4361485535), percent decoding (4361485545), dot-segment/symlink order (4365942546), and case-insensitive schemes (4365978318). Their reported triggers are fixed at this head; no existing P0/P1 remains confirmed. These new findings require neither percent escapes, dot segments nor mixed-case schemes and are distinct from those threads. No additional user focus was supplied.

Critical checkpoints:

  1. Goal and evidence: default-deny LOCAL/FILE-local admission and canonical containment are implemented; SDK-compatible file-URI identity still fails in the two inline cases. Added tests exercise the intended gates and prior fixes but omit these collisions.
  2. Scope: changes are focused on admission, range identity, filesystem validation and matching tests/configuration.
  3. Concurrency: no new threads or locks. Glob/schema RPCs use the existing heavy-work pool; readers own per-scan state, and shared session initialization retains call_once. No new lock-order or heavy-under-lock issue found.
  4. Lifecycle/static initialization: validation precedes schema and cached dataset opens; scanner/dataset release order is preserved. No new persistent owner, reference cycle or cross-TU static dependency.
  5. Configuration: enable_local_tvf defaults false, is immutable, and explicitly requires every FE restart. Pipeline opt-ins are deliberate.
  6. Compatibility: existing file-type fields and values are reused, including scan-level fallback; positive-version catalog and remote ranges preserve their providers. No new storage-format or symbol incompatibility found.
  7. Thrift/proto: no definitions changed; cross-release ID comparison is inapplicable.
  8. Parallel paths: LOCAL and FILE delegation, SELECT/EXPLAIN, legacy construction, independent INSERT/EXPLAIN sink binding, schema RPC, execution ranges, remote TVFs and local catalogs were traced. Existing sink ADMIN/LOAD admission remains.
  9. Conditions: comments explain version-zero LOCAL handling, per-range precedence, canonical containment and final remote-URI validation. Both new URI identity findings apply to schema discovery and scanning.
  10. Tests: reviewed positive/negative admission, privilege revocation, URI normalization, symlink/glob containment, literal root operators, remote/catalog preservation, range tags and actual scanner assertions. Disabled-sink regression coverage is conditional on configuration; binder unit tests cover it independently. Add distinguishable datasets and bare-path controls for the two new cases.
  11. Results: no expected-result files changed. Assertions/error text were checked statically. No builds, tests or benchmarks were run; prior thread reports are not claimed as this invocation's validation.
  12. Observability: existing errors and Lance URI/version/open profiling remain. Admission checks do not need new metrics; no added logging/counter defect found.
  13. Persistence/transactions: no EditLog, transaction-state or visibility-protocol changes. Fixed snapshot and delete/bitmap semantics are untouched.
  14. Writes/crashes: the independent LOCAL sink gains admission checking; existing writer/coordinator cleanup and file-output atomicity are unchanged. No new crash-related leak/deadlock identified.
  15. FE/BE variables: execution now explicitly carries LOCAL/S3 range identity; schema requests already carry it. No additional variable or serialization path is introduced.
  16. Performance/memory: added parsing/canonicalization occurs during admission or split setup rather than per row; no new large retained allocation, redundant dataset open or hot-path regression was substantiated.
  17. Further sweep: checked parser/provider mismatches, rootless URLs, whitespace preprocessing, second parsing of canonical paths, symlink/dot/decode ordering, partial glob-result leakage and metadata-count/catalog exemptions. No unresolved candidates or uncovered changed paths remain.

Coverage includes all 22 changed files and their relevant upstream/downstream paths. Findings are supported by checked-out code and pinned SDK sources; runtime behavior remains unverified under the static-only contract.

Review converged after 2 round(s).

Comment thread be/src/io/fs/local_file_system.cpp Resolved
Comment thread be/src/io/fs/local_file_system.cpp Resolved
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527

Problem Summary: LOCAL Lance file URLs without a slash were resolved relative to the secure root, URI whitespace was treated as literal data, and bare dot segments followed symlinks before SDK lexical normalization. Match file URI preprocessing and absolute-root semantics, and normalize bare paths only at the LOCAL Lance entry before canonical containment. Preserve literal bare whitespace, encoded spaces, glob behavior and existing remote/catalog routing.

### Release note

LOCAL Lance preserves dataset identity for rootless file URLs, raw URI whitespace and bare dot segments while retaining secure-root containment. This does not add FE execution support for rootless Hadoop paths.

### Check List (For Author)

- Test: 13 selected BE filesystem/Lance tests passed with fresh XML, zero failures/skips and unchanged sources. New regressions fail before the fix, including actual schema and row-value mismatch. Changed-line clang-format16 and git diff --check passed. FE source is unchanged.
- Behavior changed: Yes, restore valid file URI identity without changing bare paths, remote TVFs or fixed-version catalogs.
- Does this need documentation: The original PR documents the LOCAL startup opt-in; no additional public configuration or syntax is introduced.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three new findings: one P1 admission bypass, one P2 LOCAL Lance home-path regression, and one P3 drive-style file-URL compatibility regression. Request changes for the P1; all verified findings are inline.

Reviewed exact head 0d96b297bd650f18441aa054b9a4cc49cedb6ed5 against the supplied merge-base diff. All 22 changed files were covered by the main review, separate FE/BE complete reviews, and an additional parser/admission risk review. The review converged in two rounds on this unchanged finding set.

All six existing threads were independently revalidated as fixed: signed-port/mapped-IPv6 local fallback, percent decoding, URL dot-segment/symlink ordering, case-insensitive file schemes, rootless file URLs, and URL whitespace/control preprocessing. No existing P0/P1 remains applicable, so no existing blocking IDs are carried forward and none of those issues is reposted. There was no additional user-provided focus.

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and functional proof Default LOCAL admission and canonical containment improve, but the JNI route bypasses the new switch and two SDK path identities regress. New tests cover the principal named routes; they omit these three cases.
Scope and clarity Changes remain focused on admission, path handling, scan identity and regression opt-in.
Concurrency and locks No new workers, shared query state, lock ordering or heavy work inside locks; existing schema/scanner and glob execution contexts remain.
Lifecycle and initialization Early auth sees initialized superclass state; later auth rechecks resource/global privileges. Schema RAII, reader dataset ownership, session initialization and C-string lifetimes remain valid. No new cross-TU dependency or reference cycle.
Configuration enable_local_tvf=false is intentionally startup-only; ConfField defaults to immutable and ConfigBase rejects runtime changes. Every serving FE must be configured/restarted.
Compatibility No wire schema, function signature or storage-format change. Existing range file-type tags are reused. Positive-version catalog warehouses retain routing; complete enforcement requires upgraded BEs.
Thrift/protobuf definitions None changed; no cross-branch field-ID/enum comparison is applicable.
Parallel paths LOCAL/FILE reads, INSERT/EXPLAIN, schema/scan, native/JNI writers, S3/Broker normalization and fixed-version catalogs traced. The JNI omission is inline.
Special conditions URI-only preprocessing/decode/dot ordering and version-zero LOCAL dispatch are justified; tilde and drive exceptions need the inline fixes.
Test coverage BE tests include real schema/rows, containment, glob classes, symlinks and URI collisions. FE tests cover admission/tagging; S3 positives mock metadata loading. JNI bypass, tilde and drive cases are missing.
Expected results Changed assertions and expected rows/errors match their fixtures; no generated result files changed.
Observability and errors New failures are explicit and expanded denied paths are sanitized. Existing dataset profiles/errors remain; no new per-row metrics/logging requirement.
Transactions and persistence No EditLog, native visibility, delete bitmap or transaction protocol changes; no failover persistence added.
Writes and crash behavior Named LOCAL writes are gated before property validation. Existing external-file write atomicity/lifetimes are unchanged; JNI can still truncate a local target through a remote name.
FE-to-BE propagation Schema and execution retain LOCAL/S3 identity through the existing file_type field, including FILE delegation. No new transmitted variable.
Performance and memory Path checks occur before dataset open/key reuse, with no new per-row work or large allocation. No further substantiated performance or memory defect.
Other risks and disposition All R1-R3 risks and candidates are accepted, deduplicated or dismissed with evidence in the ledger; no unresolved review-scope candidate remains.

Static review only: no builds or tests were run. SDK behavior was checked against pinned primary sources. Full Groovy execution, disabled-mode SQL/JNI reproduction and real remote data reads remain unexecuted; inspected test code and the PR author's reported runs are not independent runtime validation. These execution limits do not imply uncovered static scope.

Review converged after 2 round(s).

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
Comment thread be/src/io/fs/local_file_system.cpp Resolved
…ntity

    ### What problem does this PR solve?

    Issue Number: N/A

    Related PR: apache#69527

    Problem Summary: Remote-named TVF sinks could select the built-in local JNI writer while LOCAL was disabled. Apply the shared LOCAL admission check before sink analysis and final sink binding, matching the loader's class-name representation. LOCAL Lance also needs to preserve the pinned SDK's home aliases and file-URL drive components before canonical containment; add distinguishable schema and scan fixtures without changing ordinary glob or remote/catalog routing.

    ### Release note

    The LOCAL startup setting covers sinks using the built-in local JNI writer under S3/HDFS names. LOCAL Lance preserves supported home aliases and file URL drive identity inside the secure directory.

    ### Check List (For Author)

    - Test: 39 selected FE tests and 15 selected BE filesystem/Lance tests passed through standard wrappers with fresh XML, zero failures/errors/skips and unchanged sources. Java Checkstyle, changed-line clang-format16 where applicable and git diff --check passed.
    - Behavior changed: Yes, enforce the existing default-disable policy and retain valid local dataset identity.
    - Does this need documentation: The original PR documents the startup opt-in; this supplement adds no new configuration or syntax.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: one new P1 remains in effective local-storage admission. A native HDFS sink can select fs.defaultFS=file:/// while LOCAL TVF is disabled, creating BE-local files; its optional cleanup can recursively delete an FE-local directory. The inline finding contains the concrete LOAD-user input and complete call-chain evidence. This is a distinct native path from the already-reported JNI writer issue.

Reviewed exact head d1ec854a5d1a1b6751583f0f900b805b9a702fee against the supplied authoritative merge-base diff: all 23 changed paths, all FE/BE tests and regression configurations, and relevant upstream/downstream paths. No additional user focus was supplied. Static inspection only: no builds, tests, runtime reproductions, source modifications or submodule initialization were performed.

The two existing P1 comments were independently revalidated and their reported instances are fixed: 4366199143 now gates the built-in LocalFileJniWriter across remote sink names, including slash/dot/NUL class identity; 4361485535 now rejects signed ports and malformed IPv6 before remote Lance metadata access. Therefore existing_blocking_comment_ids is empty. Existing BE percent-decoding, dot/symlink, case, whitespace, rootless-URL, drive and home-alias scenarios are also addressed; none is reposted.

Critical checkpoints:

  1. Goal and evidence: default-deny LOCAL/FILE-local reads, early ADMIN/OPERATOR authorization, secure expanded-path containment, and consistent LOCAL Lance schema/scan identity are implemented. The native HDFS admission gap prevents the complete local-storage protection goal. Tests exercise the intended controls but omit this native local-provider case.
  2. Scope and clarity: production changes are focused on admission, path resolution and range identity, with corresponding unit/regression coverage and explicit CI opt-ins.
  3. Concurrency and locks: no new threads, lock acquisitions, lock-order protocol or shared mutable request state is introduced. Added filesystem checks use per-call state; passwd fallback is reentrant. Existing execution concurrency remains unchanged.
  4. Lifecycle and static initialization: no new cross-TU initializer dependency, reference cycle or retained owner is introduced. Dataset ownership, session initialization and scanner close order remain intact.
  5. Configuration: enable_local_tvf=false is intentionally startup-only; every serving FE must be restarted after enabling it. The annotation is not mutable, and dynamic configuration rejects immutable settings. Regression deployments explicitly opt in.
  6. Compatibility: default-disabled behavior is an intentional operational change requiring the documented FE rollout action. No persisted storage format or protocol definition changes; existing optional range types are reused, and fixed-version catalogs preserve their URI routing. Older FE binaries cannot enforce the new FE admission policy during a mixed-version deployment.
  7. Thrift/protobuf definitions: no gensrc files changed, so no field-ID/enum compatibility change is present.
  8. Parallel paths: LOCAL and FILE delegation, schema and cached-scan opens, legacy scan-level and modern range tags, native/JNI sinks, remote S3 normalization/broker passthrough, HDFS/HTTP Lance rejection and fixed-version catalogs were traced. The one uncovered effective-provider admission path is reported inline.
  9. Special conditions: LOCAL/version-zero validation and positive-version catalog exclusion match their callers. URI-only preprocessing preserves ordinary bare-path spelling; root escaping and component containment cover glob operators and symlinks. Effective native HDFS selection needs the additional admission check.
  10. Test coverage: inspected positive/negative FE admission, privilege revocation, JNI aliases, remote URI controls, glob/symlink/root containment, and real Lance schema/key/row identity fixtures. End-to-end Groovy execution, real remote-service reads and the new native-HDFS reproduction were not run under this static-only contract.
  11. Expected results: no generated .out files changed. Inspected assertions and Groovy backend-error expectations match their code paths; author-reported passing runs were not treated as validation performed here.
  12. Observability: new denials provide actionable enablement/privilege errors and intentionally hide expanded external target names. Existing RPC status and reader profiles remain available; no new hot-path metric requirement was identified.
  13. Transactions and persistence: no EditLog, transaction commit, visible-version or delete-bitmap protocol changes are introduced. Positive catalog snapshot routing remains unchanged.
  14. Writes and crash behavior: accepted requests retain the existing sink lifecycle and its nontransactional file semantics. New admission normally precedes cleanup/execution, but the reported native HDFS bypass permits local side effects before those protections apply; no separate crash/cleanup regression was verified.
  15. FE-BE propagation: no new wire variable is added. Explicit Lance range file_type now matches the schema request; per-range identity takes precedence over legacy scan parameters, and URI/version/options remain in dataset cache keys.
  16. Performance and memory: parsing/allocation is bounded by input size and path resolution occurs during schema/dataset setup rather than per output row. No new substantial retained allocation, redundant batch operation or material performance regression was verified.
  17. Additional issues and limits: the final missed-issue sweep found no second actionable defect. Pathless file-URL query/fragment handling was explicitly assessed as a low-impact parser edge, rather than a root escape or ordinary valid-dataset regression. No assigned changed-file or risk-mechanism coverage remains open; runtime validation remains excluded.

Converged after 2 review rounds: all assigned reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged final finding set, required static coverage is complete, and all candidates and risk focuses have explicit dispositions. Convergence describes review completeness; the new P1 still warrants changes. No documentation file is added in this PR; its description says the documentation PR is pending.

Review converged after 2 round(s).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review converged after 2 rounds of static review of all 64 changed paths at d8dfb58. One new P2 regression is reported inline: explicitly configured JuiceFS is rejected by FE admission although the effective runtime recognizes it as remote.

Existing review context: the five earlier P1 findings (malformed Lance authorities, local JNI writers, native Hadoop local writes/cleanup, remote Lance manifest bases, and S3-to-Hadoop reads) were independently rechecked against their current consumers; their reported triggers are addressed, so no existing P0/P1 is retained. Existing P2 thread r4367753241 still covers a paired-toolchain retry gap: if Lance finishes installation under a newer selected Rust toolchain and Paimon fails before its installer begins, the next invocation can reuse new Lance with old Paimon. This is not reposted.

Critical checkpoints:

  1. Goal/behavior: default-off LOCAL admission, privilege-before-discovery, canonical containment and effective Hadoop/Lance storage validation are traced from SQL through schema/listing/sinks to native/JNI/SDK consumers. The new remote-provider regression prevents full compatibility.
  2. Scope: the FE/BE/runtime/SDK/build changes serve these boundaries; no unrelated feature change identified.
  3. Concurrency/locks: HdfsMgr lookup/insertion/cleanup remain locked; slow construction/JNI checks occur outside that lock; exact identities and atomic access timestamps are checked. No new lock-order problem established.
  4. Lifecycle/static state: temporary datasets/JSON and Java context ownership, failure cleanup, pthread completion and reused-session validation were checked; no new static-initialization defect found.
  5. Configuration: enable_local_tvf is deliberately startup-only, false by default, requiring every FE restart. Enabled Hadoop reads still require remote routes.
  6. Symbols/storage compatibility: additive Lance metadata C ABI requires matching headers/archive; interrupted individual installation is detected. The paired-package retry gap above remains. Dataset formats are unchanged.
  7. Thrift/proto: optional TLanceFileDesc field 7 has no collision against current master, 4.2, 4.1, 4.0 and 3.1 definitions. Complete enforcement requires upgraded serving FE/BE and patched Hadoop runtime.
  8. Parallel paths: LOCAL/FILE/S3/HDFS, read/write/cleanup, schema/data, native/libhdfs3/Avro, and Lance scanner/index/row-id/session reuse were covered.
  9. Conditions: marker ownership, LOCAL/version-zero compatibility, direct-remote fast path, selected ViewFS mounts and decoded versus bare paths were checked. No additional verified bypass found.
  10. Tests: added positive/negative admission, URI identity, symlink/base, cache/collision, fixture and build-gate tests were inspected. Explicit JuiceFS TVF construction and paired-build retry coverage are missing.
  11. Expected results: no golden-output changes; fixture generation uses official SDK APIs and checks actual row identities. Historical test claims are not current execution evidence.
  12. Observability: route failures and existing scan/open counters remain; local tree preflight has no separate timing counter. No additional observability defect established.
  13. Transactions/persistence: no Doris EditLog, visible-version or publish-protocol changes.
  14. Writes/atomicity: checks precede native writer and FE deletion side effects; package publication is staged, subject to the known paired retry gap.
  15. FE/BE propagation: runtime policy and Lance TVF marker were followed through schema/data senders and Avro; old peers cannot provide the new enforcement.
  16. Performance/memory: cached direct-remote checks avoid a new pthread; LOCAL tree walks are iterative and deduplicate canonical directories. They run at open/range boundaries, not each batch; no further actionable regression established.
  17. Final recheck: all changed paths, candidate dispositions, prior threads and R1-R4 risk mechanisms were swept; only the new inline P2 and existing paired-build P2 remain verified concerns.

There was no additional user focus. This was static-only: no builds, tests, filesystem read/write scenarios, Rust link, SQL/Groovy, external-service scans or mixed-version runs were executed. Local tree preflight is not an atomic filesystem snapshot. These validation limits remain; no unresolved static-review candidate or uncovered changed path remains.

Review converged after 2 round(s).

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527

Problem Summary: Align glob discovery routes and exact nested ViewFS identity, and preserve supported remote providers. Preserve local Lance alias semantics with SDK-owned URI parsing and make paired Rust repair retryable. Keep the existing LOCAL setting, privileges and containment requirements.

### Release note

Guarded Hadoop requests use exact nested ViewFS identities and reject escaped local listing routes. Supported provider compatibility is preserved.

### Check List (For Author)

- Test: Unit Test (107 FE, 28 BE, SDK and build controls), isolated ViewFS endpoint comparison
- Behavior changed: Yes (guarded route enforcement and supported provider compatibility)
- Does this need documentation: Yes (startup and upgrade guidance remains pending)
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of 43ccc01: one new P1 finding in native Hadoop write admission (inline). The runtime guard validates the final component although ViewFS creation resolves the parent; BE-only mount configuration can therefore select a local write route after admission. The earlier FE parent check is present; this finding concerns the new BE consumer boundary.

Existing threads were read and independently revalidated. The original P1 instances (4361485535, 4366199143, 4366424083, 4366876841, 4367464261) are addressed on this head; none is reposted or retained as an existing blocker. The existing parser, declared-schema, provider-set and paired-toolchain findings are also addressed.

Critical checkpoint conclusions:

  1. Goal and proof: default-off LOCAL and effective-storage admission are broadly implemented, but the inline write-route defect leaves enforcement incomplete. Positive/negative tests are present; none was executed during this static-only review.
  2. Scope: the changes consistently implement TVF admission, path identity, guarded caches and required SDK integration; no unrelated change identified.
  3. Concurrency: HdfsMgr protects lookup/insert/cleanup with its mutex and performs construction/JNI checks outside it; no new lock-order defect found.
  4. Lifecycle: dataset/string ownership uses RAII, checked datasets enter reader state only after admission, and Java contexts retain connection configuration and release on pthreads; no new static-initialization dependency found.
  5. Configuration: enable_local_tvf is intentionally startup-only, defaults false and documents restarting every FE.
  6. Compatibility: the additive Lance C ABI requires matching headers/archive; paired repair rebuilds Lance/Paimon using one selected cargo environment and retains interruption markers. Mixed-version full enforcement remains an explicit deployment limitation.
  7. Wire definitions: the new is_tvf field is optional id 7. Live master and 4.2/4.1/4.0/3.1 definitions have no conflicting id/type; schema and data senders both carry the marker.
  8. Parallel paths: LOCAL/FILE, S3/HDFS, schema/data, Avro/native, sink binding/cleanup, Lance primary/base and cache-reuse paths were covered.
  9. Conditional routing: shared raw/effective provider classification and separate Java/native path projections are appropriate; native writes still require operation-specific parent resolution as reported inline.
  10. Coverage: inspected FE/BE/Java, parser/glob, fixture schema/rows, dependency publication/retry and regression tests. Runtime tests for the reported parent-route mismatch are missing.
  11. Expected results: no .out file changes; manifest fixtures have an official-SDK generator and row assertions. Binary/runtime behavior was not executed here.
  12. Observability: policy failures are explicit and avoid resolved local-name disclosure; existing IO/reader profiling is retained.
  13. Persistence: no Doris EditLog, visible-version, rowset or MoW/delete-bitmap changes; no new transaction-persistence obligation identified.
  14. Writes and failure: FE checks precede cleanup, and dependency publication is recoverable; the inline BE route mismatch must be fixed before relying on disabled-mode local-write denial.
  15. FE/BE propagation: native policies and Lance TVF markers reach schema/data/write consumers, including FILE delegation and Avro configuration copies.
  16. Performance and memory: cached direct-remote checks avoid per-file pthread dispatch; byte caches include configuration identity, and LOCAL preflight scales with reachable entries. No additional substantiated performance or ownership regression found.
  17. Additional concerns and limits: no extra user-provided focus. No builds, tests, source modifications or submodule initialization were performed. Full SQL/Groovy, optional io_uring and mixed-version deployment behavior remain unexecuted. Local tree preflight is not an atomic filesystem snapshot.

Review converged after 3 rounds; all verified findings are inline and no unresolved review candidate or assigned coverage gap remains. Convergence describes completion of the static review, not acceptance of the P1 defect.

Review converged after 3 round(s).

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Hadoop read discovery must check slash-containing brace expansion before listing. Literal cleanup paths must retain exact-path semantics. Native writes must validate the actual create-parent route against the frozen runtime configuration, including cache reuse, while reads retain leaf-path checks. Guarded Java ViewFS requests must own and close their child clients instead of retaining path-specific clients in Hadoop's global cache. Apply these corrections without changing native filesystem ownership or unmarked catalog behavior.

### Release note

Close the remaining Hadoop TVF local-route bypass and preserve valid remote cleanup and filesystem-client lifecycle under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 selected FE tests, standard Clang21/ASAN HdfsMgrTest, actual-file Checkstyle and targeted C++ formatting/static checks. Full SQL/Groovy and full community CI are not claimed.
- Behavior changed: Yes. Match admission to the actual consumer's read/write/delete semantics and release request-owned Java ViewFS clients.
- Does this need documentation: No additional setting or interface introduced by this follow-up; the PR's existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection. Guarded Avro data readers reuse the already validated owned filesystem, retaining projection and split behavior; reader cleanup releases every stream and filesystem after failed opens and close errors.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy. Release request-owned Avro filesystem clients on normal and failed reads.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics. Preserve guarded Avro filesystem ownership and exceptional cleanup.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526, apache#69527, apache#69528

Problem Summary: Validate only the normalized filesystem probes used during Hadoop discovery. Validate returned and supplied file paths literally through their actual Java and native consumers, preserving native configuration and schema-route checks. This keeps proven remote glob routes usable while retaining local-probe rejection. Guarded Avro data readers reuse the already validated owned filesystem, retaining projection and split behavior; reader cleanup releases every stream and filesystem after failed opens and close errors.

### Release note

Preserve valid remote glob and literal-file reads under the default-off LOCAL policy. Release request-owned Avro filesystem clients on normal and failed reads.

### Check List (For Author)

- Test: Unit Test through the standard JDK17 FE wrapper with fresh required XML and frozen sources; exact aggregate PR Java Checkstyle and git diff checks. Full SQL/Groovy and complete community CI are not claimed.
- Behavior changed: Yes. Match discovery and literal-file admission to actual consumer semantics. Preserve guarded Avro filesystem ownership and exceptional cleanup.
- Does this need documentation: No additional setting or interface introduced by this follow-up; existing startup and upgrade guidance applies.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69527, apache#69528

Problem Summary: Keep GlobFilter pattern identity separate from unescaped literal path components. Encode actual probes as exact literal paths or literal subtree prefixes, retain earlier wildcard parents across dot normalization, and honor each rule mode in both FE and runtime routing. This admits valid escaped-star/question/bracket/brace/backslash paths without weakening real local-parent checks.

### Release note

Preserve valid escaped literal filenames and listing prefixes under the remote-only Hadoop TVF policy.

### Check List (For Author)

- Test: Unit Test; standard JDK17 FE wrapper, 110 selected tests passed, zero failures/errors/skips, every required report fresh and source frozen. Actual Hadoop Globber and FE/runtime controls included. All 15 Doris Java PR files passed Checkstyle; diff check passed. Complete SQL/Groovy and community CI not claimed.
- Behavior changed: Yes. Preserve literal component identity in Hadoop discovery admission.
- Does this need documentation: No new setting or interface; existing startup and upgrade guidance applies.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69527
commit: 2767d7310a277dcb94fdc5fc7f9db3cf73ecddac
base: eaf1c85512d5bb96f703e3961f9e2cfe9887df79
reviewed_at: 2026-10-25T17:45+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java:171 — Rechecked failed production open cleanup, preassignment schema input and original/suppressed failures; previous runtime-03 is fixed.
  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:3695 — Literal metacharacters and later real wildcard prefixes retain their identity; previous runtime-04 is fixed while actual-local negatives remain.
  • Full aggregate review covers 65 files and converged in round 1 with zero findings. Source-bound validation records 124 selected FE tests and 26 applicable Java Checkstyle files passing; full community CI and mixed-version behavior were not run in this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.03% (1992/2553)
Line Coverage 65.47% (35810/54696)
Region Coverage 53.27% (34425/64621)
Branch Coverage 56.07% (10719/19117)

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of #69527 at 2767d73: request changes for one new P1. JuiceFS is classified as remote by implementation class even when its namespace selects a local authoritative data backend. The inline finding gives the supported-version trigger and source evidence.

Critical checkpoints:

  • Goal and evidence: default-off LOCAL/FILE-local access, pre-discovery privileges, canonical containment and effective Hadoop/Lance storage admission are implemented across the intended paths. The JuiceFS backing-store case leaves that goal incomplete. Added positive/negative tests were inspected statically; they do not cover this case.
  • Scope and clarity: changes remain within admission, path identity, request/client isolation and the required Lance SDK/build support. Shared Hadoop interpretation reduces separate FE/runtime classifiers; the remaining issue is an unsupported assumption within that shared classifier.
  • Concurrency and locks: HdfsMgr uses its manager mutex for cache selection/insertion; expensive connection creation and JNI validation occur outside it, and the selected handle is revalidated after racing creation. No additional deadlock or unprotected shared-state defect was established.
  • Lifecycle and static initialization: native contexts retain global JNI references with OS-thread release, Java ViewFS requests own mount clients, and Avro closes readers before clients on success and partial failure. No new cross-translation-unit initialization dependency was found.
  • Configuration: enable_local_tvf defaults to false and intentionally requires fe.conf plus restart of every FE; regression configurations explicitly opt in. No new dynamically mutable setting is claimed.
  • Compatibility and definitions: TLanceFileDesc field 7 is optional bool is_tvf. HEAD and origin/master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1 definitions were compared; no field collision/reuse or new required field was found. SDK headers/archive require both new exports; paired Rust repair retains failure markers until Lance/Paimon outputs validate.
  • Parallel paths and conditions: LOCAL/FILE/S3/HDFS reads, schema/data consumers, native/JNI sinks, FE cleanup, ViewFS nested/glob routes, Avro, Lance aliases/manifest bases and reused handles were traced. Conditions document legacy version-zero LOCAL and unmarked fixed-version catalog behavior. The new inline issue concerns JuiceFS delegation and is distinct from the prior remote-JuiceFS compatibility request.
  • Test coverage and results: FE, BE, Avro, filesystem, protocol, fixture-generator, build-control and Groovy changes were reviewed. Negative/positive, alias, cache, split/projection and failure-cleanup cases exist. No expected .out result files changed. No build, test or runtime validation was performed during this review.
  • Observability: existing open/cache/profile/error paths were inspected; new admission failures report their reason and local containment failures avoid expanded private paths. No separate observability defect was established.
  • Transactions, persistence and writes: no committed-visibility, delete-bitmap, EditLog or transaction protocol changes. This PR adds admission to existing external writes/deletes; it adds no new atomicity or failover guarantee. Side-effect admission is covered by the new P1.
  • FE/BE propagation: the TVF marker, protected native read/write properties, schema request properties/THdfsParams, Avro marker and range file types have corresponding receive/fallback paths.
  • Performance and memory: exact connection/profile identity protects guarded caches; direct remote cached paths avoid per-file JNI; LOCAL Lance has one whole-dataset split and no per-batch tree walk. Required preflight remains linear in reachable entries. No additional substantiated CPU/memory regression was found.
  • Other risks and user focus: the final missed-issue recheck covered nested create parents, decoded base paths, manifest references and interrupted paired SDK publication. There was no additional user-provided focus.

Existing P1 findings 4361485535, 4366199143, 4366424083, 4366876841, 4367464261 and 4369701678 were independently revalidated and are addressed by the current guards; none is carried as still blocking or duplicated. Other existing threads remain fenced.

Review convergence: converged after two rounds with complete assigned static coverage and all candidates/risk focuses disposed. Remaining validation/deployment limits: compilation, actual test execution, Rust final linking, full SQL/Groovy/CI and optional io_uring/remote runtime behavior were not verified here. Full enforcement requires every serving FE/BE and matching patched runtimes to be upgraded; local tree preflight is not an atomic filesystem snapshot. These limits are explicit and do not represent unresolved review candidates.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.07% (227/270) 🎉
Increment coverage report
Complete coverage report

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69527
commit: a59bdfde74f4712c668575b6c6439966a4684e2b
base: eaf1c85512d5bb96f703e3961f9e2cfe9887df79
reviewed_at: 2026-10-25T21:49+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/fe-core/src/main/java/org/apache/doris/planner/TVFTableSink.java:141 — native output-path aliases are removed using first-NUL prefix, Configuration trim and case-insensitive field identity; dedicated output and real configuration/profile isolation remain. Actual ordered sender/profile/client gate observes 1,1,2 before cleanup.
  • fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java:4101 — guarded JuiceFS denial, including remote-backed namespaces, is an explicitly declared security limitation; mutable authoritative backend/reloads cannot be proved remote by class/scheme. Ordinary unmarked/catalog behavior remains.
  • be/src/format_v2/table/lance_reader.cpp:220 — every non-bare LOCAL URI uses the pinned SDK actual local_path, including whole-string malformed-file fallback, before canonical/tree/base policy. Distinct real schema/key/row identity and narrow-root rejection cover the repair.
  • Review covers all65 files, converged in round1 with zero findings. Parent source-bound selected evidence is125 FE and20 BE tests per release; this review ran no builds/tests and does not claim complete CI/SQL/mixed-version or successful optional file+uring execution.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new actionable findings on head a59bdfd. The existing P1 findings were independently revalidated against this head; none remains applicable, so no blocking comment IDs are carried forward and no duplicate inline comments are posted. Review converged after 2 rounds.

Critical checkpoints:

  1. Goal and evidence: startup-disabled LOCAL/FILE-local access, authorization before discovery, canonical containment, effective Hadoop route admission, and Lance primary/base policy are implemented across the traced consumers. Added unit and regression cases exercise those mechanisms; this invocation inspected them statically.
  2. Scope and reuse: the changes span the required FE, BE, Java/SDK and build boundaries. One shared Hadoop route interpreter and one known-provider classification serve raw and runtime checks; no unrelated actionable change was found.
  3. Concurrency and locks: native cache lookup/insertion and collision cleanup remain under the cache mutex; provider creation, JNI route checks and destruction run outside it. Guarded Java ViewFS clients are request-owned, with synchronized child ownership and close. No new lock-order cycle was established.
  4. Lifecycle and initialization: native connection creation forces a new Hadoop instance, clears/takes its thread-local context, and captures the outer frozen context after nested initialization. Avro closes readers before their filesystem, including partial-open failures. No new cross-translation-unit static-initialization dependency was found.
  5. Configuration: enable_local_tvf is intentionally startup-only, defaults false, and requires a restart of every serving FE. Test pipelines opt in explicitly. Hadoop reads still require proven remote routes when LOCAL is enabled; JuiceFS rejection is an intentional, documented restriction because its backing store is mutable.
  6. Compatibility: unmarked catalogs retain their configured policy. The additive Lance C APIs require matching headers/archive and paired Lance/Paimon Rust repair. Interrupted publication is fenced by individual and pair markers. Full enforcement requires upgraded serving FE/BE and the patched Hadoop runtime; mixed-version enforcement is not promised.
  7. Shared definitions: TLanceFileDesc field 7 is optional is_tvf. Supported release-branch definitions were compared; no reused field ID or conflicting meaning was found, and no proto definition changed.
  8. Parallel consumers: reviewed SELECT/EXPLAIN, FILE delegation, LOCAL/S3/HDFS, Java listing and cleanup, supplied/returned file paths, schema RPC versus declared CSV schema, native/JNI sinks, Avro, Lance schema/scans and cache reuse.
  9. Conditional behavior: native create-parent versus read-leaf routing, Java literal cleanup versus glob probes, C-string projection versus Java strings, and TVF versus catalog identity are explained in code and covered by targeted assertions. Unknown or runtime-dependent routes fail explicitly.
  10. Test coverage: changed FE/Avro/Hadoop and BE filesystem/Lance tests, SDK-generated fixture code, SDK C/Rust tests, build-control tests and account regression changes were inspected. Positive, negative, alias, nested-mount, cache, projection/split and cleanup-failure cases are present.
  11. Expected results: no generated SQL .out files changed. Inspected schema, row-identity, path, error and lifecycle assertions match the traced contracts; optional remote/io_uring tests depend on their stated environment.
  12. Observability: admission failures are explicit, rejected LOCAL resolution avoids exposing expanded target names, and existing filesystem/Lance profiles and error contexts remain available. No new per-row logging was introduced.
  13. Persistence and visibility: no EditLog, transaction metadata, partition visible-version or MoW delete-bitmap logic changed. Existing committed-data visibility contracts are outside the modified mechanisms.
  14. Writes and failures: effective-writer and cleanup checks precede their side effects. Existing external TVF writes remain nontransactional; this change does not add an atomicity or failover guarantee. Resource and interrupted SDK-publication failure paths were inspected.
  15. FE/BE propagation: Hadoop policy reaches schema and data properties/HDFS params, final sink properties, and Avro Java parameters. Lance TVF identity reaches both schema and scan descriptors and participates in reader policy identity.
  16. Performance and memory: no new row-loop work, redundant RPC under metadata locks, or substantiated unbounded hot-path regression was found. LOCAL Lance tree preflight is an intentional traversal with canonical-directory cycle deduplication; LOCAL uses one whole-dataset split. Temporary reader/context ownership and cleanup were reviewed without executing memory instrumentation.
  17. Final recheck and focus: all 65 authoritative changed paths were covered, existing threads were used as duplicate fences, and all investigated candidates and initial risks have evidence-backed dispositions. There was no additional user-provided focus.

Limits: static review only; no builds, tests, source modifications or submodule initialization were performed. Full SQL/cluster execution, live storage-provider behavior, io_uring and mixed-version deployment were not validated by this invocation. LOCAL tree preflight is not an atomic filesystem snapshot, as the PR documents. No unresolved static candidate or assigned coverage gap remains.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.25% (230/273) 🎉
Increment coverage report
Complete coverage report

@OIiveirra

OIiveirra commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Closing this PR because the implementation scope is being reduced to a focused FE change: disable LOCAL TVF by default, retain explicit startup opt-in, and preserve the existing implementation. Replacement work will start from the latest target branch. The broader storage/runtime hardening is outside that replacement scope. The original branch, local changes and review history are preserved.

Replacement draft PR: #69592

@OIiveirra OIiveirra closed this Oct 25, 2026
@OIiveirra OIiveirra mentioned this pull request Oct 25, 2026
4 of 12 tasks
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 65.71% (414/630) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.34% (29962/46569)
Line Coverage 49.20% (316313/642942)
Region Coverage 45.02% (254733/565861)
Branch Coverage 46.68% (120380/257864)

OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 26, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69591, apache#69527

Problem Summary: Disable LOCAL and FILE-local admission without an opt-in setting, before file discovery. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same rejection before sink property validation. Retain storage implementation and adapt only directly affected fixtures and suites.

### Release note

LOCAL TVF and FILE TVF local storage are unavailable for all users, with no enable setting. LOCAL TVF INSERT is also unavailable.

### Check List (For Author)

- Test: 8 targeted FE unit tests passed without failures/skips; changed Java Checkstyle, Groovy syntax and whitespace checks passed. Regression suite bodies not executed.
- Behavior changed: Yes. Unconditional LOCAL rejection.
- Does this need documentation: Yes. Release note included; operator documentation follow-up.
yiguolei pushed a commit that referenced this pull request Oct 27, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #69591, #69527

Problem Summary:

LOCAL TVF can discover files on a backend during analysis. Reject LOCAL
unconditionally before property parsing, backend selection and file
discovery. FILE TVF with local storage delegates to the same entry
point. Explicit LOCAL INSERT and analyzed EXPLAIN INSERT use the same
rejection before sink property validation. No configuration option
enables these paths, including for ADMIN/OPERATOR. The underlying
storage implementation is retained.

Tests assert rejection before backend access and preserve remote
FILE/sink behavior where supported. Existing LOCAL-dependent success
suites explicitly return while the feature is unavailable. Export
completion/error checks remain active; LOCAL export-readback comparisons
are retired. Remote HDFS/S3 partition cases, the existing UUID S3
fixture branch, and ORC value checks through S3 TVF using the original
uploaded fixtures remain active where present on this branch.

### Release note

LOCAL TVF and FILE TVF with local storage are disabled for all users.
There is no configuration option to re-enable them. Explicit LOCAL TVF
INSERT is also disabled.

### Check List (For Author)

- Test
- [x] Unit Test — 8 targeted FE tests passed; zero failures, errors or
skips.
- [x] Regression test — rejection coverage and affected-suite
adaptations included; suite bodies have not been executed locally.
    - [ ] Manual test
    - [ ] No need to test

Validation: Checkstyle passed for all 5 changed Java files. Changed
Groovy suites passed syntax parsing with Groovy 4.0.19. Whitespace
checks passed. Runtime regression and formal local code review have not
been performed.

- Behavior changed:
    - [ ] No.
- [x] Yes. LOCAL admission always fails; no startup opt-in is available.
- Does this need documentation?
    - [ ] No.
- [x] Yes. Release note included; operator documentation should reflect
disabled LOCAL support.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants