[feat] add Parquet metadata TVF by xylaaaaa · Pull Request #58972 · apache/doris

xylaaaaa · 2025-12-11T10:55:17Z

What problem does this PR solve?

Expose Parquet file metadata via a table-valued function for inspection and debugging.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Add a Parquet metadata TVF so users can query Parquet file metadata via SQL.
Backend adds a Parquet metadata reader and scan path; frontend wires the TVF definition.
Enables easy inspection of partitions/row groups/column stats to aid troubleshooting.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Thearas · 2025-12-11T10:55:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-12-12T00:42:48Z

+    META_STATISTICS_MAX
+};
+
+std::string join_path(const std::vector<std::string>& items) {


Use join() in be/src/util/string_util.h

morningman · 2025-12-12T00:43:49Z

+}
+
+std::string encodings_to_string(const std::vector<tparquet::Encoding::type>& encodings) {
+    std::ostringstream oss;


use fmt lib instead of std::ostringstream

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

morningman · 2025-12-15T10:01:35Z

+}
+
+template <typename ColumnType, typename T>
+void insert_numeric_impl(MutableColumnPtr& column, T value) {


There are some util method in be/src/exec/schema_scanner/schema_scanner_helper.cpp. You can reuse them

morningman · 2025-12-15T10:04:29Z

+    }
+}
+
+std::string physical_type_to_string(tparquet::Type::type type) {


extract these xx_to_xx methods to a new file parquet_utils.h. So that they can be used in other places, maybe.

morningman · 2025-12-15T10:06:54Z

+
+} // namespace


All methods in this namespaces should be moved to somewhere else, like parquet_utils.h

morningman · 2025-12-15T10:08:18Z

+    explicit ParquetSchemaModeHandler(RuntimeState* state) : ModeHandler(state) {}
+
+    void init_slot_pos_map(const std::vector<SlotDescriptor*>& slots) override {
+        std::unordered_map<std::string, int> name_to_pos = _build_name_to_pos_map(slots);


Suggested change

std::unordered_map<std::string, int> name_to_pos = _build_name_to_pos_map(slots);

const auto& name_to_pos = _build_name_to_pos_map(slots);

morningman · 2025-12-15T10:16:45Z

+    if (params.__isset.file_type) {
+        _file_type = params.file_type;
+    } else if (!_paths.empty()) {
+        std::string lower_path = to_lower(_paths.front());


the file_type should be handled on FE side. No need to handle it on BE side

morningman · 2025-12-15T10:23:06Z

            case TASKS:
                return TasksTableValuedFunction.getColumnIndexFromColumnName(columnName, params);
+            case PARQUET:
+                return null;


Why return null?

morningman · 2025-12-15T10:24:16Z

+        }
+        StorageProperties storageProperties = null;
+        if (Strings.isNullOrEmpty(scheme) || "file".equalsIgnoreCase(scheme)) {
+            this.fileType = TFileType.FILE_LOCAL;


We have LocalProperties, so I don't think we need this if..else

morningman · 2025-12-15T10:25:23Z

+        if (Strings.isNullOrEmpty(rawPath)) {
+            throw new AnalysisException("Property 'path' is required for parquet_metadata");
+        }
+        List<String> parsedPaths = Arrays.stream(rawPath.split(","))


I think we don't need file list with comma, only:

full file name

file name with wildcard

morningman · 2025-12-15T10:27:14Z

+        }
+
+        // Expand any glob patterns (e.g. *.parquet) to concrete file paths.
+        normalizedPaths = expandGlobPaths(normalizedPaths, storageProperties, this.fileType);


You can simply refer to BrokerUtil.parseFile(path, brokerDesc, fileStatuses);.
See parseFile() method in ExternalFileTableValuedFunction

morningman · 2025-12-15T10:28:12Z

@@ -0,0 +1,7 @@
+#!/bin/bash


Remove this file

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-15T10:48:45Z

+
+unset http_proxy
+unset https_proxy
+cd /mnt/disk1/chenjunwei/doris_build/doris/output/be


This file appears to be a personal development script with a hardcoded path (/mnt/disk1/chenjunwei/doris_build/doris/output/be) that should not be committed to the repository. This poses several issues: it exposes the developer's username and local directory structure, won't work on other machines, and is unrelated to the Parquet metadata feature being added.

Suggested change

cd /mnt/disk1/chenjunwei/doris_build/doris/output/be

# Set BE_DIR from first argument, environment variable, or default to current directory

BE_DIR="${1:-${BE_DIR:-$(pwd)}}"

cd "$BE_DIR"

Copilot · 2025-12-15T10:48:45Z

+        String rawMode = normalizedParams.getOrDefault(MODE, MODE_METADATA);
+        mode = rawMode.toLowerCase();
+        if (!SUPPORTED_MODES.contains(mode)) {
+            throw new AnalysisException("Unsupported mode '" + rawMode + "' for parquet_metadata");


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. This should be updated for consistency with the function name users actually invoke.

Copilot · 2025-12-15T10:48:46Z

+            expanded.addAll(expandSingleGlob(path, storageProperties, fileType));
+        }
+        if (expanded.isEmpty()) {
+            throw new AnalysisException("No files matched parquet_metadata path patterns: " + inputPaths);


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. This should be updated for consistency with the function name users actually invoke.

Suggested change

throw new AnalysisException("No files matched parquet_metadata path patterns: " + inputPaths);

throw new AnalysisException("No files matched parquet_meta path patterns: " + inputPaths);

Copilot · 2025-12-15T10:48:46Z

+            case AZURE:
+                return TFileType.FILE_S3;
+            default:
+                throw new AnalysisException("Unsupported storage type for parquet_metadata: " + type);


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. This should be updated for consistency with the function name users actually invoke.

Suggested change

throw new AnalysisException("Unsupported storage type for parquet_metadata: " + type);

throw new AnalysisException("Unsupported storage type for parquet_meta: " + type);

Copilot · 2025-12-15T10:48:46Z

+Status ParquetMetadataReader::_init_from_scan_range(const TMetaScanRange& scan_range) {
+    if (!scan_range.__isset.parquet_params) {
+        return Status::InvalidArgument(
+                "Missing parquet parameters for parquet_metadata table function");


The error message references 'parquet_metadata table function' but the actual function name is 'parquet_meta'. For consistency and clarity, the error message should use 'parquet_meta' to match the function name users actually invoke.

Suggested change

"Missing parquet parameters for parquet_metadata table function");

"Missing parquet parameters for parquet_meta table function");

Copilot · 2025-12-15T10:48:47Z

+
+private:
+
+


There are two consecutive blank lines after the 'private:' access specifier. This is inconsistent with typical C++ style guides which recommend a single blank line. Consider removing one of the blank lines to maintain consistent formatting.

Suggested change

Copilot · 2025-12-15T10:48:47Z

+        ));
+        String rawPath = normalizedParams.get(PATH);
+        if (Strings.isNullOrEmpty(rawPath)) {
+            throw new AnalysisException("Property 'path' is required for parquet_metadata");


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta' (as defined in line 64). This inconsistency could confuse users. All error messages should consistently use 'parquet_meta' to match the actual function name used in SQL queries.

Copilot · 2025-12-15T10:48:47Z

+                throw new AnalysisException(
+                        "Failed to parse storage properties for parquet_metadata: " + e.getMessage(), e);


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. This should be updated for consistency with the function name users actually invoke.

Copilot · 2025-12-15T10:48:47Z

+                throw new AnalysisException(
+                        "Failed to normalize parquet_metadata paths: " + e.getMessage(), e);


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. This should be updated for consistency with the function name users actually invoke.

Copilot · 2025-12-15T10:48:48Z

+    }
+    const TParquetMetadataParams& params = scan_range.parquet_params;
+    if (!params.__isset.paths || params.paths.empty()) {
+        return Status::InvalidArgument("Property 'path' must be set for parquet_metadata");


The error message references 'parquet_metadata' but the actual function name is 'parquet_meta'. For consistency and clarity, the error message should use 'parquet_meta' to match the function name users actually invoke.

Suggested change

return Status::InvalidArgument("Property 'path' must be set for parquet_metadata");

return Status::InvalidArgument("Property 'path' must be set for parquet_meta");

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-15T10:49:35Z

+/**
+ * Table-valued function for reading Parquet metadata.
+ * Currently works in two modes:
+ * - parquet_metadata: row-group/column statistics similar to DuckDB parquet_metadata()
+ * - parquet_schema: logical schema similar to DuckDB parquet_schema()
+ */


The class documentation describes modes as "parquet_metadata" and "parquet_schema" which matches the internal implementation, but the function name is "parquet_meta" (line 64). Consider updating the documentation to clarify that the function is called "parquet_meta" and accepts a "mode" parameter with values "parquet_metadata" or "parquet_schema".

Copilot · 2025-12-15T10:49:36Z

+
+private:
+
+


There are extra blank lines between the private access specifier and the first private method declaration. Remove one of the blank lines for consistency with the codebase style.

Suggested change

Copilot · 2025-12-15T10:49:36Z

+#!/bin/bash
+set -euo pipefail
+
+unset http_proxy
+unset https_proxy
+cd /mnt/disk1/chenjunwei/doris_build/doris/output/be
+./bin/start_be.sh --daemon


This file appears to be a personal development script with hardcoded paths (e.g., /mnt/disk1/chenjunwei/doris_build/doris/output/be) and should not be included in the PR. It should be removed from the repository.

Suggested change

#!/bin/bash

set -euo pipefail

unset http_proxy

unset https_proxy

cd /mnt/disk1/chenjunwei/doris_build/doris/output/be

./bin/start_be.sh --daemon

Copilot · 2025-12-15T10:49:36Z

+        try {
+            backend = Env.getCurrentSystemInfo().getBackendsByCurrentCluster().values().stream()
+                    .filter(Backend::isAlive)
+                    .findFirst()
+                    .orElse(null);
+        } catch (AnalysisException e) {
+            throw e;
+        }


The catch block unnecessarily catches and re-throws the same AnalysisException without adding any value. The try-catch block should be removed since getBackendsByCurrentCluster() already throws AnalysisException which will propagate naturally.

Suggested change

try {

backend = Env.getCurrentSystemInfo().getBackendsByCurrentCluster().values().stream()

.filter(Backend::isAlive)

.findFirst()

.orElse(null);

} catch (AnalysisException e) {

throw e;

}

backend = Env.getCurrentSystemInfo().getBackendsByCurrentCluster().values().stream()

.filter(Backend::isAlive)

.findFirst()

.orElse(null);

Copilot · 2025-12-15T10:49:37Z

+            case AZURE:
+                return TFileType.FILE_S3;
+            default:
+                throw new AnalysisException("Unsupported storage type for parquet_metadata: " + type);


Error messages reference "parquet_metadata" instead of the actual function name "parquet_meta". This inconsistency appears in multiple error messages (lines 125, 132, 139, 162, 174, 207, 226, 338). All error messages should use the actual function name "parquet_meta" for clarity.

Suggested change

throw new AnalysisException("Unsupported storage type for parquet_metadata: " + type);

throw new AnalysisException("Unsupported storage type for parquet_meta: " + type);

Copilot · 2025-12-15T10:49:37Z

+                "Missing parquet parameters for parquet_metadata table function");
+    }
+    const TParquetMetadataParams& params = scan_range.parquet_params;
+    if (!params.__isset.paths || params.paths.empty()) {
+        return Status::InvalidArgument("Property 'path' must be set for parquet_metadata");


Error messages in the backend reference "parquet_metadata" instead of the actual function name "parquet_meta". This is inconsistent with the function name defined on the frontend (NAME = "parquet_meta"). Update these error messages to use "parquet_meta" for consistency.

Suggested change

"Missing parquet parameters for parquet_metadata table function");

}

const TParquetMetadataParams& params = scan_range.parquet_params;

if (!params.__isset.paths || params.paths.empty()) {

return Status::InvalidArgument("Property 'path' must be set for parquet_metadata");

"Missing parquet parameters for parquet_meta table function");

}

const TParquetMetadataParams& params = scan_range.parquet_params;

if (!params.__isset.paths || params.paths.empty()) {

return Status::InvalidArgument("Property 'path' must be set for parquet_meta");

Copilot · 2025-12-15T10:49:38Z

+    public ParquetMetadataTableValuedFunction(Map<String, String> params) throws AnalysisException {
+        Map<String, String> normalizedParams = params.entrySet().stream().collect(Collectors.toMap(
+                entry -> entry.getKey().toLowerCase(),
+                Map.Entry::getValue,
+                (value1, value2) -> value2
+        ));
+        String rawPath = normalizedParams.get(PATH);
+        if (Strings.isNullOrEmpty(rawPath)) {
+            throw new AnalysisException("Property 'path' is required for parquet_metadata");
+        }
+        List<String> parsedPaths = Arrays.stream(rawPath.split(","))
+                .map(String::trim)
+                .filter(token -> !token.isEmpty())
+                .collect(Collectors.toList());
+        if (parsedPaths.isEmpty()) {
+            throw new AnalysisException("Property 'path' must contain at least one location");
+        }
+        List<String> normalizedPaths = parsedPaths;
+
+        String rawMode = normalizedParams.getOrDefault(MODE, MODE_METADATA);
+        mode = rawMode.toLowerCase();
+        if (!SUPPORTED_MODES.contains(mode)) {
+            throw new AnalysisException("Unsupported mode '" + rawMode + "' for parquet_metadata");
+        }
+
+        String firstPath = parsedPaths.get(0);
+        String scheme = null;
+        try {
+            scheme = new URI(firstPath).getScheme();
+        } catch (URISyntaxException e) {
+            scheme = null;
+        }
+        StorageProperties storageProperties = null;
+        if (Strings.isNullOrEmpty(scheme) || "file".equalsIgnoreCase(scheme)) {
+            this.fileType = TFileType.FILE_LOCAL;
+            this.properties = Collections.emptyMap();
+        } else {
+            Map<String, String> storageParams = new HashMap<>(normalizedParams);
+            // StorageProperties detects provider by "uri", we also hint support by scheme.
+            storageParams.put("uri", firstPath);
+            forceStorageSupport(storageParams, scheme.toLowerCase());
+            try {
+                storageProperties = StorageProperties.createPrimary(storageParams);
+            } catch (RuntimeException e) {
+                throw new AnalysisException(
+                        "Failed to parse storage properties for parquet_metadata: " + e.getMessage(), e);
+            }
+            this.fileType = mapToFileType(storageProperties.getType());
+            Map<String, String> backendProps = storageProperties.getBackendConfigProperties();
+            try {
+                List<String> tmpPaths = new ArrayList<>(parsedPaths.size());
+                for (String path : parsedPaths) {
+                    tmpPaths.add(storageProperties.validateAndNormalizeUri(path));
+                }
+                normalizedPaths = tmpPaths;
+            } catch (UserException e) {
+                throw new AnalysisException(
+                        "Failed to normalize parquet_metadata paths: " + e.getMessage(), e);
+            }
+            if (this.fileType == TFileType.FILE_HTTP && !backendProps.containsKey("uri")) {
+                backendProps = new HashMap<>(backendProps);
+                backendProps.put("uri", normalizedPaths.get(0));
+            }
+            this.properties = backendProps;
+        }
+
+        // Expand any glob patterns (e.g. *.parquet) to concrete file paths.
+        normalizedPaths = expandGlobPaths(normalizedPaths, storageProperties, this.fileType);
+
+        this.paths = ImmutableList.copyOf(normalizedPaths);
+    }


The new ParquetMetadataTableValuedFunction lacks test coverage. Given that other table-valued functions in the repository have test coverage (e.g., ExternalFileTableValuedFunctionTest.java), unit tests should be added to verify the function's behavior, including parameter validation, path expansion, mode handling, and error cases.

morningman · 2025-12-17T06:57:47Z

        // TINYINT/SMALLINT also need conversion via LittleIntPhysicalConverter.
        switch (type) {
        case TYPE_BOOLEAN:
+        case TYPE_TINYINT:


why adding these new types?

morningman · 2025-12-17T07:14:01Z

    std::stringstream ss;
-// clang-format off
+    // clang-format off
 #if defined(NDEBUG)


do not change this

morningman · 2025-12-17T07:15:08Z

+        if (!Strings.isNullOrEmpty(envDorisHome)) {
+            securePrefixCandidates.add(envDorisHome);
+        }
+        // 兜底：用路径所在目录作为前缀尝试剥离（针对在安全目录内的绝对路径）


morningman · 2025-12-17T07:23:00Z

+                Map.Entry::getValue,
+                (value1, value2) -> value2
+        ));
+        String rawPath = normalizedParams.get(PATH);


Use ExternalFileTableValuedFunction.URI_KEY instead of PATH

morningman · 2025-12-17T07:42:30Z

+        String tmpBloomColumn = null;
+        String tmpBloomLiteral = null;
+        if (MODE_BLOOM_PROBE.equals(mode)) {
+            tmpBloomColumn = normalizedParams.get(COLUMN);


what is different between COLUMN and COLUMN_NAME?

morningman · 2025-12-17T07:44:13Z

+        if (Strings.isNullOrEmpty(scheme) || "file".equalsIgnoreCase(scheme)) {
+            storageParams.put(StorageProperties.FS_LOCAL_SUPPORT, "true");
+        } else {
+            forceStorageSupport(storageParams, scheme.toLowerCase());


I don't think we need to force set fs.xx.support.
torageProperties.createPrimary can find the real storage type

morningman · 2025-12-17T07:46:54Z

+        if (parsedPath.isEmpty()) {
+            throw new AnalysisException("Property 'path' must contain at least one location");
+        }
+        List<String> parsedPaths = Collections.singletonList(parsedPath);


There is only one path, so no need to use List

morningman · 2025-12-17T07:49:17Z

+                                                 Map<String, String> storageParams,
+                                                 TFileType fileType) throws AnalysisException {
+        if (fileType == TFileType.FILE_LOCAL) {
+            return globLocal(pattern);


Why not reuse LocalTableValuedFunction.getFileListFromBackend() to get file list?

xylaaaaa · 2025-12-17T11:24:46Z

run buildall

hello-stephen · 2025-12-17T11:46:57Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.85% (1752/2194)
Line Coverage	65.74% (30771/46806)
Region Coverage	66.53% (15375/23111)
Branch Coverage	56.87% (8174/14374)

hello-stephen · 2025-12-17T12:26:17Z

FE UT Coverage Report

Increment line coverage 2.48% (5/202) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-12-17T14:55:28Z

BE UT Coverage Report

Increment line coverage 0.00% (0/913) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.34% (18834/35309)
Line Coverage	39.12% (174354/445746)
Region Coverage	33.74% (135022/400190)
Branch Coverage	34.59% (58133/168087)

xylaaaaa · 2025-12-18T07:47:38Z

run buildall

doris-robot · 2025-12-18T08:11:49Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.82% (1752/2195)
Line Coverage	65.72% (30763/46809)
Region Coverage	66.51% (15371/23112)
Branch Coverage	56.83% (8169/14374)

hello-stephen · 2025-12-18T08:55:47Z

FE UT Coverage Report

Increment line coverage 2.30% (5/217) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-12-18T11:23:08Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1046) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.32% (18833/35321)
Line Coverage	39.13% (174572/446080)
Region Coverage	33.73% (135085/400468)
Branch Coverage	34.60% (58213/168228)

xylaaaaa · 2025-12-19T06:05:39Z

run buildall

hello-stephen · 2025-12-19T06:35:12Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.72% (1757/2204)
Line Coverage	65.37% (30893/47256)
Region Coverage	66.04% (15408/23333)
Branch Coverage	56.56% (8195/14490)

doris-robot · 2025-12-22T09:16:21Z

BE UT Coverage Report

Increment line coverage 16.08% (169/1051) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.37% (18939/35483)
Line Coverage	39.22% (175560/447574)
Region Coverage	33.81% (135822/401712)
Branch Coverage	34.70% (58593/168834)

hello-stephen · 2025-12-22T10:10:46Z

BE Regression && UT Coverage Report

Increment line coverage 73.45% (772/1051) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.87% (25621/34686)
Line Coverage	61.26% (273392/446298)
Region Coverage	56.19% (228060/405905)
Branch Coverage	58.01% (98281/169412)

hello-stephen · 2025-12-22T10:21:01Z

FE Regression Coverage Report

Increment line coverage 64.59% (135/209) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-12-23T06:28:08Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2025-12-23T06:45:43Z

BE Regression && UT Coverage Report

Increment line coverage 73.45% (772/1051) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.87% (25621/34686)
Line Coverage	61.26% (273422/446298)
Region Coverage	56.20% (228127/405905)
Branch Coverage	58.04% (98321/169412)

### What problem does this PR solve? Expose Parquet file metadata via a table-valued function for inspection and debugging. ### Issue Number: close #xxx ### Related PR: #xxx ### Problem Summary: - Add a Parquet metadata TVF so users can query Parquet file metadata via SQL. - Backend adds a Parquet metadata reader and scan path; frontend wires the TVF definition. - Enables easy inspection of partitions/row groups/column stats to aid troubleshooting.

Expose Parquet file metadata via a table-valued function for inspection and debugging. - Add a Parquet metadata TVF so users can query Parquet file metadata via SQL. - Backend adds a Parquet metadata reader and scan path; frontend wires the TVF definition. - Enables easy inspection of partitions/row groups/column stats to aid troubleshooting.

) (#61474) Replacement for #61446 because the original PR head branch could not be updated from this environment. This branch contains: - [feat] add Parquet metadata TVF (#58972) - [fix](regression) stabilize parquet_meta tvf assertions across s3 buckets (#60938) - [fix] remove useless const_cast and explain const_cast for vec (4) (#56603) - [fix] fix parquet metadata TVF compile on branch-4.1 Please review this PR instead of #61446. --------- Co-authored-by: admiring_xm <[email protected]>

xylaaaaa marked this pull request as draft December 11, 2025 10:55

xylaaaaa changed the title ~~Impl parquet metadata~~ [feat] add Parquet metadata TVF Dec 11, 2025

morningman reviewed Dec 12, 2025

View reviewed changes

morningman requested a review from Copilot December 15, 2025 10:09

Copilot AI reviewed Dec 15, 2025

morningman reviewed Dec 15, 2025

View reviewed changes

morningman requested a review from Copilot December 15, 2025 10:28

Copilot started reviewing on behalf of morningman December 15, 2025 10:33

View session

Copilot started reviewing on behalf of morningman December 15, 2025 10:39

View session

Copilot AI reviewed Dec 15, 2025

View reviewed changes

morningman reviewed Dec 17, 2025

View reviewed changes

xylaaaaa added 7 commits December 19, 2025 16:02

init framwork

b60e9b7

impl

495528b

fix

04dfc88

fix

3c336b6

fix

4816e0f

fix

44eb1d7

fix

d45d4a4

morningman approved these changes Dec 23, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2025

suxiaogang223 approved these changes Dec 23, 2025

View reviewed changes

morningman added the dev/4.0.x label Dec 23, 2025

morningman merged commit d102aee into apache:master Dec 23, 2025
26 of 29 checks passed

github-actions Bot mentioned this pull request Dec 23, 2025

branch-4.0: [feat] add Parquet metadata TVF #58972 #59287

Closed

morningman added dev/4.1.x and removed dev/4.0.x labels Jan 12, 2026

xylaaaaa mentioned this pull request Mar 16, 2026

branch-4.1: [feat] add Parquet metadata TVF #58972 #61356

Closed

xylaaaaa mentioned this pull request Mar 16, 2026

branch-4.1: [feat] add Parquet metadata TVF #58972 #61357

Closed

morningman mentioned this pull request Mar 17, 2026

branch-4.1: [feat](tvf) add Parquet metadata TVF (#58972)(#60938)(#56603) #61446

Closed

xylaaaaa mentioned this pull request Mar 18, 2026

branch-4.1: [feat](tvf) add Parquet metadata TVF (#58972)(#60938)(#56603) #61474

Merged

morningman added dev/4.1.0-merged and removed dev/4.1.x labels Mar 19, 2026

	std::unordered_map<std::string, int> name_to_pos = _build_name_to_pos_map(slots);
	const auto& name_to_pos = _build_name_to_pos_map(slots);

-cd /mnt/disk1/chenjunwei/doris_build/doris/output/be
+# Set BE_DIR from first argument, environment variable, or default to current directory
+BE_DIR="${1:-${BE_DIR:-$(pwd)}}"
+cd "$BE_DIR"

	throw new AnalysisException("No files matched parquet_metadata path patterns: " + inputPaths);
	throw new AnalysisException("No files matched parquet_meta path patterns: " + inputPaths);

	throw new AnalysisException("Unsupported storage type for parquet_metadata: " + type);
	throw new AnalysisException("Unsupported storage type for parquet_meta: " + type);

	"Missing parquet parameters for parquet_metadata table function");
	"Missing parquet parameters for parquet_meta table function");

		throw new AnalysisException(
		"Failed to parse storage properties for parquet_metadata: " + e.getMessage(), e);

		throw new AnalysisException(
		"Failed to normalize parquet_metadata paths: " + e.getMessage(), e);

	return Status::InvalidArgument("Property 'path' must be set for parquet_metadata");
	return Status::InvalidArgument("Property 'path' must be set for parquet_meta");


		private:


		private:

Conversation

xylaaaaa commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Check List (For Author)

Uh oh!

Thearas commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

xylaaaaa commented Dec 11, 2025 •

edited

Loading