Skip to content

Commit 6b54141

Browse files
authored
[doc](lakehouse) Document Iceberg and Paimon file metadata columns (#4125)
## Versions - [x] dev - [ ] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) ## Languages - [x] Chinese - [x] English ## Docs Checklist - [x] Checked by AI - [x] Test Cases Built - [x] Updated required version and language counterparts, or explained why not - [x] If only one language changed, confirmed whether source/translation counterparts need sync ## Summary - Document the Iceberg `_file` and `_pos` hidden metadata columns. - Document the Paimon `__paimon_file_path` and `__paimon_row_index` hidden metadata columns. - Explain physical row-position semantics, examples, session variables, reader compatibility, and reserved-name restrictions. ## Related changes - Apache Doris PR: apache/doris#67207 - Requirement: apache/doris#66816 ## Version and language coverage The feature was newly merged into the Doris development branch and has not been backported to released 4.x or 3.x branches, so this PR updates only the current/dev English and Chinese documentation. ## Validation - `git diff --check` - Docs governance changed-file report - `DOCS_VERSIONS=current` English and Chinese Docusaurus production build
1 parent 56946be commit 6b54141

4 files changed

Lines changed: 208 additions & 0 deletions

File tree

‎docs/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1506,6 +1506,58 @@ SELECT * FROM iceberg_tbl LIMIT 10;
15061506
SELECT * FROM iceberg.iceberg_db.iceberg_tbl LIMIT 10;
15071507
```
15081508

1509+
### File Metadata Columns
1510+
1511+
Doris exposes two hidden metadata columns for Iceberg data tables. You can use them to locate the physical data file and row that produced each query result.
1512+
1513+
| Column | Type | Description |
1514+
| --- | --- | --- |
1515+
| `_file` | `STRING NOT NULL` | The original path recorded for the Iceberg data file. |
1516+
| `_pos` | `BIGINT NOT NULL` | The zero-based physical row position within the complete data file. |
1517+
1518+
Prerequisites: configure an Iceberg Catalog as described in the preceding examples, and create an Iceberg data table that contains an `id` column and uses Parquet or ORC files. The following query assumes that `iceberg_tbl` already contains rows with `id` values `1` and `2`:
1519+
1520+
```sql
1521+
SELECT id, `_file`, `_pos`
1522+
FROM iceberg.iceberg_db.iceberg_tbl
1523+
ORDER BY `_file`, `_pos`;
1524+
```
1525+
1526+
Example result (the file path depends on the table location):
1527+
1528+
```text
1529+
+----+---------------------------------------+------+
1530+
| id | _file | _pos |
1531+
+----+---------------------------------------+------+
1532+
| 1 | s3://bucket/warehouse/db/table/a.orc | 0 |
1533+
| 2 | s3://bucket/warehouse/db/table/a.orc | 1 |
1534+
+----+---------------------------------------+------+
1535+
```
1536+
1537+
The columns do not appear in `SELECT *` or `DESC` by default.
1538+
1539+
To include hidden columns in `SELECT *` and `DESC`, enable `show_hidden_columns` for the session:
1540+
1541+
```sql
1542+
SET show_hidden_columns = true;
1543+
```
1544+
1545+
Relevant session variables:
1546+
1547+
| Variable | Type | Default | Required | Effect |
1548+
| --- | --- | --- | --- | --- |
1549+
| `show_hidden_columns` | `BOOLEAN` | `false` | No | Includes hidden columns in `SELECT *` and `DESC`. It is not needed when the columns are referenced explicitly. |
1550+
| `enable_file_scanner_v2` | `BOOLEAN` | `true` | No | Must remain enabled when a query references these metadata columns. |
1551+
| `force_jni_scanner` | `BOOLEAN` | `false` | No | Must remain disabled because the JNI reader cannot produce these metadata columns. |
1552+
1553+
The row position is relative to the complete physical file. Predicate filtering, delete processing, and scan splitting do not renumber the remaining rows. Therefore, the combination of `_file` and `_pos` can locate a physical record within the current table snapshot. Data-file rewrites or compaction can change both values, so they must not be used as a stable logical row ID.
1554+
1555+
:::caution
1556+
These columns are available only when an Iceberg data table is read through the FileScannerV2 native Parquet or ORC reader. A query that references them fails if `enable_file_scanner_v2` is disabled, `force_jni_scanner` is enabled, or the selected data files cannot use the native Parquet/ORC reader.
1557+
1558+
`_file` and `_pos` are reserved names, compared case-insensitively. Do not use them as physical column names in an Iceberg table.
1559+
:::
1560+
15091561
### Time Travel
15101562

15111563
You can read a specific snapshot of an Iceberg table.

‎docs/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -980,6 +980,58 @@ SELECT * FROM paimon_tbl LIMIT 10;
980980
SELECT * FROM paimon_ctl.paimon_db.paimon_tbl LIMIT 10;
981981
```
982982

983+
### File Metadata Columns
984+
985+
Doris exposes two hidden metadata columns for Paimon data tables. You can use them to locate the physical data file and row that produced each query result.
986+
987+
| Column | Type | Description |
988+
| --- | --- | --- |
989+
| `__paimon_file_path` | `STRING NOT NULL` | The original path returned by Paimon for the raw data file. |
990+
| `__paimon_row_index` | `BIGINT NOT NULL` | The zero-based physical row position within the complete data file. |
991+
992+
Prerequisites: configure a Paimon Catalog as described in the preceding examples, and create a Paimon data table that contains an `id` column and uses Parquet or ORC files. The following query assumes that `paimon_tbl` already contains rows with `id` values `1` and `2`:
993+
994+
```sql
995+
SELECT id, `__paimon_file_path`, `__paimon_row_index`
996+
FROM paimon_ctl.paimon_db.paimon_tbl
997+
ORDER BY `__paimon_file_path`, `__paimon_row_index`;
998+
```
999+
1000+
Example result (the file path depends on the table location):
1001+
1002+
```text
1003+
+----+------------------------------------------+--------------------+
1004+
| id | __paimon_file_path | __paimon_row_index |
1005+
+----+------------------------------------------+--------------------+
1006+
| 1 | s3://bucket/warehouse/db/table/a.parquet | 0 |
1007+
| 2 | s3://bucket/warehouse/db/table/a.parquet | 1 |
1008+
+----+------------------------------------------+--------------------+
1009+
```
1010+
1011+
The columns do not appear in `SELECT *` or `DESC` by default.
1012+
1013+
To include hidden columns in `SELECT *` and `DESC`, enable `show_hidden_columns` for the session:
1014+
1015+
```sql
1016+
SET show_hidden_columns = true;
1017+
```
1018+
1019+
Relevant session variables:
1020+
1021+
| Variable | Type | Default | Required | Effect |
1022+
| --- | --- | --- | --- | --- |
1023+
| `show_hidden_columns` | `BOOLEAN` | `false` | No | Includes hidden columns in `SELECT *` and `DESC`. It is not needed when the columns are referenced explicitly. |
1024+
| `enable_file_scanner_v2` | `BOOLEAN` | `true` | No | Must remain enabled when a query references these metadata columns. |
1025+
| `force_jni_scanner` | `BOOLEAN` | `false` | No | Must remain disabled because the JNI reader cannot produce these metadata columns. |
1026+
1027+
The row position is relative to the complete physical file. Predicate filtering, deletion-vector processing, and scan splitting do not renumber the remaining rows. Therefore, the combination of `__paimon_file_path` and `__paimon_row_index` can locate a physical record within the current table snapshot. Data-file rewrites or compaction can change both values, so they must not be used as a stable logical row ID.
1028+
1029+
:::caution
1030+
These columns are available only when every Paimon split participating in the query can be read as a raw data file through the FileScannerV2 native Parquet or ORC reader. A query that references them fails for JNI or mixed native/JNI scans, when `enable_file_scanner_v2` is disabled, when `force_jni_scanner` is enabled, or when the selected data files cannot use the native Parquet/ORC reader.
1031+
1032+
`__paimon_file_path` and `__paimon_row_index` are reserved names, compared case-insensitively. Do not use them as physical column names in a Paimon table.
1033+
:::
1034+
9831035
### Batch Incremental Query
9841036

9851037
> Since version 3.1.0

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/iceberg-catalog.mdx‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1527,6 +1527,58 @@ SELECT * FROM iceberg_tbl LIMIT 10;
15271527
SELECT * FROM iceberg.iceberg_db.iceberg_tbl LIMIT 10;
15281528
```
15291529

1530+
### 文件元数据列
1531+
1532+
Doris 为 Iceberg 数据表提供两个隐藏元数据列,可用于定位每条查询结果对应的物理数据文件和文件内行位置。
1533+
1534+
| 列名 | 类型 | 说明 |
1535+
| --- | --- | --- |
1536+
| `_file` | `STRING NOT NULL` | Iceberg 数据文件元数据中记录的原始路径。 |
1537+
| `_pos` | `BIGINT NOT NULL` | 该行在完整数据文件中的物理位置,从 0 开始。 |
1538+
1539+
前置条件:按照前文示例配置 Iceberg Catalog,并创建一个包含 `id` 列、使用 Parquet 或 ORC 文件的 Iceberg 数据表。以下查询假设 `iceberg_tbl` 中已包含 `id` 为 `1` 和 `2` 的记录:
1540+
1541+
```sql
1542+
SELECT id, `_file`, `_pos`
1543+
FROM iceberg.iceberg_db.iceberg_tbl
1544+
ORDER BY `_file`, `_pos`;
1545+
```
1546+
1547+
示例结果(文件路径由实际表位置决定):
1548+
1549+
```text
1550+
+----+---------------------------------------+------+
1551+
| id | _file | _pos |
1552+
+----+---------------------------------------+------+
1553+
| 1 | s3://bucket/warehouse/db/table/a.orc | 0 |
1554+
| 2 | s3://bucket/warehouse/db/table/a.orc | 1 |
1555+
+----+---------------------------------------+------+
1556+
```
1557+
1558+
默认情况下,这些列不会出现在 `SELECT *` 或 `DESC` 的结果中。
1559+
1560+
如需在 `SELECT *` 和 `DESC` 中显示隐藏列,可以在当前会话开启 `show_hidden_columns`:
1561+
1562+
```sql
1563+
SET show_hidden_columns = true;
1564+
```
1565+
1566+
相关会话变量:
1567+
1568+
| 变量 | 类型 | 默认值 | 是否必需 | 作用 |
1569+
| --- | --- | --- | --- | --- |
1570+
| `show_hidden_columns` | `BOOLEAN` | `false` | 否 | 在 `SELECT *` 和 `DESC` 中显示隐藏列。显式引用列名时无需开启。 |
1571+
| `enable_file_scanner_v2` | `BOOLEAN` | `true` | 否 | 查询引用这些元数据列时必须保持开启。 |
1572+
| `force_jni_scanner` | `BOOLEAN` | `false` | 否 | 必须保持关闭,因为 JNI Reader 无法生成这些元数据列。 |
1573+
1574+
行位置以完整物理文件为基准。谓词过滤、Delete 处理及扫描切分不会对保留行重新编号。因此,`_file` 和 `_pos` 的组合可以定位当前表快照中的一条物理记录。数据文件重写或 Compaction 后,两列的值可能变化,不能将其作为稳定的逻辑行 ID。
1575+
1576+
:::caution 注意
1577+
仅当 Iceberg 数据表通过 FileScannerV2 原生 Parquet 或 ORC Reader 读取时,才能使用这些列。如果关闭 `enable_file_scanner_v2`、开启 `force_jni_scanner`,或者所选数据文件无法使用原生 Parquet/ORC Reader,引用这些列的查询会失败。
1578+
1579+
`_file` 和 `_pos` 是保留列名,名称比较不区分大小写。请勿在 Iceberg 表中使用它们作为物理列名。
1580+
:::
1581+
15301582
### 时间旅行
15311583

15321584
支持读取 Iceberg 表指定的 Snapshot。

‎i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -976,6 +976,58 @@ SELECT * FROM paimon_tbl LIMIT 10;
976976
SELECT * FROM paimon_ctl.paimon_db.paimon_tbl LIMIT 10;
977977
```
978978

979+
### 文件元数据列
980+
981+
Doris 为 Paimon 数据表提供两个隐藏元数据列,可用于定位每条查询结果对应的物理数据文件和文件内行位置。
982+
983+
| 列名 | 类型 | 说明 |
984+
| --- | --- | --- |
985+
| `__paimon_file_path` | `STRING NOT NULL` | Paimon 为原始数据文件返回的路径。 |
986+
| `__paimon_row_index` | `BIGINT NOT NULL` | 该行在完整数据文件中的物理位置,从 0 开始。 |
987+
988+
前置条件:按照前文示例配置 Paimon Catalog,并创建一个包含 `id` 列、使用 Parquet 或 ORC 文件的 Paimon 数据表。以下查询假设 `paimon_tbl` 中已包含 `id` 为 `1` 和 `2` 的记录:
989+
990+
```sql
991+
SELECT id, `__paimon_file_path`, `__paimon_row_index`
992+
FROM paimon_ctl.paimon_db.paimon_tbl
993+
ORDER BY `__paimon_file_path`, `__paimon_row_index`;
994+
```
995+
996+
示例结果(文件路径由实际表位置决定):
997+
998+
```text
999+
+----+------------------------------------------+--------------------+
1000+
| id | __paimon_file_path | __paimon_row_index |
1001+
+----+------------------------------------------+--------------------+
1002+
| 1 | s3://bucket/warehouse/db/table/a.parquet | 0 |
1003+
| 2 | s3://bucket/warehouse/db/table/a.parquet | 1 |
1004+
+----+------------------------------------------+--------------------+
1005+
```
1006+
1007+
默认情况下,这些列不会出现在 `SELECT *` 或 `DESC` 的结果中。
1008+
1009+
如需在 `SELECT *` 和 `DESC` 中显示隐藏列,可以在当前会话开启 `show_hidden_columns`:
1010+
1011+
```sql
1012+
SET show_hidden_columns = true;
1013+
```
1014+
1015+
相关会话变量:
1016+
1017+
| 变量 | 类型 | 默认值 | 是否必需 | 作用 |
1018+
| --- | --- | --- | --- | --- |
1019+
| `show_hidden_columns` | `BOOLEAN` | `false` | 否 | 在 `SELECT *` 和 `DESC` 中显示隐藏列。显式引用列名时无需开启。 |
1020+
| `enable_file_scanner_v2` | `BOOLEAN` | `true` | 否 | 查询引用这些元数据列时必须保持开启。 |
1021+
| `force_jni_scanner` | `BOOLEAN` | `false` | 否 | 必须保持关闭,因为 JNI Reader 无法生成这些元数据列。 |
1022+
1023+
行位置以完整物理文件为基准。谓词过滤、Deletion Vector 处理及扫描切分不会对保留行重新编号。因此,`__paimon_file_path` 和 `__paimon_row_index` 的组合可以定位当前表快照中的一条物理记录。数据文件重写或 Compaction 后,两列的值可能变化,不能将其作为稳定的逻辑行 ID。
1024+
1025+
:::caution 注意
1026+
仅当查询涉及的每个 Paimon Split 都能作为原始数据文件,通过 FileScannerV2 原生 Parquet 或 ORC Reader 读取时,才能使用这些列。对于 JNI 或原生/JNI 混合扫描、关闭 `enable_file_scanner_v2`、开启 `force_jni_scanner`,以及所选数据文件无法使用原生 Parquet/ORC Reader 的情况,引用这些列的查询会失败。
1027+
1028+
`__paimon_file_path` 和 `__paimon_row_index` 是保留列名,名称比较不区分大小写。请勿在 Paimon 表中使用它们作为物理列名。
1029+
:::
1030+
9791031
### 增量查询
9801032

9811033
> 该功能自 3.1.0 版本支持

0 commit comments

Comments
 (0)