Skip to content

Commit 2ff68f4

Browse files
morningmanmorningmanclaude
authored
[doc](data-operate) add Row Binlog, Table Stream and @incr user docs for 5.0 (#4136)
## Versions - [x] dev - [ ] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) The feature ships in 5.0.0 (apache/doris#65418), so only `docs/` and `i18n/.../current` are touched. ## Languages - [x] Chinese - [x] English ## Docs Checklist - [x] Checked by AI - [ ] Test Cases Built - [x] Updated required version and language counterparts, or explained why not - [x] If only one language changed, confirmed whether source/translation counterparts need sync ## What changed and why User documentation for the 5.0 row-level change data features (Row Binlog, Table Stream, `@incr` incremental query). Facts were checked against apache/doris master as of 2026-09-14 (`AlterOp.allowOpRowBinlog`, the stream DDL commands, `binlog_block_reader_utils.h`, and the regression suites under `binlog/` and `table_stream/`). New user guide category **Change Data & Table Stream** (zh-CN: 数据变更与增量消费) under *Use Doris*, after *Data Update & Delete*, six pages in both languages: | Page | Content | |---|---| | `overview` | the problem, the two layers (Row Binlog → Table Stream / `@incr`), scenario table, table-model and consumption-type matrices, prerequisites, FAQ | | `quick-start` | orders-table walkthrough: create table with Row Binlog, create Stream, write two batches, read `UPDATE_BEFORE` / `UPDATE_AFTER` / `DELETE` / `APPEND`, consume with `INSERT ... SELECT`, check `table_stream_consumption` | | `row-binlog` | `binlog.*` properties, supported models (DUP, MoW without cluster key) and column limits, change record model with hidden columns, DDL restrictions, retention (TTL not effective yet), write overhead, `binlog()` TVF marked debug-only, error lookup table | | `incremental-query` | `t@incr(startTimestamp, endTimestamp, incrementType)` with `APPEND_ONLY` / `MIN_DELTA` / `DETAIL`, folding rules, examples, limits | | `table-stream` | CREATE / SHOW / ALTER / DROP STREAM, `append_only` / `min_delta` / `detail` on the same data, `show_initial_rows`, SELECT vs `INSERT ... SELECT` semantics, virtual columns, FAQ | | `table-stream-advanced` | partition-level offsets and `LAG`, `@snapshot()` / `@reset()`, consistent dimension joins, concurrency, effect of base-table DDL, monitoring and recovery, error table | Reference pages: `CREATE/ALTER/DROP-STREAM`, `SHOW-STREAMS`, `SHOW-CREATE-STREAM` (new `stream/` folder under table-and-view), the `binlog` TVF, `information_schema.table_streams` / `table_stream_consumption`, plus three `binlog.*` rows in `CREATE-TABLE.md`. `sidebars.ts` and `current.json` gain the category, the *Table Stream* statement group, the TVF and the two system tables. Editorial decisions reflected in the text: FE switches `enable_feature_binlog` / `enable_table_stream` documented as default-off (restart FE); Row Binlog can only be enabled at `CREATE TABLE`; `binlog.ttl_seconds` / `max_bytes` / `max_history_nums` documented as "no effect in the current version"; `binlog()` is debug-only; `@incr` is documented for the integrated storage-compute mode only; time travel (`FOR TIME AS OF`) is intentionally not documented yet. The last commit restructures all six pages with the repo's doc-optimize workflow (prerequisites / step overview / FAQ / error tables, per-section knowledge-type comments, `keywords`, links to the STREAM reference pages) without changing any command, output or error string. Both language trees have the same headings, tables and comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BjEn6JqDkb3pUUw4D6W3Nq --------- Co-authored-by: morningman <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 3aae518 commit 2ff68f4

35 files changed

Lines changed: 4771 additions & 0 deletions

File tree

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
{
3+
"title": "table_stream_consumption",
4+
"language": "en",
5+
"description": "Records the consumption offset, backlog, and last consumption time of every Table Stream on each partition of its base table."
6+
}
7+
---
8+
9+
## Overview
10+
11+
Records the consumption offset, backlog, and last consumption time of every Table Stream on each partition of its base table. Each row corresponds to one consumption unit (a base table partition) of one Stream. See [Table Stream Advanced](../../../data-operate/incremental/table-stream-advanced#partition-level-consumption-offsets) for usage.
12+
13+
## Database
14+
15+
`information_schema`
16+
17+
## Table Information
18+
19+
| Column | Type | Description |
20+
| :--- | :--- | :--- |
21+
| DB_NAME | varchar(64) | The database of the Stream |
22+
| STREAM_NAME | varchar(64) | Stream name |
23+
| STREAM_ID | bigint | Stream ID |
24+
| UNIT | varchar(64) | The consumption unit, i.e. the partition name of the base table. A table without explicit partitions has a single partition named after the table |
25+
| CONSUMPTION_STATUS | varchar(64) | The commit timestamp (TSO) the partition has been consumed up to; `N/A` means the partition has never been consumed |
26+
| LAG | varchar(64) | The difference between the latest committed TSO of the partition and the consumed TSO; `0` means no backlog, `N/A` means the partition has data but has never been consumed. The low 18 bits of a TSO are a logical counter, so `LAG` divided by 262144 is roughly the backlog in milliseconds |
27+
| LAST_CONSUMPTION_TIME | bigint | The time of the most recent consumption of the partition (millisecond timestamp); `-1` means never consumed |
28+
29+
## Examples
30+
31+
```sql
32+
SELECT UNIT, CONSUMPTION_STATUS, LAG, LAST_CONSUMPTION_TIME
33+
FROM information_schema.table_stream_consumption
34+
WHERE DB_NAME = 'demo' AND STREAM_NAME = 'orders_stream'
35+
ORDER BY UNIT;
36+
```
37+
38+
```text
39+
+-----------+--------------------+-----------+-----------------------+
40+
| UNIT | CONSUMPTION_STATUS | LAG | LAST_CONSUMPTION_TIME |
41+
+-----------+--------------------+-----------+-----------------------+
42+
| p20260913 | 469041123123200000 | 0 | 1789267206000 |
43+
| p20260914 | 469067681628160003 | 262144000 | 1789351206000 |
44+
| p20260915 | N/A | 0 | -1 |
45+
+-----------+--------------------+-----------+-----------------------+
46+
```
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
{
3+
"title": "table_streams",
4+
"language": "en",
5+
"description": "Records the definition and state of every Table Stream in the cluster."
6+
}
7+
---
8+
9+
## Overview
10+
11+
Records the definition and state of every Table Stream in the cluster. Each row corresponds to one Stream. See [Table Stream Basics](../../../data-operate/incremental/table-stream) for usage.
12+
13+
## Database
14+
15+
`information_schema`
16+
17+
## Table Information
18+
19+
| Column | Type | Description |
20+
| :--- | :--- | :--- |
21+
| DB_NAME | varchar(64) | The database of the Stream |
22+
| STREAM_NAME | varchar(64) | Stream name |
23+
| STREAM_ID | bigint | Stream ID |
24+
| STREAM_TYPE | varchar(64) | Stream type; Streams on internal tables are `OLAP_TABLE_STREAM` |
25+
| CONSUME_TYPE | varchar(64) | Consumption type: `APPEND_ONLY`, `MIN_DELTA`, `DETAIL` |
26+
| STREAM_COMMENT | string | The comment of the Stream |
27+
| BASE_TABLE_NAME | varchar(64) | Base table name |
28+
| BASE_TABLE_DB | varchar(64) | The database of the base table |
29+
| BASE_TABLE_CTL | varchar(64) | The catalog of the base table |
30+
| BASE_TABLE_TYPE | varchar(64) | Base table type |
31+
| ENABLED | boolean | Whether the Stream is usable |
32+
| IS_STALE | boolean | Whether the Stream has become stale (for example, its change records were cleaned up and it can no longer read from its consumption offset) |
33+
| STALE_REASON | string | The reason for being stale; `N/A` when not stale |
34+
35+
## Examples
36+
37+
```sql
38+
SELECT STREAM_NAME, CONSUME_TYPE, BASE_TABLE_DB, BASE_TABLE_NAME, ENABLED, IS_STALE, STALE_REASON
39+
FROM information_schema.table_streams
40+
WHERE DB_NAME = 'demo';
41+
```
42+
43+
```text
44+
+---------------+--------------+---------------+-----------------+---------+----------+--------------+
45+
| STREAM_NAME | CONSUME_TYPE | BASE_TABLE_DB | BASE_TABLE_NAME | ENABLED | IS_STALE | STALE_REASON |
46+
+---------------+--------------+---------------+-----------------+---------+----------+--------------+
47+
| orders_stream | MIN_DELTA | demo | orders | 1 | 0 | N/A |
48+
+---------------+--------------+---------------+-----------------+---------+----------+--------------+
49+
```
Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
---
2+
{
3+
"title": "Incremental Query",
4+
"language": "en",
5+
"description": "Read row-level changes of a Doris table by time window with @incr, without a Table Stream: syntax, examples and limits of APPEND_ONLY, MIN_DELTA, DETAIL.",
6+
"keywords": [
7+
"@incr",
8+
"Doris incremental query",
9+
"incremental query",
10+
"startTimestamp",
11+
"endTimestamp",
12+
"incrementType",
13+
"APPEND_ONLY",
14+
"MIN_DELTA",
15+
"DETAIL",
16+
"changes by time window",
17+
"stateless incremental read",
18+
"__DORIS_BINLOG_OP__",
19+
"Row Binlog query",
20+
"external scheduler incremental ETL",
21+
"INCR query requires ROW binlog enabled on base table",
22+
"Table Stream vs @incr"
23+
]
24+
}
25+
---
26+
27+
<!-- Knowledge type: Operations guide + Syntax description -->
28+
<!-- Use cases: Ad-hoc incremental reads by time range / External systems managing their own positions -->
29+
30+
On a table with [Row Binlog](row-binlog) enabled, besides consuming changes through a [Table Stream](table-stream), you can read the changes of the table within a time window directly in a query with `@incr`, without creating any object or recording a consumption offset. It is a stateless, ordinary query and can be combined freely with WHERE, JOIN, aggregation, and so on.
31+
32+
:::caution Experimental feature
33+
This feature is available since version 5.0.0 and is experimental. Use `@incr` in the integrated storage-compute mode for now; support in the compute-storage decoupled mode is still being completed.
34+
:::
35+
36+
## Differences from Table Stream
37+
38+
<!-- Knowledge type: Comparison + Selection guide -->
39+
40+
| | `@incr` incremental query | Table Stream |
41+
|---|---|---|
42+
| Object to create | No | Yes |
43+
| Read range | A time window you specify | From the last consumption offset to now |
44+
| Who tracks the position | You (for example an external scheduler records the last window) | Doris, per partition, advanced atomically with the consuming transaction |
45+
| Repeated reads | The same window can be read again and again | Consumed changes are not returned again |
46+
| Best for | Ad-hoc analysis, looking back at changes in a period, external systems with their own position management | Continuous incremental ETL that must be exactly-once |
47+
48+
## Prerequisites
49+
50+
<!-- Knowledge type: Environment requirements -->
51+
52+
- Doris 5.0.0 or later, with `enable_feature_binlog = true` on the FE.
53+
- Row Binlog enabled on the base table, otherwise the query fails with `INCR query requires ROW binlog enabled on base table.`.
54+
- For `MIN_DELTA` mode, a Unique Key MoW base table with `binlog.need_historical_value` enabled.
55+
- The integrated storage-compute mode for now.
56+
57+
## Syntax
58+
59+
<!-- Knowledge type: Syntax reference -->
60+
61+
```sql
62+
SELECT ... FROM <table_name>@incr(
63+
["startTimestamp" = "<datetime>",]
64+
["endTimestamp" = "<datetime>",]
65+
["incrementType" = "<APPEND_ONLY | MIN_DELTA | DETAIL>"]
66+
) [PARTITION (<partition_name>, ...)] [<alias>]
67+
[WHERE ...]
68+
```
69+
70+
### Parameters
71+
72+
<!-- Knowledge type: Parameter reference -->
73+
74+
| Parameter | Default | Description |
75+
|---|---|---|
76+
| `startTimestamp` | unbounded | Optional. Start of the window, format `yyyy-MM-dd HH:mm:ss`, parsed in the session `time_zone`. Changes whose commit time is **greater than or equal to** the start are returned |
77+
| `endTimestamp` | unbounded | Optional. End of the window, same format. Changes whose commit time is **less than** the end are returned |
78+
| `incrementType` | `MIN_DELTA` | Optional. Incremental mode, see [The three incremental modes](#the-three-incremental-modes) |
79+
80+
All three parameters can be omitted; `t@incr()` reads the entire change history in `MIN_DELTA` mode. The window is a left-closed, right-open interval `[startTimestamp, endTimestamp)`; a start later than the end, or a start in the future, returns an empty result.
81+
82+
### Result columns
83+
84+
The result contains the visible columns of the base table plus the following hidden columns:
85+
86+
| Hidden column | Description |
87+
|---|---|
88+
| `__DORIS_BINLOG_OP__` | Change type: `0` insert (APPEND), `1` delete (DELETE), `2` before update (UPDATE_BEFORE), `3` after update (UPDATE_AFTER) |
89+
| `__DORIS_BINLOG_TSO__` | Commit timestamp of the change |
90+
| `__DORIS_BINLOG_LSN__` | Sequence number within the transaction, used for ordering together with the TSO |
91+
92+
## The three incremental modes
93+
94+
<!-- Knowledge type: Behavior rules + Selection guide -->
95+
96+
| Mode | Output | Base table requirements |
97+
|---|---|---|
98+
| `APPEND_ONLY` | Rows inserted within the window. Updates and deletes are not emitted | Row Binlog enabled |
99+
| `MIN_DELTA` | The net change of each key within the window; see the folding rules below | MoW tables must have `binlog.need_historical_value` enabled |
100+
| `DETAIL` | Every change within the window; an update is split into an UPDATE_BEFORE row and an UPDATE_AFTER row | Row Binlog enabled |
101+
102+
`MIN_DELTA` compares the state of each key at the start and at the end of the window:
103+
104+
| Key exists at the window start | Key exists at the window end | Output |
105+
|---|---|---|
106+
| No | Yes | One APPEND |
107+
| Yes | Yes, and modified in between | One UPDATE_BEFORE (value at the start) + one UPDATE_AFTER (value at the end) |
108+
| Yes | No | One DELETE |
109+
| No | No | Nothing |
110+
111+
On Duplicate Key tables the three modes return the same result: Duplicate Key tables only have inserts, so there is nothing to fold.
112+
113+
## Examples
114+
115+
<!-- Knowledge type: Operational examples -->
116+
<!-- Use cases: Comparing the output of the three modes on the same batch of changes -->
117+
118+
### Data preparation
119+
120+
The examples use the `orders` table from the [Quick Start](quick-start) (Unique Key MoW with the before image enabled). Assume the two batches were written at the following times:
121+
122+
```sql
123+
-- 10:00, first batch
124+
INSERT INTO orders VALUES (1, 'created', 100.00), (2, 'created', 200.00), (3, 'created', 300.00);
125+
126+
-- 10:05, second batch
127+
INSERT INTO orders VALUES (1, 'paid', 100.00); -- update order 1
128+
DELETE FROM orders WHERE order_id = 2; -- delete order 2
129+
INSERT INTO orders VALUES (4, 'created', 400.00); -- new order 4
130+
INSERT INTO orders VALUES (5, 'created', 500.00); -- new order 5
131+
DELETE FROM orders WHERE order_id = 5; -- delete order 5 again
132+
```
133+
134+
### MIN_DELTA: net changes between 10:03 and 10:10
135+
136+
```sql
137+
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
138+
FROM orders@incr(
139+
"startTimestamp" = "2026-09-14 10:03:00",
140+
"endTimestamp" = "2026-09-14 10:10:00",
141+
"incrementType" = "MIN_DELTA"
142+
)
143+
ORDER BY order_id, op;
144+
```
145+
146+
```text
147+
+----------+---------+--------+------+
148+
| order_id | status | amount | op |
149+
+----------+---------+--------+------+
150+
| 1 | created | 100.00 | 2 |
151+
| 1 | paid | 100.00 | 3 |
152+
| 2 | created | 200.00 | 1 |
153+
| 4 | created | 400.00 | 0 |
154+
+----------+---------+--------+------+
155+
```
156+
157+
Order 5 was inserted and deleted within the window; its net change is empty, so it is not emitted.
158+
159+
### APPEND_ONLY: rows inserted within the same window
160+
161+
```sql
162+
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
163+
FROM orders@incr(
164+
"startTimestamp" = "2026-09-14 10:03:00",
165+
"endTimestamp" = "2026-09-14 10:10:00",
166+
"incrementType" = "APPEND_ONLY"
167+
)
168+
ORDER BY order_id;
169+
```
170+
171+
```text
172+
+----------+---------+--------+------+
173+
| order_id | status | amount | op |
174+
+----------+---------+--------+------+
175+
| 4 | created | 400.00 | 0 |
176+
| 5 | created | 500.00 | 0 |
177+
+----------+---------+--------+------+
178+
```
179+
180+
The update of order 1 and the deletion of order 2 are filtered out; the insert of order 5 is emitted, and its later deletion does not cancel that insert record.
181+
182+
### DETAIL: every change within the same window
183+
184+
```sql
185+
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
186+
FROM orders@incr(
187+
"startTimestamp" = "2026-09-14 10:03:00",
188+
"endTimestamp" = "2026-09-14 10:10:00",
189+
"incrementType" = "DETAIL"
190+
)
191+
ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, op;
192+
```
193+
194+
```text
195+
+----------+---------+--------+------+
196+
| order_id | status | amount | op |
197+
+----------+---------+--------+------+
198+
| 1 | created | 100.00 | 2 |
199+
| 1 | paid | 100.00 | 3 |
200+
| 2 | created | 200.00 | 1 |
201+
| 4 | created | 400.00 | 0 |
202+
| 5 | created | 500.00 | 0 |
203+
| 5 | created | 500.00 | 1 |
204+
+----------+---------+--------+------+
205+
```
206+
207+
### No window: net changes since the table was created
208+
209+
```sql
210+
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
211+
FROM orders@incr()
212+
ORDER BY order_id;
213+
```
214+
215+
```text
216+
+----------+---------+--------+------+
217+
| order_id | status | amount | op |
218+
+----------+---------+--------+------+
219+
| 1 | paid | 100.00 | 0 |
220+
| 3 | created | 300.00 | 0 |
221+
| 4 | created | 400.00 | 0 |
222+
+----------+---------+--------+------+
223+
```
224+
225+
Counting from table creation, every key is absent at the start, so every key that still exists is emitted as APPEND with its latest value, and the deleted orders 2 and 5 are not emitted.
226+
227+
## Recommendations
228+
229+
<!-- Knowledge type: Usage recommendations -->
230+
<!-- Use cases: Periodic increments driven by an external scheduler -->
231+
232+
- When an external scheduler drives the increments, use the previous `endTimestamp` as the next `startTimestamp`; the left-closed, right-open window guarantees no gaps and no duplicates. Note that windows are defined by commit time: a transaction that is still running does not appear in the current window and shows up in the window that contains its commit.
233+
- If only the latest values matter and the values before an update are not needed, keep only the rows with `__DORIS_BINLOG_OP__ IN (0, 1, 3)`.
234+
- An `@incr` query reads all change records within the window; the larger the window, the more it reads. Specify the window whenever possible.
235+
236+
## Limitations and common errors
237+
238+
<!-- Knowledge type: Troubleshooting -->
239+
240+
| Limitation | Error message | Action |
241+
|---|---|---|
242+
| The base table must have Row Binlog enabled | `INCR query requires ROW binlog enabled on base table.` | Recreate the base table with `binlog.enable` + `binlog.format = ROW`, see [Row Binlog](row-binlog#enabling-row-binlog) |
243+
| `MIN_DELTA` requires a Unique Key MoW table with `binlog.need_historical_value` enabled | `MIN_DELTA INCR query requires base table to be UNIQUE KEY with enable_unique_key_merge_on_write=true` or `... requires base table to enable binlog.need_historical_value=true` | Recreate the base table accordingly, or use `APPEND_ONLY` / `DETAIL` |
244+
| Only the base index can be read | - | Materialized views and rollups cannot be specified |
245+
| The `PREAGGOPEN` hint is not supported | - | Remove the hint |
246+
| Only `startTimestamp`, `endTimestamp`, and `incrementType` are accepted | `Unsupported parameter in incr query` | Check the spelling of the parameter names |

0 commit comments

Comments
 (0)