Skip to content

[Improve](source) Follow the Doris rename of row binlog to changelog - #710

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:changelog-rename
Oct 21, 2026
Merged

morningman merged 1 commit into
apache:masterfrom
morningman:changelog-rename

Conversation

@morningman

@morningman morningman commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: None (follows apache/doris#69311)

Problem Summary:

Doris renamed the experimental row binlog feature to changelog in apache/doris#69311 (merged into Doris master as 73a987e6059). The incremental source added in #679 reads that feature, and two kinds of names it depends on changed:

  • A table turns the feature on with "changelog.enable" = "true" instead of "binlog.enable" = "true" plus "binlog.format" = "ROW". binlog.need_historical_value became changelog.need_historical_value, and the row-binlog use of binlog.ttl_seconds became changelog.ttl_seconds. Doris now rejects binlog.format and binlog.need_historical_value in every DDL.
  • The hidden columns that the source selects and orders by are now __DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__ and __DORIS_CHANGELOG_OP__ (were __DORIS_BINLOG_*__).

The binlog() table function also became changelog(), but the connector does not use it. The @incr(...) syntax, the DETAIL / MIN_DELTA / APPEND_ONLY increment types, the op values, information_schema.tso_status and the visible-wait error code 5101 that the source retries on did not change.

The incremental source in connector 26.2.0 and 26.3.0 builds SELECT ..., __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, __DORIS_BINLOG_OP__ FROM db.t@incr(...) ORDER BY ... and looks for those column names in the Arrow result. It therefore only works with Doris master builds from before the rename. Against Doris master after the rename, that query fails on a changelog table, and the table DDL used by the integration fixtures is rejected:

-- the incremental query built by the current connector
ERROR 1105 (HY000): errCode = 2, detailMessage = Unknown column '__DORIS_BINLOG_TSO__' in 'table list' in PROJECT clause(line 1, pos 69)

-- CREATE TABLE ... ("binlog.enable" = "true", "binlog.format" = "ROW", "binlog.need_historical_value" = "true")
ERROR 1105 (HY000): errCode = 2, detailMessage = Property binlog.format is no longer supported, use "changelog.enable" = "true" to enable changelog, or remove it for CCR binlog

The same query with __DORIS_CHANGELOG_*__ returns the expected rows (op 0 / 2 / 3 / 1 for insert / update-before / update-after / delete, which is what DorisSourceRecord maps to RowKind).

What this PR changes

  1. The incremental query and the Arrow reader use the new hidden column names.
  2. Every row-binlog name in the connector is renamed to changelog, the Doris feature name: option keys, constants, the increment-type enum, DorisReadOptions fields, getters and builder setters, the DorisSourceRecord metadata getters, validation and exception messages, and javadoc. The option keys keep the existing source. prefix and their suffixes.
  3. The integration-test and E2E fixtures create their tables with "changelog.enable" = "true" and "changelog.need_historical_value" = "true".

Not changed: MySQL binlog / Flink CDC / Debezium code and test data (those are MySQL's binlog), Flink's own ChangelogMode / RowKind handling, and the split and enumerator checkpoint formats (they store split ids and timestamps only, no renamed names).

Kind Old New
Option key source.binlog.increment-type source.changelog.increment-type
Option key source.binlog.poll-interval source.changelog.poll-interval
Option key source.binlog.visible-wait-timeout source.changelog.visible-wait-timeout
Option key source.binlog.offset-table source.changelog.offset-table
Option key source.binlog.consumer-id source.changelog.consumer-id
DorisConfigOptions SOURCE_BINLOG_INCREMENT_TYPE, SOURCE_BINLOG_POLL_INTERVAL, SOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT, SOURCE_BINLOG_OFFSET_TABLE, SOURCE_BINLOG_CONSUMER_ID SOURCE_CHANGELOG_* with the same suffixes
ConfigurationOptions SOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT SOURCE_CHANGELOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT
Class org.apache.doris.flink.source.DorisBinlogIncrementType org.apache.doris.flink.source.DorisChangelogIncrementType
DorisReadOptions getBinlogIncrementType(), getBinlogPollIntervalMs(), getBinlogVisibleWaitTimeoutMs(), getBinlogOffsetTable(), getBinlogConsumerId() and the matching Builder.setBinlog*() getChangelog*() / Builder.setChangelog*()
DorisSourceRecord (@PublicEvolving) getBinlogTso(), getBinlogLsn(), getBinlogOp() getChangelogTso(), getChangelogLsn(), getChangelogOp()
Hidden columns read by the source __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, __DORIS_BINLOG_OP__ __DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__, __DORIS_CHANGELOG_OP__
Doris table properties in the IT / E2E fixtures "binlog.enable" = "true" + "binlog.format" = "ROW", "binlog.need_historical_value" "changelog.enable" = "true", "changelog.need_historical_value"

Compatibility

  • This is a hard rename with no deprecated keys. A Flink table that still sets a source.binlog.* option fails the factory's option validation as an unsupported option, and Java code that uses the renamed class or methods has to be updated (for example a custom deserialization schema that calls DorisSourceRecord.getBinlogTso()).
  • The old keys shipped in 26.2.0 (four of them; visible-wait-timeout came in 26.3.0) and 26.3.0, but they only ever worked against unreleased Doris master builds from before the rename. A connector with this change cannot read those builds either, because the hidden column names differ, and Doris does not read tables created with "binlog.format" = "ROW" after the rename: they have to be dropped and recreated with "changelog.enable" = "true". Users of the incremental source therefore move Doris, their tables and the connector together. If maintainers prefer to keep old Flink DDL parsing, the five options can take withDeprecatedKeys(...); I left that out on purpose and can add it.
  • Doris has no release with changelog yet. A connector release with this change is needed before, or together with, the first Doris release that contains it; until then no released connector can read changelog tables.

Checklist(Required)

  1. Does it affect the original behavior: Yes. The five source.binlog.* option keys and the listed class and method names are renamed, and the incremental source reads the renamed Doris hidden columns.
  2. Has unit tests been added: No Need. The existing unit tests, integration tests and the incremental E2E case are updated to the new names and pass (see below).
  3. Has document been added or modified: No. Nothing in this repository documents these options. The incremental-read page on doris.apache.org is renamed in [doc](changelog) Rename row binlog to changelog doris-website#4426, which also states 26.4.0 as the first connector version for Doris changelog.
  4. Does it need to update dependencies: No
  5. Are there any changes that cannot be rolled back: No

Further comments

Tests, all on JDK 17 (CI builds Flink 1.x with JDK 8; only JDK 17 was available locally):

  • Unit tests, mvn clean test -Pflink1 -pl flink-doris-connector-flink1 -am and mvn clean test -Pflink2 -pl flink-doris-connector-flink2 -am. Spotless check passes as part of both builds.

    Profile Module Tests run Failures Errors Skipped
    flink1 (Flink 1.20.0) base 455 0 0 1
    flink1 (Flink 1.20.0) flink1 67 1 0 1
    flink2 (Flink 2.3.0) base 455 0 0 1
    flink2 (Flink 2.3.0) flink2 20 0 0 0

    The counts are the same as on master 1050cd0 on the same machine. The one failure, DatabaseSyncTest.getJdbcUrlTemplateTest, fails identically on master and is unrelated: Db2DatabaseSync.getJdbcUrlTemplate iterates a java.util.Properties, whose iteration order changed after JDK 8, and the test expects the JDK 8 order.

  • Incremental integration tests against a single-FE / single-BE Doris master build that contains [refactor](changelog) Rename row binlog to changelog and split it from CCR binlog doris#69311. These cases only run in customer-env mode, so the CI workflows (which use the 4.1.3 Docker image) skip them:

    mvn clean test -Pflink2 -pl flink-doris-connector-it -am \
      -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
      -Dtest=DorisIncrementalSourceITCase,DorisIncrementalSourceFailoverITCase,DorisIncrementalSourceE2ECase \
      -Dcustomer_env=true -Ddoris_host=127.0.0.1 -Ddoris_query_port=<fe query port> \
      -Ddoris_http_port=<fe http port> -Ddoris_user=root -Ddoris_passwd= \
      -Ddoris_flight_sql_port=<fe arrow flight sql port>
    
    Profile DorisIncrementalSourceITCase DorisIncrementalSourceFailoverITCase (JM, TM) DorisIncrementalSourceE2ECase
    flink2 (Flink 2.3.0) 6 / 6 pass 2 / 2 pass 1 / 1 pass
    flink1 (Flink 1.20.0) 6 / 6 pass 2 / 2 pass 1 / 1 pass
  • After this change, the only remaining binlog strings in the repository are MySQL's: Debezium source metadata in test JSON (mysql_binlog_source, MySQL binlog file names), the Flink CDC MySQL BinlogOffset API in MysqlDatabaseSync, and the job name of a MySQL CDC example.

Doris renamed the unreleased row binlog feature, which the incremental
source reads, to changelog (apache/doris#69311). A table now enables it
with "changelog.enable" = "true" instead of "binlog.enable" = "true"
plus "binlog.format" = "ROW", Doris rejects the old binlog.format and
binlog.need_historical_value properties, and the hidden columns are now
__DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__ and
__DORIS_CHANGELOG_OP__. The @incr syntax, the increment types,
information_schema.tso_status and error code 5101 are unchanged. The
source still selected and ordered by the __DORIS_BINLOG_*__ columns, so
it could not read Doris master after that rename.

Query and read the new hidden columns, and rename the connector's own
row binlog names to changelog so they match the Doris feature:

- source.binlog.increment-type -> source.changelog.increment-type
- source.binlog.poll-interval -> source.changelog.poll-interval
- source.binlog.visible-wait-timeout -> source.changelog.visible-wait-timeout
- source.binlog.offset-table -> source.changelog.offset-table
- source.binlog.consumer-id -> source.changelog.consumer-id
- DorisConfigOptions.SOURCE_BINLOG_* -> SOURCE_CHANGELOG_*
- ConfigurationOptions.SOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT
  -> SOURCE_CHANGELOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT
- DorisBinlogIncrementType -> DorisChangelogIncrementType
- DorisReadOptions getBinlog*() and Builder.setBinlog*()
  -> getChangelog*() and setChangelog*()
- DorisSourceRecord getBinlogTso/Lsn/Op() -> getChangelogTso/Lsn/Op()

This is a hard rename without deprecated keys: a Flink table that still
sets a source.binlog.* option fails option validation as an unsupported
option. Those keys only ever worked against Doris master builds from
before the rename, and the connector no longer reads such builds. The
integration and end-to-end fixtures create their tables with
"changelog.enable" and "changelog.need_historical_value". MySQL binlog,
Flink CDC and Debezium code is unchanged, and so are the split and
checkpoint formats.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@morningman
morningman merged commit 4286a05 into apache:master Oct 21, 2026
10 checks passed
morningman added a commit to apache/doris-website that referenced this pull request Oct 21, 2026
## Summary

Rename the experimental row binlog to **changelog** in the English and
Chinese docs, following apache/doris#69311. Three pages move to new URLs
(old URLs redirect in every locale), every page that mentions the
feature uses the new property names, table function, hidden columns,
error messages and switch, and the changelog page now describes the
behavior #69311 introduced.

## Why

apache/doris#69311 renamed row binlog (`"binlog.enable" = "true"` +
`"binlog.format" = "ROW"`, the change stream that Table Stream and
incremental materialized views read) to changelog and split it from CCR
binlog. It is a hard cut on an unreleased feature, so the docs on master
describe names that no longer exist:

| Before | After |
|---|---|
| `"binlog.enable" = "true"` + `"binlog.format" = "ROW"` |
`"changelog.enable" = "true"` |
| `binlog.ttl_seconds` (ROW tables), `binlog.need_historical_value` |
`changelog.ttl_seconds`, `changelog.need_historical_value` |
| `binlog(...)` table function | `changelog(...)` |
| `__DORIS_BINLOG_{TSO,LSN,OP}__`, `__BEFORE__<col>__` |
`__DORIS_CHANGELOG_{TSO,LSN,OP}__`, `__DORIS_CHANGELOG_BEFORE_<col>__` |
| `enable_feature_binlog` (FE, BE) for row binlog and TSO |
`enable_feature_changelog`; `enable_feature_binlog` is CCR only |

CCR binlog, `SHOW PROC '/binlog'`, MySQL binlog CDC, Canal binlog load
and Paimon `$binlog` do not change.

## Renamed pages and redirects

| Old URL | New URL |
|---|---|
| `/docs/dev/data-operate/incremental/row-binlog` |
`/docs/dev/data-operate/incremental/changelog` |
| `/docs/dev/sql-manual/sql-functions/table-valued-functions/binlog` |
`/docs/dev/sql-manual/sql-functions/table-valued-functions/changelog` |
| `/docs/dev/key-features/binlog-table-stream` |
`/docs/dev/key-features/changelog-table-stream` |
| `/docs/4.x/key-features/binlog-table-stream` |
`/docs/4.x/key-features/changelog-table-stream` |

The same four mappings apply under `/zh-CN`, `/ja`, `/ko` and `/ru`. The
redirects are one loop over `ENABLED_LOCALES` in `buildRedirectIndex()`
(`docusaurus.config.js`), next to the DOG Stack block and in the style
of the hackathon redirects, so each optional locale gets them as soon as
it is enabled. They are not in `config/redirects-dev.json`, which lists
en and zh-CN paths only, nor in `config/redirects-4.x.json`, which
`scripts/migrate-4.x/build-redirects.py` generates. Evaluating
`createRedirects` from the site config gives 20 old paths with
`DORIS_ENABLE_LOCALES=ja,ko,ru` and 8 with the default locales.

The PR has two commits so the moves stay reviewable: the first only
moves the six files (en / zh-CN, dev / 4.x), and the second rewrites
them. The rewritten pages keep 36–41% of their text, so the combined
diff shows them as delete + add; look at the second commit to see what
changed in them.

The key-features slug is now `/key-features/changelog-table-stream`. The
card image keeps its name (`binlog-table-stream.jpg`): pages reference
it by path, nothing pairs it with the page by file name. `sidebars.ts`
points to the new doc ids; the table function moves to its alphabetical
place.

## Other changes

- The changelog page documents the new behavior: `changelog.enable` and
`changelog.need_historical_value` can only be set by `CREATE TABLE`,
`changelog.ttl_seconds` can be altered, `binlog.format` and
`binlog.need_historical_value` are rejected, CCR binlog and changelog
are mutually exclusive (table, database, `RECOVER TABLE`), temporary
tables never have changelog, a database can make changelog the default
and `SHOW CREATE TABLE` then prints `"changelog.enable" = "false"` for
tables without it, views / CTAS / MTMV over `changelog()` must list and
alias the `__DORIS_` columns, and `enable_feature_changelog` is needed
in `fe.conf` (and `be.conf` for cleanup). A short note says the feature
was called row binlog in earlier development builds and that tables
created that way must be recreated.
- Every quoted error message matches the FE on Doris master, for example
`Changelog offset has expired according to changelog.ttl_seconds`, `INCR
query requires changelog enabled on base table.`, `Base Olap table ...
need to enable changelog for table stream`, `Not allowed to perform
current operation on Table With changelog`.
- Stale content fixed:
  - `binlog.format` case sensitivity (the property no longer exists);
- the key-features pages enabled ROW with `ALTER TABLE ... SET
("binlog.format" = "ROW")`; changelog can only be enabled by `CREATE
TABLE`;
  - the OP hidden column is `BIGINT`, not `TINYINT`;
- the changelog page said backup / restore work as usual; the FE rejects
backup and restore of tables with changelog;
- the key-features pages described CCR binlog and ROW as two formats of
one binlog; changelog is a separate, per-table change stream that cannot
be combined with CCR binlog. The same pages also used `CREATE STREAM ...
ON <table>` and `"type" = "default"` (now `ON TABLE` and `min_delta`),
listed a `default` consume type (now `detail` / `append_only` /
`min_delta`), and said retention waits for streams and is reported in
`STALE_REASON` (it does neither).
- Table Stream, incremental query, IVM, CREATE TABLE / CREATE STREAM /
DROP STREAM, CREATE ASYNC MATERIALIZED VIEW, MV FAQ, TSO and FE
configuration pages use the new names. CREATE TABLE lists
`changelog.ttl_seconds`.
- Flink Doris Connector incremental reading uses the Doris-side names
and the connector's renamed options
`source.changelog.{increment-type,poll-interval,visible-wait-timeout,offset-table,consumer-id}`
(apache/doris-flink-connector#710); it says "Doris changelog" where
Flink changelog records are also discussed. The required connector
version is now 26.4.0, the release expected to carry
apache/doris-flink-connector#710: 26.3.0 and earlier read the hidden
columns under their old names, so they only work with Doris development
builds from before the rename.
- The Data Warehousing use-case page links to the new key-features URL,
and the query cache decision-flow SVG says "changelog scan".

## Not changed on purpose

- Japanese, Korean and Russian docs and `translations/`: the translation
pipeline re-translates the edited pages and handles the moved ones.
Until then their old translated files are orphaned and the moved pages
show English. The ja / ko labels for `marketing.warehouse.075` still
read "Binlog / Table Stream".
- `src/pages/community-report/` (historical weekly reports),
`scripts/last-update/`, `scripts/migrate-4.x/`.
- CCR docs, `SHOW PROC '/binlog'`, MySQL binlog, Canal and Paimon
content.
- Raster images that still show the old name:
`static/images/next/data-operate/incremental/two-layers.png` and
`static/images/next/query-acceleration/ivm-workflow.jpg` ("Row Binlog"),
and the key-features card
`static/images/next/key-features/binlog-table-stream.jpg`
(`binlog.format = ROW`, and a CCR consumer arrow). Alt texts are
updated; the pictures need to be redrawn.

## Validation

- Static checks on every changed doc: front matter, relative links, MDX
tags, no CJK in English trees, table column counts: pass.
- `scripts/check_md_links_single.py --no-external` on all added /
modified en, zh-CN and 4.x docs: pass. Anchors in the touched files were
checked separately; the only unresolved ones are five pre-existing
anchors in zh-CN `CREATE-TABLE.md` and the MV FAQ, on lines this PR does
not change.
- No en / zh-CN doc, sidebar, component or config links to the old
paths; the remaining mentions of the old names are the rename note, the
rejected-property error messages, CCR / MySQL binlog, the redirect
sources and the card image path.
- `node scripts/check_move.js HEAD`, `python3 check_move.py HEAD` and
`python3 check_move.py <merge-base>`: pass.
- `node scripts/i18n/check-configuration.cjs --all`: pass.
- `I18N_LOCALE=<ja|ko|ru> node scripts/i18n/pr-preflight.cjs --base
<merge-base>` for each of the three locales: pass, 0 errors (source
freshness for these locales is reported, not blocking).
- `node --test` for the i18n `marketing-pages`, `doc-navigation`,
`configuration`, `t04`, `inventory`, `planner`, `multi-locale`,
`locale-registry` and `heading-ids` tests: pass.
- `scripts/key-features/generate.js` produces the card with href
`/docs/dev/key-features/changelog-table-stream`.
- The edited SVG parses and renders with the new text inside its boxes.
- No site build was run.

---------

Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
@morningman
morningman deleted the changelog-rename branch October 22, 2026 10:15
@JNSimba JNSimba mentioned this pull request Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant