Repository navigation
[Improve](source) Follow the Doris rename of row binlog to changelog - #710
Merged
Merged
Conversation
Doris renamed the unreleased row binlog feature, which the incremental source reads, to changelog (apache/doris#69311). A table now enables it with "changelog.enable" = "true" instead of "binlog.enable" = "true" plus "binlog.format" = "ROW", Doris rejects the old binlog.format and binlog.need_historical_value properties, and the hidden columns are now __DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__ and __DORIS_CHANGELOG_OP__. The @incr syntax, the increment types, information_schema.tso_status and error code 5101 are unchanged. The source still selected and ordered by the __DORIS_BINLOG_*__ columns, so it could not read Doris master after that rename. Query and read the new hidden columns, and rename the connector's own row binlog names to changelog so they match the Doris feature: - source.binlog.increment-type -> source.changelog.increment-type - source.binlog.poll-interval -> source.changelog.poll-interval - source.binlog.visible-wait-timeout -> source.changelog.visible-wait-timeout - source.binlog.offset-table -> source.changelog.offset-table - source.binlog.consumer-id -> source.changelog.consumer-id - DorisConfigOptions.SOURCE_BINLOG_* -> SOURCE_CHANGELOG_* - ConfigurationOptions.SOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT -> SOURCE_CHANGELOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULT - DorisBinlogIncrementType -> DorisChangelogIncrementType - DorisReadOptions getBinlog*() and Builder.setBinlog*() -> getChangelog*() and setChangelog*() - DorisSourceRecord getBinlogTso/Lsn/Op() -> getChangelogTso/Lsn/Op() This is a hard rename without deprecated keys: a Flink table that still sets a source.binlog.* option fails option validation as an unsupported option. Those keys only ever worked against Doris master builds from before the rename, and the connector no longer reads such builds. The integration and end-to-end fixtures create their tables with "changelog.enable" and "changelog.need_historical_value". MySQL binlog, Flink CDC and Debezium code is unchanged, and so are the split and checkpoint formats. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
morningman
added a commit
to apache/doris-website
that referenced
this pull request
Oct 21, 2026
## Summary Rename the experimental row binlog to **changelog** in the English and Chinese docs, following apache/doris#69311. Three pages move to new URLs (old URLs redirect in every locale), every page that mentions the feature uses the new property names, table function, hidden columns, error messages and switch, and the changelog page now describes the behavior #69311 introduced. ## Why apache/doris#69311 renamed row binlog (`"binlog.enable" = "true"` + `"binlog.format" = "ROW"`, the change stream that Table Stream and incremental materialized views read) to changelog and split it from CCR binlog. It is a hard cut on an unreleased feature, so the docs on master describe names that no longer exist: | Before | After | |---|---| | `"binlog.enable" = "true"` + `"binlog.format" = "ROW"` | `"changelog.enable" = "true"` | | `binlog.ttl_seconds` (ROW tables), `binlog.need_historical_value` | `changelog.ttl_seconds`, `changelog.need_historical_value` | | `binlog(...)` table function | `changelog(...)` | | `__DORIS_BINLOG_{TSO,LSN,OP}__`, `__BEFORE__<col>__` | `__DORIS_CHANGELOG_{TSO,LSN,OP}__`, `__DORIS_CHANGELOG_BEFORE_<col>__` | | `enable_feature_binlog` (FE, BE) for row binlog and TSO | `enable_feature_changelog`; `enable_feature_binlog` is CCR only | CCR binlog, `SHOW PROC '/binlog'`, MySQL binlog CDC, Canal binlog load and Paimon `$binlog` do not change. ## Renamed pages and redirects | Old URL | New URL | |---|---| | `/docs/dev/data-operate/incremental/row-binlog` | `/docs/dev/data-operate/incremental/changelog` | | `/docs/dev/sql-manual/sql-functions/table-valued-functions/binlog` | `/docs/dev/sql-manual/sql-functions/table-valued-functions/changelog` | | `/docs/dev/key-features/binlog-table-stream` | `/docs/dev/key-features/changelog-table-stream` | | `/docs/4.x/key-features/binlog-table-stream` | `/docs/4.x/key-features/changelog-table-stream` | The same four mappings apply under `/zh-CN`, `/ja`, `/ko` and `/ru`. The redirects are one loop over `ENABLED_LOCALES` in `buildRedirectIndex()` (`docusaurus.config.js`), next to the DOG Stack block and in the style of the hackathon redirects, so each optional locale gets them as soon as it is enabled. They are not in `config/redirects-dev.json`, which lists en and zh-CN paths only, nor in `config/redirects-4.x.json`, which `scripts/migrate-4.x/build-redirects.py` generates. Evaluating `createRedirects` from the site config gives 20 old paths with `DORIS_ENABLE_LOCALES=ja,ko,ru` and 8 with the default locales. The PR has two commits so the moves stay reviewable: the first only moves the six files (en / zh-CN, dev / 4.x), and the second rewrites them. The rewritten pages keep 36–41% of their text, so the combined diff shows them as delete + add; look at the second commit to see what changed in them. The key-features slug is now `/key-features/changelog-table-stream`. The card image keeps its name (`binlog-table-stream.jpg`): pages reference it by path, nothing pairs it with the page by file name. `sidebars.ts` points to the new doc ids; the table function moves to its alphabetical place. ## Other changes - The changelog page documents the new behavior: `changelog.enable` and `changelog.need_historical_value` can only be set by `CREATE TABLE`, `changelog.ttl_seconds` can be altered, `binlog.format` and `binlog.need_historical_value` are rejected, CCR binlog and changelog are mutually exclusive (table, database, `RECOVER TABLE`), temporary tables never have changelog, a database can make changelog the default and `SHOW CREATE TABLE` then prints `"changelog.enable" = "false"` for tables without it, views / CTAS / MTMV over `changelog()` must list and alias the `__DORIS_` columns, and `enable_feature_changelog` is needed in `fe.conf` (and `be.conf` for cleanup). A short note says the feature was called row binlog in earlier development builds and that tables created that way must be recreated. - Every quoted error message matches the FE on Doris master, for example `Changelog offset has expired according to changelog.ttl_seconds`, `INCR query requires changelog enabled on base table.`, `Base Olap table ... need to enable changelog for table stream`, `Not allowed to perform current operation on Table With changelog`. - Stale content fixed: - `binlog.format` case sensitivity (the property no longer exists); - the key-features pages enabled ROW with `ALTER TABLE ... SET ("binlog.format" = "ROW")`; changelog can only be enabled by `CREATE TABLE`; - the OP hidden column is `BIGINT`, not `TINYINT`; - the changelog page said backup / restore work as usual; the FE rejects backup and restore of tables with changelog; - the key-features pages described CCR binlog and ROW as two formats of one binlog; changelog is a separate, per-table change stream that cannot be combined with CCR binlog. The same pages also used `CREATE STREAM ... ON <table>` and `"type" = "default"` (now `ON TABLE` and `min_delta`), listed a `default` consume type (now `detail` / `append_only` / `min_delta`), and said retention waits for streams and is reported in `STALE_REASON` (it does neither). - Table Stream, incremental query, IVM, CREATE TABLE / CREATE STREAM / DROP STREAM, CREATE ASYNC MATERIALIZED VIEW, MV FAQ, TSO and FE configuration pages use the new names. CREATE TABLE lists `changelog.ttl_seconds`. - Flink Doris Connector incremental reading uses the Doris-side names and the connector's renamed options `source.changelog.{increment-type,poll-interval,visible-wait-timeout,offset-table,consumer-id}` (apache/doris-flink-connector#710); it says "Doris changelog" where Flink changelog records are also discussed. The required connector version is now 26.4.0, the release expected to carry apache/doris-flink-connector#710: 26.3.0 and earlier read the hidden columns under their old names, so they only work with Doris development builds from before the rename. - The Data Warehousing use-case page links to the new key-features URL, and the query cache decision-flow SVG says "changelog scan". ## Not changed on purpose - Japanese, Korean and Russian docs and `translations/`: the translation pipeline re-translates the edited pages and handles the moved ones. Until then their old translated files are orphaned and the moved pages show English. The ja / ko labels for `marketing.warehouse.075` still read "Binlog / Table Stream". - `src/pages/community-report/` (historical weekly reports), `scripts/last-update/`, `scripts/migrate-4.x/`. - CCR docs, `SHOW PROC '/binlog'`, MySQL binlog, Canal and Paimon content. - Raster images that still show the old name: `static/images/next/data-operate/incremental/two-layers.png` and `static/images/next/query-acceleration/ivm-workflow.jpg` ("Row Binlog"), and the key-features card `static/images/next/key-features/binlog-table-stream.jpg` (`binlog.format = ROW`, and a CCR consumer arrow). Alt texts are updated; the pictures need to be redrawn. ## Validation - Static checks on every changed doc: front matter, relative links, MDX tags, no CJK in English trees, table column counts: pass. - `scripts/check_md_links_single.py --no-external` on all added / modified en, zh-CN and 4.x docs: pass. Anchors in the touched files were checked separately; the only unresolved ones are five pre-existing anchors in zh-CN `CREATE-TABLE.md` and the MV FAQ, on lines this PR does not change. - No en / zh-CN doc, sidebar, component or config links to the old paths; the remaining mentions of the old names are the rename note, the rejected-property error messages, CCR / MySQL binlog, the redirect sources and the card image path. - `node scripts/check_move.js HEAD`, `python3 check_move.py HEAD` and `python3 check_move.py <merge-base>`: pass. - `node scripts/i18n/check-configuration.cjs --all`: pass. - `I18N_LOCALE=<ja|ko|ru> node scripts/i18n/pr-preflight.cjs --base <merge-base>` for each of the three locales: pass, 0 errors (source freshness for these locales is reported, not blocking). - `node --test` for the i18n `marketing-pages`, `doc-navigation`, `configuration`, `t04`, `inventory`, `planner`, `multi-locale`, `locale-registry` and `heading-ids` tests: pass. - `scripts/key-features/generate.js` produces the card with href `/docs/dev/key-features/changelog-table-stream`. - The edited SVG parses and renders with the new text inside its boxes. - No site build was run. --------- Co-authored-by: morningman <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: None (follows apache/doris#69311)
Problem Summary:
Doris renamed the experimental row binlog feature to changelog in apache/doris#69311 (merged into Doris master as
73a987e6059). The incremental source added in #679 reads that feature, and two kinds of names it depends on changed:"changelog.enable" = "true"instead of"binlog.enable" = "true"plus"binlog.format" = "ROW".binlog.need_historical_valuebecamechangelog.need_historical_value, and the row-binlog use ofbinlog.ttl_secondsbecamechangelog.ttl_seconds. Doris now rejectsbinlog.formatandbinlog.need_historical_valuein every DDL.__DORIS_CHANGELOG_TSO__,__DORIS_CHANGELOG_LSN__and__DORIS_CHANGELOG_OP__(were__DORIS_BINLOG_*__).The
binlog()table function also becamechangelog(), but the connector does not use it. The@incr(...)syntax, theDETAIL/MIN_DELTA/APPEND_ONLYincrement types, the op values,information_schema.tso_statusand the visible-wait error code5101that the source retries on did not change.The incremental source in connector 26.2.0 and 26.3.0 builds
SELECT ..., __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, __DORIS_BINLOG_OP__ FROM db.t@incr(...) ORDER BY ...and looks for those column names in the Arrow result. It therefore only works with Doris master builds from before the rename. Against Doris master after the rename, that query fails on a changelog table, and the table DDL used by the integration fixtures is rejected:The same query with
__DORIS_CHANGELOG_*__returns the expected rows (op 0 / 2 / 3 / 1 for insert / update-before / update-after / delete, which is whatDorisSourceRecordmaps toRowKind).What this PR changes
DorisReadOptionsfields, getters and builder setters, theDorisSourceRecordmetadata getters, validation and exception messages, and javadoc. The option keys keep the existingsource.prefix and their suffixes."changelog.enable" = "true"and"changelog.need_historical_value" = "true".Not changed: MySQL binlog / Flink CDC / Debezium code and test data (those are MySQL's binlog), Flink's own
ChangelogMode/RowKindhandling, and the split and enumerator checkpoint formats (they store split ids and timestamps only, no renamed names).source.binlog.increment-typesource.changelog.increment-typesource.binlog.poll-intervalsource.changelog.poll-intervalsource.binlog.visible-wait-timeoutsource.changelog.visible-wait-timeoutsource.binlog.offset-tablesource.changelog.offset-tablesource.binlog.consumer-idsource.changelog.consumer-idDorisConfigOptionsSOURCE_BINLOG_INCREMENT_TYPE,SOURCE_BINLOG_POLL_INTERVAL,SOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT,SOURCE_BINLOG_OFFSET_TABLE,SOURCE_BINLOG_CONSUMER_IDSOURCE_CHANGELOG_*with the same suffixesConfigurationOptionsSOURCE_BINLOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULTSOURCE_CHANGELOG_VISIBLE_WAIT_TIMEOUT_MS_DEFAULTorg.apache.doris.flink.source.DorisBinlogIncrementTypeorg.apache.doris.flink.source.DorisChangelogIncrementTypeDorisReadOptionsgetBinlogIncrementType(),getBinlogPollIntervalMs(),getBinlogVisibleWaitTimeoutMs(),getBinlogOffsetTable(),getBinlogConsumerId()and the matchingBuilder.setBinlog*()getChangelog*()/Builder.setChangelog*()DorisSourceRecord(@PublicEvolving)getBinlogTso(),getBinlogLsn(),getBinlogOp()getChangelogTso(),getChangelogLsn(),getChangelogOp()__DORIS_BINLOG_TSO__,__DORIS_BINLOG_LSN__,__DORIS_BINLOG_OP____DORIS_CHANGELOG_TSO__,__DORIS_CHANGELOG_LSN__,__DORIS_CHANGELOG_OP__"binlog.enable" = "true"+"binlog.format" = "ROW","binlog.need_historical_value""changelog.enable" = "true","changelog.need_historical_value"Compatibility
source.binlog.*option fails the factory's option validation as an unsupported option, and Java code that uses the renamed class or methods has to be updated (for example a custom deserialization schema that callsDorisSourceRecord.getBinlogTso()).visible-wait-timeoutcame in 26.3.0) and 26.3.0, but they only ever worked against unreleased Doris master builds from before the rename. A connector with this change cannot read those builds either, because the hidden column names differ, and Doris does not read tables created with"binlog.format" = "ROW"after the rename: they have to be dropped and recreated with"changelog.enable" = "true". Users of the incremental source therefore move Doris, their tables and the connector together. If maintainers prefer to keep old Flink DDL parsing, the five options can takewithDeprecatedKeys(...); I left that out on purpose and can add it.Checklist(Required)
source.binlog.*option keys and the listed class and method names are renamed, and the incremental source reads the renamed Doris hidden columns.Further comments
Tests, all on JDK 17 (CI builds Flink 1.x with JDK 8; only JDK 17 was available locally):
Unit tests,
mvn clean test -Pflink1 -pl flink-doris-connector-flink1 -amandmvn clean test -Pflink2 -pl flink-doris-connector-flink2 -am. Spotless check passes as part of both builds.The counts are the same as on master
1050cd0on the same machine. The one failure,DatabaseSyncTest.getJdbcUrlTemplateTest, fails identically on master and is unrelated:Db2DatabaseSync.getJdbcUrlTemplateiterates ajava.util.Properties, whose iteration order changed after JDK 8, and the test expects the JDK 8 order.Incremental integration tests against a single-FE / single-BE Doris master build that contains [refactor](changelog) Rename row binlog to changelog and split it from CCR binlog doris#69311. These cases only run in customer-env mode, so the CI workflows (which use the 4.1.3 Docker image) skip them:
DorisIncrementalSourceITCaseDorisIncrementalSourceFailoverITCase(JM, TM)DorisIncrementalSourceE2ECaseAfter this change, the only remaining
binlogstrings in the repository are MySQL's: Debezium source metadata in test JSON (mysql_binlog_source, MySQL binlog file names), the Flink CDC MySQLBinlogOffsetAPI inMysqlDatabaseSync, and the job name of a MySQL CDC example.