[Feature] add row type for doris binlog #61956

Search before asking

• I had searched in the issues and found no similar issues.

Description

Doris has become a key infrastructure component for near-real-time data warehousing.
As business requirements for data freshness and pipeline flexibility continue to grow, the ability to efficiently and flexibly deliver data to downstream systems, as well as to perform incremental computation based on change data, has become an increasingly urgent core requirement.
Against this background, we plan to introduce Binlog support into the current storage-compute integrated architecture of Doris. With Binlog support, Doris will be able to provide a more standard and complete view of data changes, thereby better supporting downstream subscription, data synchronization, incremental materialized views, and stream-batch integrated computation scenarios.

Overall, the implementation of Binlog can be divided into the following core modules:

1. Meta moduleResponsible for the definition, organization, persistence, and management of Binlog-related metadata, including version information, transaction associations, lifecycle management, and maintenance of metadata required for queries.
2. Read & Write module
   Responsible for the writing, storage, and reading of Binlog data, forming the core data path of the Binlog capability. On the write side, it needs to work together with the existing loading, transaction commit, and Rowset generation flows; on the read side, it needs to provide stable and efficient change querying and consumption capabilities.
3. Compaction & Schedule module
   Responsible for background organization, merging, scheduling, and cleanup of Binlog data, ensuring that the system can maintain controllable storage costs, stable query performance, and reasonable resource efficiency over the long term.

Notice

No response

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct