You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit be49e67
Browse filesBrowse the repository at this point in the historyBrowse files
[improve](streaming-job) async chunk splitting for cdc source job (apache#63079)
## Summary
`StreamingInsertJob` (CDC FROM-TO and `cdc_stream` TVF paths) used to
call `splitChunks()` synchronously inside `CREATE STREAMING JOB`, asking
cdc_client to cut every chunk of every table before returning. On
large/non-uniform PK tables this can take 30+ minutes — far beyond the
BE→cdc_client BRPC 60s timeout, and the SQL client blocks the whole
time.
This PR makes splitting tick-driven by the FE scheduler:
- `CREATE` returns immediately; no more synchronous `splitChunks()`.
- Each scheduler tick `advanceSplits()` issues one short fetchSplits RPC
(default `batchSize=100`) and pushes that batch into `remainingSplits`.
Tasks dispatch as soon as the first batch lands, so end-to-end
first-byte latency stays close to flink-cdc's.
- cdc_client is **stateless** — every RPC reconstructs `ChunkSplitter`
from the `(currentSplittingTable, nextSplitStart, nextSplitId)` triple
supplied by FE; flink-cdc internals are untouched (uses the public
`ChunkSplitter` API only).
- Crash recovery uses three sources of truth:
- editlog persists `committedSplitProgress` (3-field `SplitProgress`) +
existing `chunkHighWatermarkMap` / `binlogOffsetPersist`
- `streaming_job_meta` system table holds full `chunk_list` JSON per
table (UPSERT each `advanceSplits`)
- cdc_client memory holds nothing
- Both FROM-TO (multi-table) and TVF (single-table) paths share the same
`SourceOffsetProvider#initSplitProgress` / `noMoreSplits` /
`advanceSplits` interface;
`StreamingJobSchedulerTask.handlePendingState` pre-advances one batch so
the first task doesn't wait a full `max_interval`.
0 commit comments