Repository navigation
branch-4.1: [fix](fe) Persist routine load job before scheduling #67681 - #68560
Closed
github-actions[bot] wants to merge 1 commit into
Closed
github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
|
run feut |
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Routine load creation published a mutable job to the scheduler before serializing its create journal. The scheduler could change the job from `NEED_SCHEDULE` to `RUNNING` before serialization, while runtime task membership was not persisted. Replaying such a create record restored a `RUNNING` job with no tasks. With stable source topology, the job could remain stuck without consuming data. This PR makes the create journal durable before publishing the job to the scheduler. The scheduler cannot observe or mutate the job while its initial `NEED_SCHEDULE` state is being serialized. Before: ```text publish job -> scheduler may change it to RUNNING -> serialize create journal ``` After: ```text serialize create journal in NEED_SCHEDULE -> publish job -> scheduler creates tasks ``` The change does not modify `RoutineLoadJob.read()` or normalize states during replay. There are no deployed historical Kinesis create records that require compatibility recovery, and existing Kafka replay behavior remains unchanged. #### Separate cloud Kafka follow-up found during review While evaluating a broader replay-time `RUNNING` to `NEED_SCHEDULE` conversion, we found an independent ordering issue in cloud Kafka automatic-partition recovery. `isKafkaPartitionsChanged()` can update `currentKafkaPartitions` before `updateCloudProgress()` successfully fetches the authoritative offset from Meta Service. If that fetch fails, a later scheduler round can see stable topology and skip the Meta Service retry. A recovery path that makes the job schedulable at that point could create a task from stale FE metadata, repeat already committed data, and regress stored routine-load progress. This PR does not add that replay-time conversion and therefore does not change or expose the cloud Kafka path. The cloud issue will be handled separately by making authoritative Meta Service progress synchronization a prerequisite for task creation and preserving the retry requirement across scheduler rounds.
yiguolei
force-pushed
the
auto-pick-67681-branch-4.1
branch
from
October 8, 2026 04:51
99a4299 to
8310bc0
Compare
Contributor
|
run buildall |
3 of 11 tasks
Contributor
yiguolei
pushed a commit
that referenced
this pull request
Oct 13, 2026
…- #68560 (#68773) ### What problem does this PR solve? Issue Number: None Related PR: #67681, #68560 Problem Summary: Manual backport of #67681 to branch-4.1, replacing the closed automatic backport #68560. Persist the routine load create journal before publishing the job to the scheduler. This keeps the initial NEED_SCHEDULE state stable during serialization and prevents replay from restoring a RUNNING job without tasks. Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes in their existing routineload package and remove unsupported Kinesis references. The Kafka test retains assertions for journal-before-publication ordering and successful scheduling after publication. This addresses the test compilation failure in TeamCity build 1062738. ### Release note Fix routine load jobs potentially becoming stuck after replay due to scheduling before the create journal is persisted. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test - Unit test adapted for branch-4.1. Local compilation and tests were not run, as requested. `git diff --check` passed. - Behavior changed: - [ ] No. - [x] Yes. The create journal is persisted before the job becomes visible to the scheduler. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
yiguolei
pushed a commit
that referenced
this pull request
Oct 15, 2026
…- #68560 (#68773) ### What problem does this PR solve? Issue Number: None Related PR: #67681, #68560 Problem Summary: Manual backport of #67681 to branch-4.1, replacing the closed automatic backport #68560. Persist the routine load create journal before publishing the job to the scheduler. This keeps the initial NEED_SCHEDULE state stable during serialization and prevents replay from restoring a RUNNING job without tasks. Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes in their existing routineload package and remove unsupported Kinesis references. The Kafka test retains assertions for journal-before-publication ordering and successful scheduling after publication. This addresses the test compilation failure in TeamCity build 1062738. ### Release note Fix routine load jobs potentially becoming stuck after replay due to scheduling before the create journal is persisted. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test - Unit test adapted for branch-4.1. Local compilation and tests were not run, as requested. `git diff --check` passed. - Behavior changed: - [ ] No. - [x] Yes. The create journal is persisted before the job becomes visible to the scheduler. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67681