Repository navigation
branch-4.1: [fix](fe) Persist routine load job before scheduling #67681- #68560 - #68773
Merged
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Routine load creation published a mutable job to the scheduler before serializing its create journal. The scheduler could change the job from `NEED_SCHEDULE` to `RUNNING` before serialization, while runtime task membership was not persisted. Replaying such a create record restored a `RUNNING` job with no tasks. With stable source topology, the job could remain stuck without consuming data. This PR makes the create journal durable before publishing the job to the scheduler. The scheduler cannot observe or mutate the job while its initial `NEED_SCHEDULE` state is being serialized. Before: ```text publish job -> scheduler may change it to RUNNING -> serialize create journal ``` After: ```text serialize create journal in NEED_SCHEDULE -> publish job -> scheduler creates tasks ``` The change does not modify `RoutineLoadJob.read()` or normalize states during replay. There are no deployed historical Kinesis create records that require compatibility recovery, and existing Kafka replay behavior remains unchanged. #### Separate cloud Kafka follow-up found during review While evaluating a broader replay-time `RUNNING` to `NEED_SCHEDULE` conversion, we found an independent ordering issue in cloud Kafka automatic-partition recovery. `isKafkaPartitionsChanged()` can update `currentKafkaPartitions` before `updateCloudProgress()` successfully fetches the authoritative offset from Meta Service. If that fetch fails, a later scheduler round can see stable topology and skip the Meta Service retry. A recovery path that makes the job schedulable at that point could create a task from stale FE metadata, repeat already committed data, and regress stored routine-load progress. This PR does not add that replay-time conversion and therefore does not change or expose the cloud Kafka path. The cloud issue will be handled separately by making authoritative Meta Service progress synchronization a prerequisite for task creation and preserving the retry requirement across scheduler rounds. (cherry picked from commit 8310bc0)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
yiguolei
previously approved these changes
Oct 9, 2026
Contributor
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#68773 Problem Summary: The branch-4.1 FE test compiler targets Java 8, but RoutineLoadJobPersistenceTest uses Map.of, which was introduced in Java 9. Both COMPILE and FE UT fail at testCompile with cannot find symbol for Map.of(int, long). Use Collections.singletonMap to preserve the same partition offset fixture with a Java 8 compatible API. The persistence ordering and scheduling assertions remain unchanged. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt ./run-fe-ut.sh --run org.apache.doris.load.routineload.RoutineLoadJobPersistenceTest: exit 0; 1 test, 0 failures, 0 errors - cd fe && mvn checkstyle:check -pl fe-core: exit 0; 0 violations - javac --release 8 API comparison: Map.of fails; Collections.singletonMap compiles - git diff --check: passed - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Author
|
run cloudp0 |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run cloud_p0 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Oct 13, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
yiguolei
pushed a commit
that referenced
this pull request
Oct 15, 2026
…- #68560 (#68773) ### What problem does this PR solve? Issue Number: None Related PR: #67681, #68560 Problem Summary: Manual backport of #67681 to branch-4.1, replacing the closed automatic backport #68560. Persist the routine load create journal before publishing the job to the scheduler. This keeps the initial NEED_SCHEDULE state stable during serialization and prevents replay from restoring a RUNNING job without tasks. Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes in their existing routineload package and remove unsupported Kinesis references. The Kafka test retains assertions for journal-before-publication ordering and successful scheduling after publication. This addresses the test compilation failure in TeamCity build 1062738. ### Release note Fix routine load jobs potentially becoming stuck after replay due to scheduling before the create journal is persisted. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test - Unit test adapted for branch-4.1. Local compilation and tests were not run, as requested. `git diff --check` passed. - Behavior changed: - [ ] No. - [x] Yes. The create journal is persisted before the job becomes visible to the scheduler. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #67681, #68560
Problem Summary:
Manual backport of #67681 to branch-4.1, replacing the closed automatic backport #68560.
Persist the routine load create journal before publishing the job to the scheduler. This keeps the initial NEED_SCHEDULE state stable during serialization and prevents replay from restoring a RUNNING job without tasks.
Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes in their existing routineload package and remove unsupported Kinesis references. The Kafka test retains assertions for journal-before-publication ordering and successful scheduling after publication. This addresses the test compilation failure in TeamCity build 1062738.
Release note
Fix routine load jobs potentially becoming stuck after replay due to scheduling before the create journal is persisted.
Check List (For Author)
Test
git diff --checkpassed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)