Skip to content

branch-4.1: [fix](fe) Persist routine load job before scheduling #67681- #68560 - #68773

Merged
yiguolei merged 3 commits into
apache:branch-4.1from
0AyanamiRei:pick-67681-branch-4.1
Oct 13, 2026
Merged

yiguolei merged 3 commits into
apache:branch-4.1from
0AyanamiRei:pick-67681-branch-4.1

Conversation

@0AyanamiRei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #67681, #68560

Problem Summary:

Manual backport of #67681 to branch-4.1, replacing the closed automatic backport #68560.

Persist the routine load create journal before publishing the job to the scheduler. This keeps the initial NEED_SCHEDULE state stable during serialization and prevents replay from restoring a RUNNING job without tasks.

Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes in their existing routineload package and remove unsupported Kinesis references. The Kafka test retains assertions for journal-before-publication ordering and successful scheduling after publication. This addresses the test compilation failure in TeamCity build 1062738.

Release note

Fix routine load jobs potentially becoming stuck after replay due to scheduling before the create journal is persisted.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test
    • Unit test adapted for branch-4.1. Local compilation and tests were not run, as requested. git diff --check passed.
  • Behavior changed:

    • No.
    • Yes. The create journal is persisted before the job becomes visible to the scheduler.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

0AyanamiRei and others added 2 commits October 8, 2026 14:33
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Routine load creation published a mutable job to the scheduler before
serializing its create journal. The scheduler could change the job from
`NEED_SCHEDULE` to `RUNNING` before serialization, while runtime task
membership was not persisted. Replaying such a create record restored a
`RUNNING` job with no tasks. With stable source topology, the job could
remain stuck without consuming data.

This PR makes the create journal durable before publishing the job to
the scheduler. The scheduler cannot observe or mutate the job while its
initial `NEED_SCHEDULE` state is being serialized.

Before:

```text
publish job -> scheduler may change it to RUNNING -> serialize create journal
```

After:

```text
serialize create journal in NEED_SCHEDULE -> publish job -> scheduler creates tasks
```

The change does not modify `RoutineLoadJob.read()` or normalize states
during replay. There are no deployed historical Kinesis create records
that require compatibility recovery, and existing Kafka replay behavior
remains unchanged.

#### Separate cloud Kafka follow-up found during review

While evaluating a broader replay-time `RUNNING` to `NEED_SCHEDULE`
conversion, we found an independent ordering issue in cloud Kafka
automatic-partition recovery. `isKafkaPartitionsChanged()` can update
`currentKafkaPartitions` before `updateCloudProgress()` successfully
fetches the authoritative offset from Meta Service. If that fetch fails,
a later scheduler round can see stable topology and skip the Meta
Service retry. A recovery path that makes the job schedulable at that
point could create a task from stale FE metadata, repeat already
committed data, and regress stored routine-load progress.

This PR does not add that replay-time conversion and therefore does not
change or expose the cloud Kafka path. The cloud issue will be handled
separately by making authoritative Meta Service progress synchronization
a prerequisite for task creation and preserving the retry requirement
across scheduler rounds.

(cherry picked from commit 8310bc0)
@0AyanamiRei
0AyanamiRei requested a review from yiguolei as a code owner October 8, 2026 06:34
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei 0AyanamiRei changed the title [branch-4.1](fix) Persist routine load job before scheduling (#67681) branch-4.1: [fix](fe) Persist routine load job before scheduling #67681- #68560 Oct 8, 2026
@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

yiguolei
yiguolei previously approved these changes Oct 9, 2026
@yiguolei

yiguolei commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68773

Problem Summary: The branch-4.1 FE test compiler targets Java 8, but
RoutineLoadJobPersistenceTest uses Map.of, which was introduced in Java 9.
Both COMPILE and FE UT fail at testCompile with cannot find symbol for
Map.of(int, long). Use Collections.singletonMap to preserve the same
partition offset fixture with a Java 8 compatible API. The persistence
ordering and scheduling assertions remain unchanged.

### Release note

None

### Check List (For Author)

- Test: Unit Test / Manual test
    - CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt ./run-fe-ut.sh --run org.apache.doris.load.routineload.RoutineLoadJobPersistenceTest: exit 0; 1 test, 0 failures, 0 errors
    - cd fe && mvn checkstyle:check -pl fe-core: exit 0; 0 violations
    - javac --release 8 API comparison: Map.of fails; Collections.singletonMap compiles
    - git diff --check: passed
- Behavior changed: No
- Does this need documentation: No
@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run cloudp0

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run cloud_p0

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 1.18% (1/85) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit d91f703 into apache:branch-4.1 Oct 13, 2026
28 of 29 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Oct 15, 2026
…- #68560 (#68773)

### What problem does this PR solve?

Issue Number: None

Related PR: #67681, #68560

Problem Summary:

Manual backport of #67681 to branch-4.1, replacing the closed automatic
backport #68560.

Persist the routine load create journal before publishing the job to the
scheduler. This keeps the initial NEED_SCHEDULE state stable during
serialization and prevents replay from restoring a RUNNING job without
tasks.

Adapt RoutineLoadJobPersistenceTest to branch-4.1: use the Kafka classes
in their existing routineload package and remove unsupported Kinesis
references. The Kafka test retains assertions for
journal-before-publication ordering and successful scheduling after
publication. This addresses the test compilation failure in TeamCity
build 1062738.

### Release note

Fix routine load jobs potentially becoming stuck after replay due to
scheduling before the create journal is persisted.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test
- Unit test adapted for branch-4.1. Local compilation and tests were not
run, as requested. `git diff --check` passed.

- Behavior changed:
    - [ ] No.
- [x] Yes. The create journal is persisted before the job becomes
visible to the scheduler.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants