Repository navigation
[fix](fe) Preserve Kinesis job state when ALTER validation fails - #67696
Closed
0AyanamiRei wants to merge 1 commit into
Closed
0AyanamiRei wants to merge 1 commit into
0AyanamiRei wants to merge 1 commit into
Conversation
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
A failed `ALTER ROUTINE LOAD` can leave a Kinesis job partially modified in memory, even though the statement returns an error and no ALTER journal entry is written.
For example, pause a job whose progress contains `shard-0`, then alter its region, endpoint, or default position together with an explicit shard list containing an unknown shard. Omit `kinesis_stream` so that the existing-progress membership check applies. Before this change, `modifyPropertiesInternal()` updates the live data-source properties and replaces `customKinesisShards` before `KinesisProgress.checkShards()` rejects the unknown shard. The exception releases the job lock without restoring those fields.
The leader can therefore retain configuration that journal replay does not apply to other FEs. A residual explicit shard list can also bypass remote shard discovery after resume. There is another failure point in the same ALTER path: common-property validation for flexible partial updates can fail after Kinesis state and some common properties have already changed.
This change orders validation before mutation:
1. Read the requested Kinesis changes into local variables and check shard membership before modifying any job state. Preserve the existing progress-reset behavior when a stream is provided.
2. Validate the requested unique-key update mode and flexible partial-update constraints before assigning any common job properties in `RoutineLoadJob`.
3. Apply the common properties before updating Kinesis configuration, shard lists, and progress. The existing successful ALTER path then writes its journal entry.
The guarantee covers shard-membership and common-property validation failures. These failures now occur before any live state changes, so they require no rollback. The shared `RoutineLoadJob` adjustment is necessary to keep common-property validation from becoming another source of partial Kinesis ALTER updates.
### Release note
Kinesis Routine Load preserves its previous configuration, shard selection, and progress when ALTER fails shard-membership or common-property validation.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Failed validation leaves the Kinesis job state unchanged and does not write an ALTER journal entry.
- Does this need documentation?
- [x] No.
- [ ] Yes.
Validation:
- Before the fix, the four new failure-atomicity test cases fail because job state changes after the expected ALTER error. The other 11 Kinesis job tests pass.
- After the fix, all 47 tests pass, with no failures, errors, or skipped tests:
```bash
./run-fe-ut.sh --run org.apache.doris.load.routineload.KinesisRoutineLoadJobTest,org.apache.doris.load.routineload.RoutineLoadJobTest,org.apache.doris.load.routineload.KafkaRoutineLoadJobTest,org.apache.doris.load.routineload.kinesis.KinesisDataSourcePropertiesTest
```
- New tests cover dynamic and explicit shard selection, shared open/explicit shard list instances, unchanged state and no journal entry on failure, shard discovery after the resume state transition, common-property validation failures with and without a stream change, and successful ALTER journal serialization/replay.
- FE build via `build.sh --fe -j48` passes with `DISABLE_BUILD_UI=ON` and `DISABLE_JAVA_CHECK_STYLE=OFF`; output is directed to a separate local build directory. Checkstyle reports zero violations. UI building is disabled because the local npm dependency installation fails.
- `git diff --check` passes.
- Real AWS Kinesis consumption and multi-FE failover have not been tested. Shard discovery is mocked in the unit tests.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 16669 ms |
Contributor
TPC-DS: Total hot run time: 81908 ms |
Contributor
ClickBench: Total hot run time: 14.73 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
0AyanamiRei
marked this pull request as draft
September 15, 2026 19:08
Contributor
Author
|
draft，计划是希望对routine load kinese先去持久化的维护一个shards父子关系的拓扑图 |
Contributor
Author
|
Absorbed into the Kinesis shard topology change, closing this draft.
|
0AyanamiRei
added a commit
to 0AyanamiRei/doris
that referenced
this pull request
Sep 23, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67678, apache#67696, apache#67869, apache#67874, apache#67879 Problem Summary: Kinesis shards can split and merge while a routine load job is running. Maintaining only open/closed shard lists and progress cannot express parent-child lineage or the COMMITTED-to-VISIBLE completion barrier, which can schedule children too early, lose lineage, or recreate completed parents. This change adds a durable shard topology state machine, concrete initial-position persistence, idempotent journal/image replay, and Kinesis BE progress handling for EOF and ChildShards metadata. New FE, BE, and regression coverage verifies shard transitions, transaction visibility, latest-position resolution, reshard boundaries, and deterministic Kinesis upstream behavior. ### Release note Add parent-child shard topology tracking and visibility-gated scheduling for Kinesis Routine Load. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter='*Kinesis*' -j48` - Result: 18 Kinesis tests passed - No FE unit test or AWS regression test was run in this environment. - Behavior changed: Yes (Kinesis Routine Load now schedules shards through the topology state machine.) - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
A failed
ALTER ROUTINE LOADcan leave a Kinesis job partially modified in memory, even though the statement returns an error and no ALTER journal entry is written.For example, pause a job whose progress contains
shard-0, then alter its region, endpoint, or default position together with an explicit shard list containing an unknown shard. Omitkinesis_streamso that the existing-progress membership check applies. Before this change,modifyPropertiesInternal()updates the live data-source properties and replacescustomKinesisShardsbeforeKinesisProgress.checkShards()rejects the unknown shard. The exception releases the job lock without restoring those fields.The leader can therefore retain configuration that journal replay does not apply to other FEs. A residual explicit shard list can also bypass remote shard discovery after resume. There is another failure point in the same ALTER path: common-property validation for flexible partial updates can fail after Kinesis state and some common properties have already changed.
This change orders validation before mutation:
RoutineLoadJob.The guarantee covers shard-membership and common-property validation failures. These failures now occur before any live state changes, so they require no rollback. The shared
RoutineLoadJobadjustment is necessary to keep common-property validation from becoming another source of partial Kinesis ALTER updates.Release note
Kinesis Routine Load preserves its previous configuration, shard selection, and progress when ALTER fails shard-membership or common-property validation.
Check List (For Author)
Validation:
Before the fix, the four new failure-atomicity test cases fail because job state changes after the expected ALTER error. The other 11 Kinesis job tests pass.
After the fix, all 47 tests pass, with no failures, errors, or skipped tests:
New tests cover dynamic and explicit shard selection, shared open/explicit shard list instances, unchanged state and no journal entry on failure, shard discovery after the resume state transition, common-property validation failures with and without a stream change, and successful ALTER journal serialization/replay.
FE build via
build.sh --fe -j48passes withDISABLE_BUILD_UI=ONandDISABLE_JAVA_CHECK_STYLE=OFF; output is directed to a separate local build directory. Checkstyle reports zero violations. UI building is disabled because the local npm dependency installation fails.git diff --checkpasses.Real AWS Kinesis consumption and multi-FE failover have not been tested. Shard discovery is mocked in the unit tests.
Check List (For Reviewer who merge this PR)