Skip to content

[To dev/1.3] Pipe: Try to persist progressIndex in local for resend event more efficently (#15599) #15669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Caideyipi merged 1 commit into from
Jun 7, 2025
Merged

[To dev/1.3] Pipe: Try to persist progressIndex in local for resend event more efficently (#15599) #15669

Caideyipi merged 1 commit into from
Jun 7, 2025

Conversation

@Caideyipi
Copy link
Collaborator

@Caideyipi Caideyipi commented Jun 7, 2025

Description

As the title said.
#15599

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.
Key changed/added classes (or packages if there are too many classes) in this PR

@Pengzna @Caideyipi
Pipe/IoTV2: Try to persist progressIndex in local for resend event mo…
cc5a483 
…re efficently (apache#15599)

* feat: progressIndex persistent in local process

* fix ci and support cancel persist task

* pipeMeta in cn doesn't need to persist progressIndex

* remove duplicate

* clean progressIndex background service

* fix and add conf

* Update PipeConfig.java

* Update PipeMetaDeSerTest.java

* cp-gap

* Update PipeTaskMeta.java

* partial

* Update PipeRuntimeMeta.java

* Update PipeRuntimeMeta.java

* Partial

* Update PipeTaskMeta.java

* Added config

* clean

* Change default

* Update PipeConnectorConstant.java

* Partial

* Update PipeTaskMeta.java

* Update PipeTaskMeta.java

---------

Co-authored-by: Caideyipi <[email protected]>
@Caideyipi Caideyipi changed the title Pipe: Try to persist progressIndex in local for resend event more efficently (#15599) [To dev/1.3] Pipe: Try to persist progressIndex in local for resend event more efficently (#15599) Jun 7, 2025
@Caideyipi Caideyipi merged commit 59974f4 into apache:dev/1.3 Jun 7, 2025
25 checks passed
@Caideyipi Caideyipi deleted the fix-progress-13 branch June 7, 2025 08:28
Caideyipi added a commit to Caideyipi/iotdb that referenced this pull request Jul 12, 2025
@Caideyipi
Revert "[To dev/1.3] Pipe/IoTV2: Try to persist progressIndex in loca…
234df24 
…l for resend event more efficently (apache#15599) (apache#15669)"

This reverts commit 59974f4.
Caideyipi added a commit that referenced this pull request Jul 12, 2025
@Caideyipi
[To dev/1.3] Revert Pipe: Try to persist progressIndex in local for r…
ae14450 
…esend event more efficently (#15599) (#15926)

* Revert "[To dev/1.3] Pipe/IoTV2: Try to persist progressIndex in local for resend event more efficently (#15599) (#15669)"

This reverts commit 59974f4.

* Update DataNodeShutdownHook.java

* continue revert

* fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Caideyipi @Pengzna