Skip to content

Conversation

@walterddr
Copy link
Contributor

@walterddr walterddr commented Sep 1, 2022

This is a cherry-pic of #9289

Description

With this PR, now an operator can change the underlying stream without disabling the table. The operator needs to do the followings:

  1. Issue a pause request to the table,
  2. Change the stream by modifying table's stream configs like topic name, cluster name, etc.
  3. Issue a resume request with the desired offset criteria

When a stream is changed, partitions will have new offsets. Before this PR, resume request could only resume the stream from the offsets at which the stream was paused. Since the offsets from previous stream can't be used for the new stream, the resume request should specify offset criteria for the starting point of consumption from the new stream.

Testing Done

  • Modified LLCRealtimeClusterIntegrationTest locally and created two topics (the consumption started with the first topic)
  • Issued a pause request
  • Modified the table config to point to the second topic
  • Issued a resume request with 'smallest' as offset criteria
  • Verified that the records in the table include data from both topics
  • Verified the desired values in segment ZK metadata
  • Repeated the above steps with 'largest' offset criteria
  • Repeated the above steps with no offset criteria (for this case topic wasn't changed)

@walterddr walterddr changed the title Add capabilities to ingest from another stream without disabling the … Add capabilities to ingest from another stream without disabling the realtime table (#9289) Sep 1, 2022
@walterddr walterddr added the release-notes Referenced by PRs that need attention when compiling the next release notes label Sep 1, 2022
@xiangfu0 xiangfu0 merged commit aaebcef into apache:release-0.11-rc Sep 1, 2022
@walterddr walterddr deleted the cp_9289 branch December 6, 2023 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants