feat(core): use timestamp range replace updates to update mat views#5679
Merged
bluestreak01 merged 76 commits intomasterfrom Jun 5, 2025
Merged
feat(core): use timestamp range replace updates to update mat views#5679bluestreak01 merged 76 commits intomasterfrom
bluestreak01 merged 76 commits intomasterfrom
Conversation
…place-range Conflicts: core/src/main/java/io/questdb/cairo/wal/WalEventWriter.java core/src/main/java/io/questdb/cairo/wal/WalTxnDetails.java core/src/main/java/io/questdb/cairo/wal/WalWriter.java core/src/test/java/io/questdb/test/tools/TestUtils.java
… creating non-wal table
…place-range Conflicts: core/src/main/java/io/questdb/cairo/TableWriter.java core/src/main/java/io/questdb/cairo/mv/MatViewRefreshExecutionContext.java core/src/main/java/io/questdb/cairo/mv/MatViewRefreshJob.java core/src/main/java/io/questdb/griffin/engine/QueryProgress.java core/src/test/java/io/questdb/test/cairo/fuzz/CheckpointFuzzTest.java core/src/test/java/io/questdb/test/cairo/fuzz/FuzzRunner.java core/src/test/java/io/questdb/test/cairo/fuzz/WalWriterFuzzTest.java
…place-range Conflicts: core/src/test/java/io/questdb/test/cairo/mv/MatViewTest.java
Contributor
|
@ideoma tests are failing + there is a number of merge conflicts to resolve |
bluestreak01
removed
the
DO NOT MERGE
puzpuzpuz
reviewed
Jun 4, 2025
puzpuzpuz
reviewed
Jun 4, 2025
Contributor
[PR Coverage check]😍 pass : 384 / 423 (90.78%) file detail
|
puzpuzpuz
approved these changes
Jun 5, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements a new "replace commit" mechanism for materialized view refreshes, using timestamp ranges to replace affected rows rather than relying on deduplication. Key changes include removing dedup key handling in MV creation, extending WAL commit methods to incorporate new timestamp range and deduplication mode parameters, and updating downstream consumers (e.g. query logging, O3 partition jobs, TxWriter) to support the new update scheme.
An example of a replace commit can be:
2022-02-24T04to2022-02-24T05:31MVexists with hourly aggregationsMVReresh re-calculates the changes from2022-02-24T04to2022-02-24T06MVReresh commits the refresh query output into the mat view. This new commit specifies the replace range from2022-02-24T04to2022-02-24T062022-02-24T04to2022-02-24T06with the new rows.This feature removes dedup restrictions on the base table, it can also solve a few more restrictions, like partition drop in the future.