Skip to content

fix(sql): invalidate dependent materialized views on base table rename#5693

Merged
bluestreak01 merged 48 commits intomasterfrom
puzpuzpuz_no_mat_view_invalidation_on_table_rename
Jun 2, 2025
Merged

fix(sql): invalidate dependent materialized views on base table rename#5693
bluestreak01 merged 48 commits intomasterfrom
puzpuzpuz_no_mat_view_invalidation_on_table_rename

Conversation

@puzpuzpuz
Copy link
Copy Markdown
Contributor

@puzpuzpuz puzpuzpuz commented May 23, 2025

Fixes #5602

Fixes dependent materialized views kept valid after a base table rename. The fix is moves the invalidation task being enqueued directly by the engine instead of WAL apply job like with other DDLs/DMLs. That's because dependent views are looked up by the base table name, not base table directory.

Other than that, includes the following fixes and changes:

  • Fixes "valid" being displayed instead of "refreshing" status during initial mat view refresh
  • Changes default cairo.mat.view.rows.per.query.estimate value to 1M (used to be 10M) as the old default was too high considering that the estimate is approximate
  • Adds new cairo.mat.view.timer.start.epsilon property to make sure that new mat views with immediate start timestamp are triggered as soon as they're created
  • Remove exe/fin logging for ALTER and UPDATE statements generated by WAL apply job. WAL apply job now doesn't generate progress logging for SQLs it applies
  • Include full interval string into Invalid unit error message for sample by queries and mat view DDLs (Confusing error when an incorrect sample by time unit is specified #5602). Also adds actual position to the error instead of the hard-coded -1
  • Removes redundant "base table is dropped or renamed" logging message generated by mat view refresh job. This message was printed when any base table or mat view was dropped

puzpuzpuz and others added 30 commits May 15, 2025 17:03
@puzpuzpuz puzpuzpuz self-assigned this May 23, 2025
@puzpuzpuz puzpuzpuz added Bug Incorrect or unexpected behavior SQL Issues or changes relating to SQL execution labels May 23, 2025
Base automatically changed from puzpuzpuz_timer_refreshed_mat_views to master May 23, 2025 15:57
@puzpuzpuz puzpuzpuz marked this pull request as ready for review May 23, 2025 18:22
@puzpuzpuz puzpuzpuz marked this pull request as draft May 23, 2025 18:50
@puzpuzpuz puzpuzpuz force-pushed the puzpuzpuz_no_mat_view_invalidation_on_table_rename branch from 076f424 to cfa90d7 Compare May 23, 2025 18:50
@puzpuzpuz puzpuzpuz marked this pull request as ready for review May 29, 2025 11:42
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 34 / 36 (94.44%)

file detail

path covered line new line coverage
🔵 io/questdb/cairo/DefaultCairoConfiguration.java 0 1 00.00%
🔵 io/questdb/cairo/mv/NoOpMatViewStateStore.java 0 1 00.00%
🔵 io/questdb/griffin/engine/functions/catalogue/MatViewsFunctionFactory.java 1 1 100.00%
🔵 io/questdb/PropServerConfiguration.java 3 3 100.00%
🔵 io/questdb/cairo/mv/MatViewTimerJob.java 5 5 100.00%
🔵 io/questdb/griffin/engine/functions/date/TimestampFloorFromFunctionFactory.java 1 1 100.00%
🔵 io/questdb/griffin/SqlParser.java 1 1 100.00%
🔵 io/questdb/cairo/wal/OperationExecutor.java 2 2 100.00%
🔵 io/questdb/griffin/engine/functions/date/TimestampFloorFromOffsetFunctionFactory.java 1 1 100.00%
🔵 io/questdb/cairo/mv/MatViewRefreshJob.java 1 1 100.00%
🔵 io/questdb/cairo/mv/MatViewStateStoreImpl.java 4 4 100.00%
🔵 io/questdb/cairo/CairoEngine.java 2 2 100.00%
🔵 io/questdb/PropertyKey.java 1 1 100.00%
🔵 io/questdb/std/datetime/microtime/Timestamps.java 1 1 100.00%
🔵 io/questdb/griffin/SqlCompilerImpl.java 8 8 100.00%
🔵 io/questdb/cairo/pool/SqlCompilerPool.java 1 1 100.00%
🔵 io/questdb/cairo/CairoConfigurationWrapper.java 1 1 100.00%
🔵 io/questdb/griffin/SqlOptimiser.java 1 1 100.00%

@bluestreak01
Copy link
Copy Markdown
Member

If I rename base table, view becomes "invalid", which is expected. If I rename table back to what it was, how do i get it to a valid state?

This doesn't do it:

REFRESH MATERIALIZED VIEW price_1h incremental;

@puzpuzpuz
Copy link
Copy Markdown
Contributor Author

If I rename table back to what it was, how do i get it to a valid state?

The table's contents/schema may have changed completely after it's renamed back, so you need to run a full refresh:

REFRESH MATERIALIZED VIEW price_1h FULL;

@bluestreak01 bluestreak01 merged commit aa817d4 into master Jun 2, 2025
40 checks passed
@bluestreak01 bluestreak01 deleted the puzpuzpuz_no_mat_view_invalidation_on_table_rename branch June 2, 2025 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Incorrect or unexpected behavior SQL Issues or changes relating to SQL execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Confusing error when an incorrect sample by time unit is specified

3 participants