Skip to content

Use proper federation in max_timestamp_df during acceleration refresh - #7055

Merged
krinart merged 21 commits into
trunkfrom
viktor/fix_acceleration_timestamp_search_2
Sep 10, 2025
Merged

krinart merged 21 commits into
trunkfrom
viktor/fix_acceleration_timestamp_search_2

Conversation

@krinart

@krinart krinart commented Sep 6, 2025 •

Copy link
Copy Markdown
Contributor

Closes #6908

The way we currently initilize dataframe in max_timestamp_df results in a full TableScan (see original ticket for more details)

In order to fix this we can downcast accelerator to PolyTableProvider, and then extract federated table source.

After doing this I ran into an issue with casting timestamp to TIMESTAMP in sqlite, which was addressed here spiceai/datafusion#99 - hence updated datafusion dependencies

@github-actions

github-actions Bot commented Sep 6, 2025 •

Copy link
Copy Markdown
Contributor

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: kind/refactor, kind/bug, kind/enhancement, kind/documentation, kind/optimization, kind/dependencies, kind/endgame
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: krinart

@krinart krinart self-assigned this Sep 8, 2025
@krinart krinart added the kind/bug Something isn't working label Sep 8, 2025
@krinart krinart changed the title [DO NOT REVIEW - DRAFT] Version #2 Use proper federation during acceleration refresh Sep 8, 2025
@krinart krinart changed the title Use proper federation during acceleration refresh Use proper federation in max_timestamp_df during acceleration refresh Sep 8, 2025
@krinart
krinart marked this pull request as ready for review September 8, 2025 23:17
@krinart
krinart requested a review from a team as a code owner September 8, 2025 23:17
@krinart
krinart marked this pull request as draft September 9, 2025 22:02
@krinart
krinart marked this pull request as ready for review September 10, 2025 00:55
sgrebnov
sgrebnov previously approved these changes Sep 10, 2025
phillipleblanc
phillipleblanc previously approved these changes Sep 10, 2025
@krinart
krinart dismissed stale reviews from phillipleblanc and sgrebnov via fe5a473 September 10, 2025 16:03
@krinart
krinart enabled auto-merge September 10, 2025 16:04
@krinart
krinart added this pull request to the merge queue Sep 10, 2025
Merged via the queue into trunk with commit f3b127e Sep 10, 2025
@krinart
krinart deleted the viktor/fix_acceleration_timestamp_search_2 branch September 10, 2025 17:32
phillipleblanc added a commit that referenced this pull request Sep 15, 2025
github-merge-queue Bot pushed a commit that referenced this pull request Sep 15, 2025
…on refresh (#7055)" (#7156)

This reverts commit f3b127e.

Co-authored-by: Luke Kim <[email protected]>
krinart added a commit that referenced this pull request Sep 19, 2025
github-merge-queue Bot pushed a commit that referenced this pull request Sep 20, 2025
…mp + integration tests (#7228)

* Revert "Revert "Use proper federation in `max_timestamp_df` during acceleration refresh (#7055)" (#7156)"

This reverts commit 068edc6.

* Add unit tests for acceleration refresh

* Add full refresh tests

* Fix lint

* Fix lint

* Fix lint

* Address comments

* Address comments

* Fix

* Fix

* Fix lint

* Fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max_timestamp_df query on accelerator table: top-k not pushed down and thus O(n)

3 participants