Add algorithms folder to docker - #13

commit 9c7a25ed95b8a575bea4318a09d037d102f67649
Author: Luke Kim <[email protected]>
Date:   Fri Aug 27 11:56:05 2021 +0900

    Restructure CLI, remove "pod" command, properly do context, remove model downloader (#318)

commit 5a498cd8c1898301af314c806d186475a5aac926
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 26 20:00:11 2021 +0900

    Adding more E2E tests (#317)

    * Add cleanup for temp directories

    * Use common tempdir package

    * fix linting issues

    * Add to e2e test in GitHub

    * Rework e2e test as a normal go test

    * Require flag to run e2e test

    * Add observation test and output test

commit a7f335deb0af2275893951f9f228f96d17e52227
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 26 10:37:43 2021 +0900

    Add Import/Export functionality for trained models (#314)

    New CLI commands:
    `spice export trader -o ./trader.spicepod`
    will export the currently trained model into a .spicepod archive.
    `spice import ./trader.spicepod`
    will import the spice pod archive from file

commit 224a25ddf58cbba47e9ba51667423b1841b69cee
Merge: a40222a 3df6304
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 26 09:43:00 2021 +0900

    Merge pull request #316 from spiceai/spicerack-changes

    Bring over spicerack changes

commit 3df6304f41b64b46186899ac2e3f0e4a0fd758ab
Author: Luke Kim <[email protected]>
Date:   Wed Aug 25 20:24:45 2021 +0900

    Update URL to api.spicerack.org (#63)

commit cac448964db209f3120407eb2cd830b56f8bd4b4
Author: Luke Kim <[email protected]>
Date:   Wed Aug 25 19:54:09 2021 +0900

    Swap to using spicerack.org as registry (#61)

commit a40222ae34e604d84d3cd9e26aeda9bd5d2a50d1
Merge: ea919fa 5608590
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Aug 24 12:17:28 2021 +0900

    Merge pull request #313 from spiceai/merge-spiceai.3

    Merge spiceai 0.1.0-alpha.3

commit 5608590699394dc99acb33962417c7e53704aa19
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Aug 24 12:00:31 2021 +0900

    fix merge marker

commit 31d1ec08d29edbfc9193b884936692fe24778b2e
Merge: ea919fa cba6f47
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Aug 24 11:57:28 2021 +0900

    Merge branch 'trunk' into release-spice

commit ea919fa5e1ecaef4d4f9450ac3eccf9137a0ee1a
Author: Luke Kim <[email protected]>
Date:   Mon Aug 23 12:04:49 2021 +0900

    Re-worked dashboard using React stack (#304)

commit d2947cad6a1da6eb574104fbfbb0a95996723b75
Author: Luke Kim <[email protected]>
Date:   Sat Aug 21 11:02:37 2021 +0900

    Support file structure and add tests (#301)

commit 3203407416636ef0a475eff44e8c54e2db418308
Author: Lane Harris <[email protected]>
Date:   Fri Aug 20 09:32:19 2021 -0700

    Include correct annotations in influx query (#302)

commit 710521510504851de20aa67555a0fc0993f32aee
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Aug 20 17:47:01 2021 +0900

    Safer python code execution (#300)

    * Adds initial safe eval

    * Add test for unsafe reward function

    * safe -> somewhat_safe

commit 20e46cb26e51692141ca6c122b19a84156479477
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Aug 20 16:41:19 2021 +0900

    Communication between spiced and ai engine is via gRPC (#299)

    Communication between spiced and ai engine is via gRPC

commit fc8b3f591b140b91b65986e609adf61b8249c29f
Author: Lane Harris <[email protected]>
Date:   Thu Aug 19 17:56:19 2021 -0700

    Allow environment variables in config and pod manifests (#298)

    * Env vars injected into container

    * Adds replacement of environment variables in config and pod manifests

commit f4f72bfc6de70f2fe5f791573b86aedc04c15d58
Author: Lane Harris <[email protected]>
Date:   Thu Aug 19 13:28:20 2021 -0700

    Update link to Trader and add Kubernetes to next steps (#297)

commit b4d7fe8b1a15bfa0367f23f7a9ca717d66bcffa9
Author: Luke Kim <[email protected]>
Date:   Thu Aug 19 17:50:09 2021 +0900

    Use zaplog + lumberjack for file logging and fix permissions issue (#296)

commit 0791d23d93c09b8098e72dbbefeb07ce310f17a1
Author: Luke Kim <[email protected]>
Date:   Thu Aug 19 16:06:13 2021 +0900

    Tweaks requirements, create a dev requirements and update gitignore (#295)

commit ea276262382ebc9a23bdb5e5084a92cac4602a39
Author: Luke Kim <[email protected]>
Date:   Thu Aug 19 15:46:40 2021 +0900

    Support loading a custom dashboard from filesystem (#294)

commit 2ad44b69eec518f03b24357ecac3063c25101594
Merge: 646b03f f28ad53
Author: Luke Kim <[email protected]>
Date:   Wed Aug 18 16:26:42 2021 +0900

    Merge pull request #292 from spiceai/merge-spicepre

    Merge from `spiceai/spiceai` to `spice-pre/release-spice`

commit 646b03f57ce1932634d5943f7b6955898f0ea2d7
Author: Luke Kim <[email protected]>
Date:   Wed Aug 18 12:34:49 2021 +0900

    Support decoupled data connectors and data processors (fetch only) (#288)

commit a46c035867bb302d4f4d9df52f656bbf8423951d
Author: Luke Kim <[email protected]>
Date:   Wed Aug 18 12:32:50 2021 +0900

    Fix new line

commit 9e1dee2b33892a39d294d2d87497e871a67c96d6
Author: Lane Harris <[email protected]>
Date:   Tue Aug 17 14:57:51 2021 -0700

    Allow spiced to access local services running on the host (#290)

commit 915609bf3501b65be4a4fe6b5b3fd2af3d9102f1
Author: Lane Harris <[email protected]>
Date:   Tue Aug 17 12:20:25 2021 -0700

    Adds log-pruner example (#289)

    * Adds log-pruner example

    * Use more realistic params

    * Add note for docker vs baremetal

commit 90207b12a24cec723d8b04444fc395fe7750ae64
Author: Lane Harris <[email protected]>
Date:   Mon Aug 16 14:04:32 2021 -0700

    Respects http_port in Docker mode (#278)

commit 67f8a1d2f3a5e738c564aadf37d62511f739e371
Merge: 976e367 bfad701
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 17:20:55 2021 +0900

    Merge pull request #276 from spiceai/merge-spiceai

    Merge spiceai/spiceai into spice-pre/release-spice

commit bfad7015da71f619ef1e009aca95bc27b7d5d951
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 17:18:02 2021 +0900

    Update flight.go

commit 06995b56808ff1765056a03d7840c8e090d08fbe
Merge: f144ab6 a4e38b0
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 17:16:42 2021 +0900

    Merge remote-tracking branch 'spiceai/trunk' into merge-spiceai

commit f144ab631067403ac37ecb21857e01747e481539
Merge: e4f5785 976e367
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 17:16:05 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 976e367c3af0720f462f5e8bca7e9e2519b8bd2b
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 16:57:59 2021 +0900

    Move getRelativePath outside of loop (#274)

commit 8ceaf3a46596b8a71dc8105ce7a9b1d8d6c893ad
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 16:56:48 2021 +0900

    Check for SPICE_GH_TOKEN on GitHubRegistry calls (#275)

commit ebb9ec50ab5dcc0e3b0f005edd3a2a9e00997d84
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 16:37:01 2021 +0900

    Restructure Single Run mode not to be coupled with Server (#273)

commit e4f5785ae9a9f49445cfdc4eddb2276cc191f3e3
Merge: cc21e19 ba6c407
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 13:06:52 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit cc21e19fee3a423bc1edae0fab7ea10a6b635292
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 13:01:49 2021 +0900

    Update tests

commit cc218bfeb8ac6b52261fceca93b0373c81712eab
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 12:59:39 2021 +0900

    Restructure single run

commit ba6c40700871cb30230609ba3603ab43c28fad0c
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Aug 16 12:52:25 2021 +0900

    Handle when a manifest is deleted (#271)

    * Handle when a manifest is deleted

    * Use zap logging

commit 22b439021d941016df6a61138ae5fa3ce9d8076e
Author: Lane Harris <[email protected]>
Date:   Sun Aug 15 19:24:27 2021 -0700

    Adds Zap (#267)

commit 3d9c65ab753da96737fd364a36d5b45236535f10
Merge: bba41c2 dd0d322
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:44:49 2021 +0900

    Merge pull request #270 from spiceai/spiceai-merge

    Merge spiceai/spiceai into spice-pre/release-spice

commit dd0d3227eb6911ab36859b5559e18f5227410d8f
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:40:23 2021 +0900

    Update test

commit e6bd9a617aa75fd4801154a1f8359da2ddec4c83
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Aug 16 10:39:09 2021 +0900

    Add datetime back

commit d7c539374324e33bcbde373183f9126b94dc8e35
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:18:13 2021 +0900

    Fix test

commit 1b0da1b4ae1d30302a083ae54fca4cec5c13eef7
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:17:31 2021 +0900

    Add back Shutdown

commit 4afbdc391dd7e996453df9d4cd6313e8a0f7ca6c
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:15:37 2021 +0900

    Update formatting

commit f2090e9064e3534dcdce9b315eb311643c004a91
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:14:36 2021 +0900

    Re-add getPodInitForTraining

commit bd53bdfb8037a8d43948e05667d25b1eb16668bf
Merge: ded0230 af5ac37
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:09:29 2021 +0900

    Merge remote-tracking branch 'spiceai/trunk' into spiceai-merge

commit ded0230816b4707aa43adb621779ca642ed9a7b9
Merge: 19d2a27 bba41c2
Author: Luke Kim <[email protected]>
Date:   Mon Aug 16 10:07:49 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit bba41c2d185696d6791a568414f3cfded2412b2d
Author: Lane Harris <[email protected]>
Date:   Fri Aug 13 12:48:21 2021 -0700

    Fixes inferencing blowing up on no or too little data (#266)

commit 542bbfe3e6ed5ff688dd56f903b77f47ca07c7d6
Author: Lane Harris <[email protected]>
Date:   Fri Aug 13 10:52:15 2021 -0700

    Changes inference times to ISO8601 format (#265)

commit b23c8199e3237be8779236f8c11c7ba2dde9029f
Author: Lane Harris <[email protected]>
Date:   Fri Aug 13 09:57:30 2021 -0700

    Fixes orphaned python issue (#263)

    * Do not orphan python process

    * Adds DQL test

commit 19d2a270382fa7c8ceb03b2660e4cb592a09e736
Merge: 4e36507 d0d1f7d
Author: Luke Kim <[email protected]>
Date:   Fri Aug 13 14:43:50 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit d0d1f7d50072bd24d9d07e6bcd11f5dd25277a9d
Author: Lane Harris <[email protected]>
Date:   Thu Aug 12 22:41:29 2021 -0700

    E2E test using a "single run" mode (#253)

    E2E test using a "single run" mode

commit b68d6bab95339b8e593c6e03e44cec4ae508d4e9
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Aug 13 09:40:03 2021 +0900

    Common interface for Spice RL algorithms (#250)

    Implements a common interface for RL algorithms

    Also implements a new algorithm "dql" -> Deep Q-Learning.

commit eea2e57e25c07620d3a0eff45755982235aaabef
Author: Luke Kim <[email protected]>
Date:   Fri Aug 13 09:15:40 2021 +0900

    Fixes Put a colon (:) after "Estimated time remaining" in the episode progress bar #237 (#258)

commit c65f6af1a63522114723b81a7c1bcb7d76619706
Author: Lane Harris <[email protected]>
Date:   Thu Aug 12 15:18:03 2021 -0700

    Do not check for new updates if running on edge (#262)

commit 9e2063d39285dd7eb3c4e527997b0b830e08634d
Author: Lane Harris <[email protected]>
Date:   Thu Aug 12 14:53:34 2021 -0700

    Brings cartpole and associated tests in line with what actually works (#261)

commit 5e32b933a8df7f364ea13c5386c8352d58e8d384
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 20:28:41 2021 +0900

    Fix ScanForPods should not return error on no .spice or no pods (#260)

commit 459755421ab93bcf7ba7de2ae9211311245698da
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 19:43:12 2021 +0900

    Fix The reward function needs to convert periods to underscores before sending to the AIEngine (#257)

commit 4e36507078c46eaf2f188d3e6d2eb446aa6b63de
Merge: 10da41f 8d9e945
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 17:50:32 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 8d9e945c3b1bceb620d1650df1dcbfd021959f35
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 17:02:52 2021 +0900

    Fix training goal not being passed to AI Engine (#256)

commit 10da41faa014a41e244e6ed8d19865123d5ba65d
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 16:42:44 2021 +0900

    Remove extra files

commit e40091e75b56f8f491877ce7e93b25b61d7587ed
Merge: 6cfb308 657f1e1
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 16:16:41 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 657f1e1c3e2ead7122dc7589e9d4b1cf213441a8
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 16:16:28 2021 +0900

    Merge spiceai/spiceai into spiceai/spice-pre (#255)

commit 6cfb308c7ef09a0b28a542fbaaeb15757408ed4d
Merge: a445035 980e6bb
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 13:08:13 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 980e6bb8303a74f4b5ff2f476445a4d3ab100a6f
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 12 13:07:38 2021 +0900

    Fixing AI Engine tests (#254)

commit a44503540b8a8828050c8069d1fc7c129c3c280f
Merge: 78e653a 8e2fe47
Author: Luke Kim <[email protected]>
Date:   Thu Aug 12 12:41:28 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 8e2fe4714b73e83165321c2175a52237ff62dce3
Author: Lane Harris <[email protected]>
Date:   Wed Aug 11 19:25:37 2021 -0700

    Do not attempt to merge empty data (#252)

commit 78e653af43d769d40a014fe0160b707fd0ff9931
Merge: 38e87bd 497f12c
Author: Luke Kim <[email protected]>
Date:   Wed Aug 11 19:41:41 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit 497f12c4c17daade8c034e0b95749e329e809be1
Author: Luke Kim <[email protected]>
Date:   Wed Aug 11 19:39:54 2021 +0900

    Restructure State and Observations (#251)

commit 38e87bd85398e362388cffc7e3076e24d7546b41
Author: Luke Kim <[email protected]>
Date:   Wed Aug 11 18:09:32 2021 +0900

    Merge

commit 4dbe9575f65fbed18799a3311b82b552d37764f8
Merge: ee4f7e4 5096a91
Author: Luke Kim <[email protected]>
Date:   Wed Aug 11 18:09:24 2021 +0900

    Merge branch 'release-spice' of github.com:spiceai/spice-pre into release-spice

commit ee4f7e40ebd9ea486e0e7c93069c597ac92ad642
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 17:26:28 2021 +0900

    Update README.md

commit bda8ad1707fba7b123069633bd1346fc492956dd
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 13:18:58 2021 +0900

    Remove Copyright (#231)

commit 6c6d14842a7659f8ef9d782e44c726a9f96acb64
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Aug 9 12:57:01 2021 +0900

    Handle when pod & datasource are not found in the registry (#229)

    * Handle when pod & datasource are not found in the registry

    * Review feedback

commit 60735111350dd7b4a0b71aa513fe6581b8bf5883
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Aug 9 12:56:35 2021 +0900

    Remove CI and banner in CLI (#230)

commit e8af3de6fb1c0f3e14a4a90cf33249f2a7858ce7
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Aug 9 12:23:27 2021 +0900

    Adding more detailed contributing guide. (#224)

    * Adding more detailed contributing guide.

    * Removing code of conduct link

    * Update CONTRIBUTING.md

commit 1aebdee556fb620b972a096db8670a875f348350
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 12:04:29 2021 +0900

    Use spiceai/spiceai instead of spice-pre (#228)

commit ddb1dbbdb4284d28c0b3de16902eaa78480428bc
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:37:23 2021 +0900

    Update README.md

commit b7ec4f4d749c9348b1445c4fe274747e70c88a6b
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:35:27 2021 +0900

    Update README.md

commit aa20d7ac88fa48683ab8faf990673feaced4cb3c
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:34:38 2021 +0900

    Update README.md

commit 9cbe59826802c85c01dac3cddc6552ca5066830c
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:33:28 2021 +0900

    Update README.md

commit b27e4280de24c967fb2939a565f1325a68469fd3
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:32:47 2021 +0900

    Update README.md

commit 89d7608fb9f1e40f8a63be9f7a6e96bd95ab3c6c
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:32:25 2021 +0900

    Update README.md

commit 9239319cce35ba6777c76e0ffd59fc2d8ef879e7
Author: Luke Kim <[email protected]>
Date:   Mon Aug 9 11:30:45 2021 +0900

    Update docs (#225)

commit a8959caa41768b2453aed666890c3deceacf1c95
Author: Lane Harris <[email protected]>
Date:   Sun Aug 8 16:22:48 2021 -0700

    Update README and CoC (#219)

commit 5dedc324f04b6dac55f8f3d4269348de169c0a44
Author: Lane Harris <[email protected]>
Date:   Fri Aug 6 14:34:02 2021 -0700

    Fix panic on infer (#218)

commit 2e0069ff27d99bd73e91bf94683592c7672d12f4
Author: Lane Harris <[email protected]>
Date:   Fri Aug 6 10:48:03 2021 -0700

    Adds a check for jq and cleans up error output during requirements check (#215)

commit 02f5590903121ffac5cb1ca28b614c1f9562bac9
Author: Luke Kim <[email protected]>
Date:   Fri Aug 6 22:10:14 2021 +0900

    Fix some bugs in local observation round-tripping (#214)

    * Fix bugs

    * Record with only time

commit f00110763309935000283a5c8f5192013dd5ed69
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Aug 6 21:32:17 2021 +0900

    Changes to make docker work as the default (#212)

    * Use non-root user for Docker

    * Changes to make docker work as the default

    * Fixing err return

    * Change latest -> dev for not upgrading the docker

commit d3026b9d08be2c0fa2e12f332d40a0285c3543ca
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Aug 6 15:06:32 2021 +0900

    Fixing the OpenAI Gym connector (#211)

    * Fixing the OpenAI Gym connector

    * Fix formatting

    * Adding test for cartpole

    * Adding test data file

commit 34db68b3986011cb8ee17f65281551c5352dd161
Author: Lane Harris <[email protected]>
Date:   Thu Aug 5 16:23:08 2021 -0700

    Adds simple test for local registry (#210)

commit 7305c9b7c4f82b06af34c9650ebe7ec0786d6bd8
Author: Lane Harris <[email protected]>
Date:   Thu Aug 5 15:16:32 2021 -0700

    Adds test for flights (#209)

commit 1294f20c9389b69c3494f22012415f8a2ea0d61c
Author: Lane Harris <[email protected]>
Date:   Thu Aug 5 15:09:34 2021 -0700

    Test environment (#208)

commit acdb99de26045c658e1813a726903e3834e6a5eb
Author: Lane Harris <[email protected]>
Date:   Thu Aug 5 14:17:49 2021 -0700

    Extract useful test utils (#207)

    * Extract out test utils

    * Removes duped definition

    * Replace test

    * Put fake test where it belongs

commit 2dcad457686983bf0c1a9ab6682e202abdab141f
Author: Luke Kim <[email protected]>
Date:   Fri Aug 6 02:52:34 2021 +0900

    Enable start training by CLI (#206)

    * WIP

    * WIP

    * WIP

    * WIP

    * Add tests

    * Fix formatting

    * Add missing module update

    * Update Gitignore

    * Update tests

    * Fix

    * Remove demo hack

    * Formatting

    * Add test

    * Fix path

    * More paths

    * WIP

    * Yeah

    * Fixes broken AI engine test

    Co-authored-by: haardvark <[email protected]>

commit 7581cf890cbd814f696ffb3f355f8cf6a3a81146
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 5 20:42:10 2021 +0900

    Improve the performance of the AIEngine during training (#205)

    * Adding progress bar to show estimated time remaining in training

    * Add debug environment var to F5 debugging

    * Adding metrics

    * Add test for stateful connector

    * Test that we are calling merge_row on apply_action

    * Remove slash

commit c36319e4ea645751640bbcf3df61dab0ab907c1f
Author: Luke Kim <[email protected]>
Date:   Thu Aug 5 20:30:38 2021 +0900

    Remove Influx test hack (#204)

commit 2beb5aed7fe5017ffb0d1f0a4b5021ab405b84b7
Author: Luke Kim <[email protected]>
Date:   Thu Aug 5 20:30:20 2021 +0900

    Remove Influx test hack (#204)

commit 1e57a95df9a99887e09ea79d405088c02a7ef7de
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Aug 5 14:23:52 2021 +0900

    Adding unit tests for AI Engine (#202)

    * Adding unit tests for AI Engine

    * Fix flaky test

    * Test StartTraining

    * Making error more robust

    * Fixing log dir not found issue

commit 3f2c21e362ec9bdba3920fb716760d821be8756b
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 15:34:12 2021 -0700

    Adds dashboard tests (#201)

commit e253ba263a343d549818797de3a7234c8b5a63aa
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 14:26:22 2021 -0700

    Adds CLI tests (#200)

commit 89377814539b45183c3fe4e9924755fcbc30cea5
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 12:57:05 2021 -0700

    Adds context test (#199)

commit 9319ddfb12c940889513d709cf3517c02593104b
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 12:32:00 2021 -0700

    Adds stub connector tests (#198)

commit d9a48efb9f06bec892b4b3c0bd62bf7ae95acaf5
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 10:41:45 2021 +0900

    Adds a basic test for loading config (#197)

    * Adds basic config test

    * lint

commit bcff01d05f53b8192c3c32663329a25b695a2481
Author: Lane Harris <[email protected]>
Date:   Wed Aug 4 08:52:18 2021 +0900

    Only consume data that falls within period (#196)

    * Do not advance beyond epoch plus period

    * Test inferred epoch time

    * Updates end time if a new epoch is passed in

commit c86cb9eb47ce8c54d74bdb7a693f2b80840466db
Author: Luke Kim <[email protected]>
Date:   Wed Aug 4 04:19:37 2021 +0900

    Pass along health status to spiced (#195)

    * Pass along health status to spiced

    * Tweak

commit b8b22e9dc877c2057aff340d6ecd88447392cd7e
Author: Luke Kim <[email protected]>
Date:   Tue Aug 3 22:19:04 2021 +0900

    Re-architecture (#159)

commit 0643faf8702ebae57d5e35ad7e3aa3005877d695
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 29 18:15:42 2021 +0900

    Creating new train API that will receive data from spiced in timestamp format (#149)

    * simplify the openai gym connector

    * remove dataloader

    * Simplify the env

    * Move dataloaders to separate interface

    * Get training data from spiced

    * Training loop works (#152)

    * Training loop actually works

    * Don't explode on missing rows

    * Don't carry time forward

    * Fix window size

    * Using data frames for data storage

    * infer start time

    * Adding gitignore

    * Add connector managers with POST /connector endpoint

    * add start time

    * Initialization and merging works

    * Connector initializers works

    * Make massive_table naming clearer

    * OpenAI Gym almost working

    * handle cartpole, rewards and laws properly

    * reworking to match expected endpoint

    * inferencing working

    * removing unused files

    * Bump pandas

    Co-authored-by: Lane Harris <[email protected]>
    Co-authored-by: Lane Harris <[email protected]>

commit 5b89b291749ba8810a09da90db712ca6ed36d466
Author: Luke Kim <[email protected]>
Date:   Wed Jul 28 17:57:43 2021 +0900

    Refactor datasources/connectors and add CSV support (#156)

commit 855bdd2fc468cfc5b1bb9867655b40ae5250c31c
Author: Lane Harris <[email protected]>
Date:   Wed Jul 28 03:49:22 2021 +0900

    Set perms to 766 (#150)

commit 61e0c4fdc0852721359ffdaf5e20891dc1892d9d
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 27 12:40:15 2021 +0900

    Adds linter and test to release-* branch (#148)

commit 8aacca6c87e795510032c1b39d596171726ff542
Author: Lane Harris <[email protected]>
Date:   Tue Jul 27 07:54:32 2021 +0900

    Don't set context in init (#146)

    * Only CLI commands that run docker should use docker context

    * Bump version

    * Do not set context when adding command

    * Bump version

    * Remove init

commit cc53914aefa9855e87b1b2d6314a4d141d6f9c57
Author: Lane Harris <[email protected]>
Date:   Tue Jul 27 06:58:24 2021 +0900

    CLI commands that don't interact with docker shouldn't use docker context (#145)

    * Only CLI commands that run docker should use docker context

    * Bump version

commit 1935a7ef3d0d3f6ed264077d34d3ec2013ae9b8c
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 26 21:32:21 2021 +0900

    Bump version

commit 0cd88f1c787a02daf32ffab5c24651395ada06e8
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 26 21:30:39 2021 +0900

    Only retrieve the past 24 hours of data (#144)

commit 01713d753f8e14a52ca51e751ab85ce9e8aa7f2a
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 26 19:23:43 2021 +0900

    Adding code linter & PR tests (#143)

    * Fixing golangci-lint issues

    * Add linter workflow

    * turn off copy/paste detection

    * superlinter is broken, use golangci directly

    * Adding black linter

    * format with black

    * format with black

    * Simplify the linter

    * Add testing workflow

    * Use latest go

    * Add example test file

    * Run python tests

commit 735eaa0b6402d7fb069296dee2182d793f8de2b0
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 26 17:04:15 2021 +0900

    Use correct timestamp (#142)

commit 6d317d011f717edfc9c65e3e3a5cd0c891e90d0d
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 26 11:24:29 2021 +0900

    Change wait time to 1 minute for inferencing server to start (#141)

    * Change wait time to 1 minute

    * Bump version

commit becd39e6ceae0f665dfadca76f1fb79356501a7f
Author: Lane Harris <[email protected]>
Date:   Sat Jul 24 09:24:38 2021 +0900

    Handle OpenGL import error inside container (#137)

    * Handle OpenGL import error inside container

    * Bump version

commit b59cb2ed1e79ef7f0d2503abdfa681dc9847c7bd
Author: Lane Harris <[email protected]>
Date:   Fri Jul 23 16:04:09 2021 -0700

    Bump semver

commit 569739d6b34078ae23de22f0576527a2ccd98385
Author: Lane Harris <[email protected]>
Date:   Fri Jul 23 15:57:46 2021 -0700

    Bump version to next legal semver

commit cc79a87b3860334a41ae9a5fa358a94fa7c461dd
Author: Lane Harris <[email protected]>
Date:   Fri Jul 23 15:37:00 2021 -0700

    Catch rendering exceptions (#136)

    * Do not explode if no windowing environment is available

    * Bump version

commit 38702fbc95f1a75f6e98a70472a480aef90b1b88
Author: Lane Harris <[email protected]>
Date:   Fri Jul 23 15:17:29 2021 -0700

    Do not explode if reward is not assigned (#135)

commit 68d59cef9186a77bf6bb477ddd3af0394cb990c1
Author: Lane Harris <[email protected]>
Date:   Fri Jul 23 15:08:04 2021 -0700

    Adds default rewards (#134)

commit 289b855c2cb596a0883220a98f70daeb994f50cf
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 21:17:31 2021 +0900

    Check inferencing health endpoint for ready and cleanly shutdown (#130)

    * Check inferencing health endpoint for ready and cleanly shutdown

    * Tweak

commit dd10d78d4643494cd3cbd846a12aaf71a6f2b9f1
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 21:16:00 2021 +0900

    Colorize basic output (#131)

    * Colorize basic output

    Closes Colorize output #110

    * Add go modules

commit 08f5a84c7d6a55acbde2f81a0d78d75d0e6286ac
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 21:14:11 2021 +0900

    Support workflow run pagination (#132)

commit 0b7a597d04b9216722fb0a0aa31daa56bff365cc
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 23 13:27:59 2021 +0900

    Add model metadata and bump version (#128)

    * Add model metadata on inference

    * bump version

commit 219f58f01f718c451161411948e78cb867f9e142
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 23 13:18:41 2021 +0900

    Retry on 500 for inferencing & use the length of the probabilities for act (#126)

commit 19d77486fc22aa948e662a2282f363aa7a76e106
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 13:18:34 2021 +0900

    Add back observations (#127)

commit 9b5aed6e13862d78d264fe9d410ec87d5ca24785
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 11:58:30 2021 +0900

    Fix visualization, add logging and bump version to alpha.8 (#125)

    * Add additional logging

    * Add logging and fix visualization

    * Bump version

commit 3115b6b64c731e12f3e374145be66ccb017ac711
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 17:47:02 2021 -0700

    Show whole numbers if score is int, two decimals otherwise

commit 6f3392916d528224af9e3d5d70dd7ea5c59f54af
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 17:37:22 2021 -0700

    Bump version

commit ee1832766633dfc28e38b72a950cde489d23993d
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 09:30:36 2021 +0900

    Fix path checks

commit ef3a907121f5d54f074b347d830c76f834a997f6
Author: Luke Kim <[email protected]>
Date:   Fri Jul 23 09:25:34 2021 +0900

    Update configuration.go

commit c06202dd129dbea36eaa4ef5aaf5a2f62d0ea7e0
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 17:11:58 2021 -0700

    Bump version

commit 6c088fd7467d1c87c9ec8e508ae387b3f12711cf
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 17:11:03 2021 -0700

    Change to trader.yml

commit 090fd095f4747a8e64a43fdaff362777a7180879
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 17:02:37 2021 -0700

    Bump version

commit cc621240ef2dec1b7e54caeca9d42ad118578476
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 14:49:08 2021 -0700

    Change to using SPICE_GH_TOKEN everywhere (#119)

commit ec23b48b35add7cd0a83872d674b119507333f6c
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 13:42:09 2021 -0700

    Rounds score to two decimal places in output (#118)

commit 6bafeafbf80e74fdce95207b175c90f63471ed18
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 13:06:35 2021 -0700

    Changes result to reward (#117)

    * Updates phrasing of result to reward

    * Removes examples from repo in favor of using the ones in the registry

commit cf3b1e30522ec87c4e7ed1743ce5a891f8959ef4
Author: Lane Harris <[email protected]>
Date:   Thu Jul 22 12:19:40 2021 -0700

    Makes initializer a float (#116)

commit da2d42061f4852ac9347e282b4464f4d4df5df9f
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 22 21:08:54 2021 +0900

    Better error message for not being able to load influx data (#115)

    * better error message for not being able to load influx data

    * Adding messages for data loading

commit 7699b08e1e0d3fb48d581e647364bd0ace9c0b1f
Author: Luke Kim <[email protected]>
Date:   Thu Jul 22 19:58:06 2021 +0900

    Wait for config file to exist (#114)

commit 2f6d64185c4bb9c4569041a101093353438a6a4b
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 22 17:42:06 2021 +0900

    Fix yaml marshalling for action add (#102)

    * Fix marshalling for action add

    * Bump version

commit a2f74a1cb2f37a34ee0d3d867e323af1039eea3a
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 22 17:17:51 2021 +0900

    Fixing the release publishing (#101)

commit 29f89cef48c96d034c6d27766dad9b132f6e65eb
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 22 17:07:05 2021 +0900

    Fixing the model download error (#100)

    * Fixing the model download error

    * Bumping version

commit 2813d3d9ad67b10b0b70cf8f30972bcb56bcdc37
Author: Luke Kim <[email protected]>
Date:   Thu Jul 22 16:13:07 2021 +0900

    Publish Linux binaries (#99)

commit 9c3cc8d7352d24c1b651ea7f1aabfda1e8d37dd4
Author: Lane Harris <[email protected]>
Date:   Wed Jul 21 16:53:41 2021 -0700

    Makes symlink work again (#97)

commit bb30b47b950630c0a26007cafb3999e256c5fe9a
Author: Luke Kim <[email protected]>
Date:   Thu Jul 22 07:47:30 2021 +0900

    Marshal the PodSpec

commit 3ead619c30896fac2b097056dc9051fa1543aff4
Author: Luke Kim <[email protected]>
Date:   Thu Jul 22 07:36:43 2021 +0900

    Unmarshal Pod correctly (#96)

commit 12e9214324bac06b852b839ab27075381ecbd34b
Author: Luke Kim <[email protected]>
Date:   Wed Jul 21 18:07:31 2021 +0900

    Write pod init manifest from actual PodSpec (#94)

    Closes #93

commit 255113644393bec329b26889bb3641bc2ae4528c
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 21 18:00:01 2021 +0900

    Add support for versioning and auto-upgrade (#90)

commit 00795c675d27498bc502e57e11b9d9534845e169
Author: Luke Kim <[email protected]>
Date:   Wed Jul 21 17:27:04 2021 +0900

    Get datasources from "registry" (#91)

commit cae22ad601782b1cd2f7b845cab8034008411e1e
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 21 10:25:54 2021 +0900

    If the current environment is Docker, don't fall through (#88)

commit e66dd0ee80850f61a668d311bceed9bb657ac203
Author: Lane Harris <[email protected]>
Date:   Tue Jul 20 16:04:12 2021 -0700

    Adds actions using actual pod structure (#86)

    * Update action add example

    * Refactor getSinglePod

    * Working

    * gofmt

    * Refeactor WriteToFile

    * Fix bug in AddElementToString

commit 5d548999f15f17b964b4c76b92259c06991346ff
Author: Luke Kim <[email protected]>
Date:   Tue Jul 20 22:31:32 2021 +0900

    Update spiced_docker.yml

commit 770888ed30cf1f018e858ab163b79f7e7ab3c83b
Author: Luke Kim <[email protected]>
Date:   Tue Jul 20 22:16:37 2021 +0900

    Update spiced_docker.yml

commit 54bd1dbb5bd9df5db54eb06d81c862ccd47af13b
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 20 21:06:59 2021 +0900

    Use docker to run spiced and the ai engine (#84)

commit 5a62898bdd21d111d6c0f4555bb8ee0788393f5c
Author: Luke Kim <[email protected]>
Date:   Tue Jul 20 21:03:16 2021 +0900

    Download Pods from "registry" (#85)

commit 049aef758a4d6260e4880a373edfdad17296326e
Author: Luke Kim <[email protected]>
Date:   Tue Jul 20 19:36:26 2021 +0900

    Download spiced for real (#83)

    * Download spiced for real
    Closes #81

    Also refactor github client code into common pkg

commit 85311b9283e740a813f4b7ce3699162801b084b6
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 20 15:58:10 2021 +0900

    Adding vscode launch.json for Go debugging (#82)

commit 481defdaab8bff2e09b6b56c99cc2d943595a790
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 19 22:50:39 2021 +0900

    Fixing the release publishing (#80)

commit 68568e1edcda6ca125ba24a3f6c293376c96efae
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 19 22:35:01 2021 +0900

    go mod tidy (#79)

    * go mod tidy

    * Fixing release publishing

    * Only publish on trunk builds

commit 0725ad77d4d556398282bdb4e7d6c515a82b2590
Author: Luke Kim <[email protected]>
Date:   Mon Jul 19 19:37:17 2021 +0900

    Embed Dashboard static files into spiced (#78)

commit 88071691d01e30860d5f853b3882e40d74f243bb
Author: Luke Kim <[email protected]>
Date:   Mon Jul 19 18:37:34 2021 +0900

    Standardize back to "Pod" naming (#77)

commit c3384e6f831cb5ca5451f1465e5cbd9362e8652d
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 19 17:46:39 2021 +0900

    Adding Spice CLI into main repo (#76)

    * Adding Spice CLI into main repo

    * Fixing github actions workflow

commit 8104bc72f1a29d82f5fd5b0cef6bb455a9f3c7ea
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 19:14:50 2021 +0900

    Fix 400 after training (#63)

commit f2575a5c23e46a3127309cda5f1eb7140836faae
Author: Lane Harris <[email protected]>
Date:   Fri Jul 16 03:11:46 2021 -0700

    Sleep longer (#62)

commit cce97d49094431a15d8462786481c46dd773ccb3
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 18:01:55 2021 +0900

    Don't show an error message if we can't load a model (#61)

commit 5bf5ebb3117d732e6508d7f5c390033558488814
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 17:29:49 2021 +0900

    fix negative rewards (#60)

commit ed8466b8341674b42a1a110978cf0a469c16e685
Author: Lane Harris <[email protected]>
Date:   Fri Jul 16 01:27:29 2021 -0700

    Don't show TF warnings (#59)

commit 214c062689605abc38f16e00f65fb9993d888761
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 17:17:25 2021 +0900

    Fix config loading & inference server text (#58)

    * Fix config loading & inference server text

    * Revert name change

commit 11b4824c6636625ff5fc32bdcd5a2ad4cbe5c30e
Author: Lane Harris <[email protected]>
Date:   Thu Jul 15 23:52:10 2021 -0700

    Rename httpport (#57)

commit 02fe65258c35664b10a91bac264a5ff368c902c2
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 15:47:15 2021 +0900

    Get the formation from a query string (#56)

commit d90aeb1988f599f9d1d14993306e4ab0d9e976ec
Author: Lane Harris <[email protected]>
Date:   Thu Jul 15 23:25:40 2021 -0700

    Stop training after not learning for several episodes (#55)

    * Stop training after not learning for several episodes

    * Updates wording

commit 4a36c2b65775c6f27c9c5e752e53cbb851d5edb8
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 16 15:17:16 2021 +0900

    Fix inferencing and config not reloading (#54)

    * wip

    * Changing config back

    * Fix the config loading

    * Start inferencing server after header

commit 3f84398fe0de8e1bf74d8bbea0efef0bbd2b7e9b
Author: Luke Kim <[email protected]>
Date:   Fri Jul 16 12:07:39 2021 +0900

    Fix inferencing on start training issue (#53)

commit a4bb7b8ec41d6d0c5327d7291117693b26826193
Author: Luke Kim <[email protected]>
Date:   Thu Jul 15 21:23:49 2021 +0900

    WIp (#52)

commit d5d8e261302bf5f92ba9040536124c60607eb672
Author: Lane Harris <[email protected]>
Date:   Thu Jul 15 04:10:56 2021 -0700

    Lane/add reload by path (#51)

    * Adds reloading by path

    * Actually change inference

commit f80244750563c6279b885009451c2b0024d17165
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 15 20:05:03 2021 +0900

    Download models from GitHub API (#50)

    * Download models from github api

    * Fix formatting

    * Merging with trunk

commit 9da4e69c2d60e55692e7ec8f7e797bf112b14b44
Author: Luke Kim <[email protected]>
Date:   Thu Jul 15 18:19:01 2021 +0900

    Call downloader (#49)

commit 3e2c4e2afebb3de6247e8a39c35dc03ce620436e
Author: Luke Kim <[email protected]>
Date:   Thu Jul 15 16:19:32 2021 +0900

    Config updates (#48)

commit eec14cd0c397ec73f4b2d456fcca524b781644d3
Author: Lane Harris <[email protected]>
Date:   Wed Jul 14 20:27:46 2021 -0700

    AI docker (#47)

    * Build a docker image for the ai project

    * change period for demo

    Co-authored-by: Phillip LeBlanc <[email protected]>

commit 1ba632a2b9e4751123e4a053ae19f6da0b58c9f7
Author: Luke Kim <[email protected]>
Date:   Wed Jul 14 22:40:52 2021 +0900

    Add hacky visualizations (#45)

commit e199c37688c41b87c6dd6848b415cc7bd4c3aecf
Author: Lane Harris <[email protected]>
Date:   Wed Jul 14 06:33:12 2021 -0700

    Infer actions from datasources (#46)

    * Infer actions from datasources

    * invert conditions

commit 2c0a23ab75d9c64bdd5085d663029fd223de3bf6
Author: Lane Harris <[email protected]>
Date:   Wed Jul 14 04:09:06 2021 -0700

    Update validation (#44)

    * Updates validation

    * WIP

    * Update validation for rewards

    Co-authored-by: Luke Kim <[email protected]>

commit 9305fe569b02aeacdf7afe2ae6e07f2ac22731d5
Author: Luke Kim <[email protected]>
Date:   Wed Jul 14 19:08:39 2021 +0900

    WIP (#43)

commit cafc0692199577bca9fa45dce5fd4ab0f76121f4
Author: Lane Harris <[email protected]>
Date:   Wed Jul 14 02:51:24 2021 -0700

    Updates validation (#42)

commit 1b6b8613ab12db4c2b420d24831058ba26455c4c
Author: Luke Kim <[email protected]>
Date:   Wed Jul 14 18:27:24 2021 +0900

    Add API for visualizations (#39)

commit 56933095217b9fc828e4d82112448dce920d846b
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 18:02:15 2021 +0900

    Change the schema + uniform rewards (#41)

    * wip

    * Change the schema + uniform rewards

commit 5b988bdd7cc189156f4fe8217a42451440135577
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 17:40:22 2021 +0900

    Fixing inferencing (#40)

commit a47702704be48d9174d3c81cba924c7d0fee7fc0
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 15:19:46 2021 +0900

    Fix the training validation (#37)

commit 805091b6e257044fad1adfcccff5f44b779ca88b
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 15:07:38 2021 +0900

    Add training & goal + Training output fixes (#36)

    * wip

    * Training output fixes

commit 6ab1229c554c4b2b6b623e2c03cc9f86f6777e3c
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 12:16:19 2021 +0900

    Start inferencing server and reload after training (#35)

commit 3bdf759f5778a3196f7bb7c582cd3201ee730b98
Author: Luke Kim <[email protected]>
Date:   Wed Jul 14 11:33:57 2021 +0900

    Add basic API (#34)

    * WIP

    * WIP

commit a6fa46ded6d936bf0839c0af1a3cfe956976db07
Author: Phillip LeBlanc <[email protected]>
Date:   Wed Jul 14 11:19:59 2021 +0900

    Save and load models & provide inference API from python (#33)

    * wip

    * save and load models

commit 6683b8f38b7967d81077f1ba518deb80a25a7d67
Author: Lane Harris <[email protected]>
Date:   Tue Jul 13 05:13:39 2021 -0700

    Adds Cartpole render and removes artificial sleep (#32)

commit d1a5b9e224b423b58eb06f0912ea699969bd101f
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 20:18:39 2021 +0900

    Use symlink for "ai" and other tweaks (#31)

commit 979135761dc8323f4c7e8a2e19a65ed3e66b9055
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 19:52:53 2021 +0900

    background the watcher for real (#30)

commit b6a45105fcb20c3f69e65d814b8c4018f6d37d80
Author: Lane Harris <[email protected]>
Date:   Tue Jul 13 03:50:30 2021 -0700

    Disable TF deprecation warning (#29)

commit 58c0ed78b57c0457e89206d018ae6777eaaf080a
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 19:31:39 2021 +0900

    background the filesystem watcher (#28)

commit d9256c2b0bcf433e2f6f7a0297e48837e9f85f19
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 19:20:52 2021 +0900

    output (#27)

commit fab80ef3b1f857a8198e79b3a15b86d17e61a093
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 18:50:42 2021 +0900

    remove onnx (#26)

commit 6547f5511f14789ba5b0c75cc31cfbc0aaea3c23
Author: Lane Harris <[email protected]>
Date:   Tue Jul 13 02:42:58 2021 -0700

    Cleanup more output (#25)

commit 9b34c6cfd503914c4f208b17053cea691869d7f1
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 18:39:01 2021 +0900

    remove buffering output (#24)

commit c5a5025b89cac22980c77dbd885ea46cc1ccaaa6
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 18:29:30 2021 +0900

    wip (#23)

commit 9054c81ec04f682f638a36a19a5a69d5de3f8aa7
Author: Lane Harris <[email protected]>
Date:   Tue Jul 13 02:27:17 2021 -0700

    Fix double training (#22)

    * Only retrain when hash changes

    * Simplify

commit ca4c72521bfe38f75973315806bb51678bec17db
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 18:24:56 2021 +0900

    Add very basic validation (#21)

commit ee55e306b7d4263ce17c0050be7a17fa87b82830
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 13 17:59:59 2021 +0900

    remove binary

commit eada4a51ee23109ca2215dca20b664a505f290fe
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 16:22:57 2021 +0900

    Read config into internal structure for validation (#20)

commit d80ef82deb283da99cc6d04976fba3c61c5d283c
Author: Lane Harris <[email protected]>
Date:   Mon Jul 12 22:59:46 2021 -0700

    Fixes CWD bug (#19)

commit c330bfce571f29b53f9cbdf7240477db0e000341
Author: Lane Harris <[email protected]>
Date:   Mon Jul 12 22:42:52 2021 -0700

    Fix pathing (#18)

    * Fixes pathing

    * Only respond to yaml changes

commit a999d9827cb9f40393169f092d5cc1369e5c48fa
Author: Lane Harris <[email protected]>
Date:   Mon Jul 12 21:12:16 2021 -0700

    Copies AI project (#17)

commit 6beae170301d4930a4138c3994e7518f790b4ac8
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 13:00:39 2021 +0900

    Update spiced.yml

commit 00a522924404b53b25eefac9fb5a9ef73f71a295
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:56:37 2021 +0900

    Update spiced.yml

commit 3f644c552eaf5a14c5a41ba54e8300e0243d3145
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:48:24 2021 +0900

    Update spiced.yml

commit 88d5442d9872d9964ea0378e6dc767f5560fe21e
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:33:36 2021 +0900

    Update spiced.yml

commit 98eb6a6015d2539bb9ff0402047a0723dcde085d
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:31:07 2021 +0900

    Update spiced.yml

commit a72f932db1398064c5d0d54572e890911de03bbd
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:27:21 2021 +0900

    Update spiced.yml

commit 33bf9524b1a32248fcb84b6c4f8f63a052f2793f
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:23:43 2021 +0900

    Update spiced.yml

commit 1c4591e7f65eb7f5d063751eec6bf6d35b0b6ae2
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:19:57 2021 +0900

    Update spiced.yml

commit 9c2150b0920cdb0bcfb98b61417c4014c9b5dadc
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:14:53 2021 +0900

    Update spiced.yml

commit 4e28eb7b50bb9ac3d34a44d612bfb1e3cc4204cb
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 12:10:31 2021 +0900

    Update spiced.yml

commit 813e032279ca309a6a334785517909b4d88a089b
Author: Lane Harris <[email protected]>
Date:   Mon Jul 12 20:03:36 2021 -0700

    Makes spiced watch .spice directory and train whenever changes are detected (#16)

    * Get Cartpole on generic MacOS working again

    * Makes spiced watch spice directory and trigger training on changes to it

    * Use relative path for python

commit d5235a86cb1bfac543c979672bd581cd53450323
Author: Luke Kim <[email protected]>
Date:   Tue Jul 13 11:31:19 2021 +0900

    Add publish stage (#15)

commit ee05d475748c8c54b1e2bcef0bacb7c2a4f83b35
Author: Phillip LeBlanc <[email protected]>
Date:   Mon Jul 12 19:05:11 2021 +0900

    Finalize the yaml syntax (for now) (#14)

    * wip

    * ship it

commit a76034274403c0b11d642202ace1f5826d824e2f
Author: Phillip LeBlanc <[email protected]>
Date:   Sat Jul 10 14:29:26 2021 +0900

    Adding variable support (#13)

    * wip

    * wip

    * it works!

    * wip

    * wip

    * Adding normalization of balance

    * Try 2 at normalizing

    * wip

    * tweaking reward

    * wip

    * use initializer

    * laws

    * wip

    * merge in from trunk

    * Fixing the reward function input

commit e112db1edd6a914d277f2f0dcbf1ddb4ce4b3518
Author: Luke Kim <[email protected]>
Date:   Sat Jul 10 14:01:46 2021 +0900

    Support TensorFlow macOS and Metal (#12)

commit e02269e86fc5727b91e13e7ab08063267b175667
Author: Luke Kim <[email protected]>
Date:   Fri Jul 9 22:06:43 2021 +0900

    Lukim/loader (#11)

commit 9d095d8279f25f56c6c0f0528644a974cd908812
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 9 22:03:59 2021 +0900

    Add the actions yaml parser (#10)

    * wip

    * wip

    * it works!

    * wip

    * wip

    * Adding normalization of balance

    * Try 2 at normalizing

    * wip

    * tweaking reward

    * wip

    * use initializer

    * laws

    * wip

commit 1a367e64d2f7772e7685221f292a380096ea98c4
Author: Phillip LeBlanc <[email protected]>
Date:   Fri Jul 9 09:03:57 2021 +0900

    Phiilip/phase 2 (#9)

    * wip

    * Passing state correctly

    * Training!

    * wip

commit 3ca1c2cb667a5bcb3679816ae55f99eaade927be
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 8 16:30:28 2021 +0900

    Phillip/secret sauce (#8)

    * Adding ai project

    * Train using special sauce

    * fix

commit 7b94c0890717f9c779e6695bb911e236d359a595
Author: Phillip LeBlanc <[email protected]>
Date:   Thu Jul 8 11:06:45 2021 +0900

    Adding ai project (#7)

commit fd0ff1f655fedfdf12a78fc7802b801f793e2537
Author: Luke Kim <[email protected]>
Date:   Tue Jul 6 20:52:31 2021 +0900

    Add tweaks (#6)

commit e92693f6cfd664209297f4fba9ced438dff23026
Author: Luke Kim <[email protected]>
Date:   Tue Jul 6 20:48:01 2021 +0900

    Additional README updates (#5)

commit 566d8540407c05b28a6586a60ebbb2b1874880de
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 6 20:37:20 2021 +0900

    Add windows CI (#4)

commit 853886e2332104fe2bb02879a64a7339ecb1baae
Author: Luke Kim <[email protected]>
Date:   Tue Jul 6 20:27:16 2021 +0900

    Initial README with badges (#3)

commit 12158b3ac425b6058a9d427aa783ce24eb4ae2f6
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 6 20:17:47 2021 +0900

    Add initial CI (#2)

    * Add initial CI

    * Upload artifact

    * Add more os/arch combos

commit 63a3adb568dbf1b837518b7ae7e2ee9fa3717221
Merge: 4fdf25b 0182a39
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 6 20:00:02 2021 +0900

    Merge pull request #1 from spiceai/phillip/add-spiced

    Adding spiced starter

commit 0182a3988f37c397820981d57d19e4db3f531e1f
Author: Phillip LeBlanc <[email protected]>
Date:   Tue Jul 6 19:56:45 2021 +0900

    Adding spiced starter

commit 4fdf25b2dca7f099fce3dd3b6de2f631e77ddfb7
Author: Luke Kim <[email protected]>
Date:   Tue Jul 6 19:32:48 2021 +0900

    Initial commit

…gration tests directly querying checkpoint (#5232)

* WIP: Verify the checkpoints existence before shutting down runtime in `test_acceleration_duckdb_single_instance`

* allow concurrent integration runs & test attempt #1

* fix linter

* test #1

* test #2

* test #3

* test #4

* test #5

* test #6

* test #7

* test #8

* test #9

* test #10

* test #11

* test #12

* checkpoint verification for all tests containing direct checkpoint query

* test #1

* test #2

* test #3

* test #4

* test #5

* test #6

* test #78

* test #8

* test #9

* test #10

* test #11

* test #12

* test #13

* test #14

* test #15

* remove temporary change

* Update crates/runtime/tests/acceleration/checkpoint_duckdb.rs

Co-authored-by: Phillip LeBlanc <[email protected]>

* Update crates/runtime/tests/acceleration/checkpoint_sqlite.rs

Co-authored-by: Phillip LeBlanc <[email protected]>

* Update crates/runtime/tests/acceleration/checkpoint_postgres.rs

Co-authored-by: Phillip LeBlanc <[email protected]>

* Update crates/runtime/tests/acceleration/single_instance_duckdb.rs

Co-authored-by: Phillip LeBlanc <[email protected]>

* Update crates/runtime/tests/acceleration/mod.rs

Co-authored-by: Phillip LeBlanc <[email protected]>

* Remove the sleep after checkpoints validation

---------

Co-authored-by: Phillip LeBlanc <[email protected]>

…0b9)

Pulls 6 commits onto the previous pin (25ce3658), including the sqlite
acceleration data-correctness fixes that were failing the MySQL/PostgreSQL
sqlite-accelerated quickstart tests:

- fix(sqlite): round-trip Decimal columns losslessly
- fix(sqlite): serialize List/Struct columns as valid JSON text
- fix(sqlite): guard rows_to_arrow cast loop on empty results

spiceai-53 merged tip after fork PR #13 (Sync upstream spiceai-53).

…ity (#11278)

* Vortex Data Accelerator

* Fix imports

* Add back feature

* fix vortex

* fix build

* Remove memory

* Fix tests

* Add tests

* Update tests

* Fixes

* Update

* Fix

* Update tests

* Use StreamTable instead of ListingTable

* Update tests

* Use async buffered writes

* Update tests

* Works!

* Perf improvements

* Fix

* Add check for partition_by

* Fix memory leak

* fix lint issues

* fmt

* Improve benchmark tests

* fix lint

* Fix duplicate code.

* vendor vortex-datafusion

* fix

* finally clean lint

* Don't create dummy file, just specify the schema

* fix lint

* Property integrate vendored vortex-datafusion

* WIP

* Fix tests

* remove custom writing code

* WIP

* Only support append mode for Vortex

* Add additional validation

* Add List to vortex supported types

* Fix linting issues

* Remove memory mode test, not supported anymore.

* Refactor and enhance Vortex and DataFusion integration

- Updated `simplify_expr` to use default `SimplifyContext` with query execution start time.
- Changed `StreamingExec` to use `Arc<PlanProperties>` for thread safety.
- Improved expression conversion in `DefaultExpressionConvertor` for better handling of `CASE` expressions.
- Added error handling for unsupported scalar conversions in `TryToDataFusion`.
- Enhanced schema calculation in `calculate_physical_schema` to utilize `ArrowSession`.
- Introduced `FunctionSupport` and related utilities for managing function restrictions in query plans.
- Added compatibility functions for converting between source Arrow and standard Arrow record batches.
- Implemented query execution functions for Spice client to handle SQL queries with optional parameters.
- Updated various modules to improve code organization and maintainability.

* Drop optional geo (geodatafusion) spatial UDFs

geodatafusion 0.3 only supports DataFusion 52, so it pinned a second
datafusion 52.5.0 tree in the lock alongside 53.1.0. Remove the optional
`geo` feature, the geodatafusion dependency, and the `runtime.params.geo`
ST_* UDF registration + tests. The lock now resolves a single DataFusion
53.1.0 (verified via cargo metadata).

* docs: improve DataFusion upgrade issue template

Add a live PR-tracking table, upstream-vs-fork guidance, a dual-Arrow/duckdb
note, a 'cargo tree -d' duplicate-version check, and a DataFusion v53 breaking
change (ExecutionPlan::statistics -> partition_statistics). Fix the stale
branch-49 changelog link.

* Bump Vortex fork to upstream 0.74.0

Merge upstream vortex-data/vortex 0.74.0 into the Spice fork
(branch lukim/spiceai-53-vortex-0.74.0, rev a677d111). vortex-datafusion
compiles cleanly against 0.74; cayenne sees only deprecation warnings
(into_arrow_preferred -> execute_arrow) — no API-break errors from vortex.

* Fix DF53: flightsql ExecutionPlan::statistics -> partition_statistics

DF53 removed ExecutionPlan::statistics; partition_statistics already
implemented alongside. Removes the leftover override so data_components
compiles under the flightsql feature.

* Migrate cayenne Join::try_new to DF53 (null_aware 8th arg)

DF53 added null_aware: bool to datafusion_expr::Join::try_new. Fresh joins pass
false (DF52-equivalent); rebuild_inner_join preserves inner.null_aware. Covers
lib + test call sites.

* Fix data_components mongodb for table-providers 0.11.0 table_provider arity

* Upgrade duckdb-rs fork to arrow 58 (eliminate duckdb arrow-57 boundary)

Point duckdb at spiceai/duckdb-rs lukim/spiceai-58-arrow58 (DuckDB 1.5.2 + arrow 58).
duckdb now resolves arrow 58.3.0, matching the workspace.

* Upgrade to DataFusion 53.1.0 + Arrow 58.3.0

Upgrades the DataFusion/Arrow stack and forked dependencies in lockstep
(spiceai/spiceai#10280).

Forks (re-pointed via [patch.crates-io]):
- datafusion: lukim/spiceai-53.1-patches @ 06e9d850f (rebuilt onto 53.1.0;
  carries UDTF-arg-coercion fallback, UDTF cache-name #98, BigQuery FLOAT64
  unparser, vortex columnar shuffle)
- arrow-rs 58.3.0, delta-kernel 0.23, spicebench, tpchgen (arrow 58)
- federation 0.5.3, table-providers 0.11.0, ballista 53.0.0 (crates.io upstream)

Key migration fixes:
- object_store 0.13 (Path::child->join, ObjectStoreExt), sqlparser 0.61
- DF53 APIs: partition_statistics, QueryHandle::into_stream, RemoteTableRef,
  AccessMode via spiceai_duckdb, reqsign provide_credential
- rrf/vector_search UDTF table-name arg coercion fallback (search feature)
- partitioned_duckdb sink test: blocking_send -> send().await
- vector_table snapshots regenerated (DF53 dropped HashJoinExec custom accumulators)

NOTE: 2 cayenne explain-plan snapshots (cayenne_probe_join_uses_exact_accumulator)
need review -- DF53 removed the ExactLeftAccumulator the test exercised.

* Point spice2 at spiceai DF53 forks (ballista/federation/table-providers/iceberg) + restore iceberg DELETE

Re-point off crates.io/apache to the spiceai forks rebuilt on DataFusion 53:
- ballista-core/executor/scheduler -> spiceai/datafusion-ballista @e0a78666 (clean upstream 53.0.0)
- datafusion-federation -> spiceai/datafusion-federation @ac7bde41 (v0.5.3, DF53)
- datafusion-table-providers -> spiceai/datafusion-table-providers @15d4d41 (v0.11.0, DF53)
- iceberg/iceberg-* -> spiceai/iceberg-rust @0805971c (apache 28c4800f + 5 spice patches)

Restore iceberg DELETE FROM: re-enable IcebergDeleteExec::execute() (equality
delete files committed via RowDeltaAction), stubbed out when apache iceberg
0.9.0 dropped RowDeltaAction. Validated: cargo check -p runtime --features iceberg-write.

* Point object_store at spiceai/arrow-rs-object-store fork (S3 versioning)

spice2 now uses spiceai forks for all forkable deps. object_store ->
spiceai/arrow-rs-object-store @046f6d8 (apache f0a772cd + VersionId-in-list +
ListVersions; get_range_opts already upstreamed as apache #517).

* CI: validate table-providers commit against spiceai-53 (DF53 upgrade)

The datafusion-table-providers pin moved to the spiceai-53 branch (head 15d4d41)
for the DataFusion 53 upgrade; update validate_table_providers_commit.sh's BRANCH
env in pr.yml + pr-develop.yml from spiceai-52 so the check_changes gate passes.

* Fix full build for table-providers v0.11.0 API changes

The full spiced build (release,models,postgres-accel default feature set)
failed to compile data_components with 7 errors against table-providers
v0.11.0, even though `cargo check -p runtime` passed.

1. `merge_inferred_and_declared_schemas` was removed from
   `datafusion_table_providers::util::schema` in v0.11.0. Reimplement it
   (verbatim from the upstream version that shipped it) in the
   feature-agnostic `data_components::schema_discovery` module and point both
   consumers at it: dynamodb/provider.rs and runtime debezium.rs (the latter
   only surfaces under the `debezium` feature, which the full build enables).

2. The 6 `SqlTable<T, P>: SQLExecutor is not satisfied` errors were a
   duplicate-crate problem: table-providers v0.11.0 pulls crates.io
   datafusion-federation, while the rest of the workspace uses the spiceai
   fork. The two `SQLExecutor` traits are distinct, so table-providers'
   `impl SQLExecutor for SqlTable` (built against crates.io) did not satisfy
   the fork-typed bounds in data_components' snowflake federation wrapper.
   Add a `[patch.crates-io]` redirect so the whole graph shares one
   federation crate.

Verified: `cargo check -p data_components -p runtime` with the full SQL
connector + debezium feature set compiles cleanly (0 errors).

* Fix full-feature build: adbc arrow-58 unification + sqlparser TableAlias

Two latent migration errors only surfaced by the full feature set (not by
`cargo check -p runtime`):

1. ADBC/arrow-58 boundary (18 errors in datafusion-table-providers adbcconn.rs
   / adbc/write.rs): adbc_core 0.23.0 was locked to crates.io arrow-array/
   arrow-schema 57.3.1 while the workspace uses the spiceai arrow-rs fork
   (58.3.0), so the ADBC connection code mixed two incompatible RecordBatch/
   Schema/RecordBatchReader types (E0599 as_list, E0282, E0308, E0277, E0271).
   adbc_core 0.23.0 allows `arrow >=53.1.0, <59`, so it unifies cleanly on the
   58.3.0 fork; retarget its lock entries (adbc_core/adbc_driver_manager/
   adbc_ffi) from 57.3.1 to 58.3.0. The registry arrow-57 subtree remains only
   for the snowflake-api source-arrow compat shim (intended cross-version IPC
   bridge), so the lock change is minimal.

2. sqlparser 0.61 TableAlias gained an `explicit: bool` field; add
   `explicit: false` to the implicit alias built in test-framework's reference
   schema rewriter (matches the existing duckdb intermediate_index_cte.rs).

Verified compile/link (release profile) and --locked for every failing CI
build job: E2E spiced+spice (release,models,postgres-accel,tpc-extension);
integration test binary (postgres,mysql,delta_lake,duckdb,sqlite,spark,
databricks,dynamodb,oracle,kafka,iceberg-write,snapshots,mongodb); models
test binary (flightsql,models,metal,s3_vectors,kafka,duckdb,bedrock,
extended_tests); ADBC integration test target (adbc); testoperator
--all-features --locked.

* Fix SQLite/Turso accelerator pragma init under upgraded rusqlite

PRAGMA mmap_size=N / cache_size=N echo the applied value as a result row, and
the upgraded rusqlite driver rejects execute() on row-returning statements
("Execute returned results - did you mean to call query?"), which failed
SQLite accelerator initialization. Apply pragmas via query_arrow + drain
instead. Turso doesn't implement PRAGMA mmap_size, so only cache_size is tuned.

* Re-point datafusion-table-providers to datafusion-contrib (canonical repo)

table-providers spice branches live in upstream datafusion-contrib (spiceai-53/
spiceai-53-patches @ v0.11.0 = 15d4d41), not the spiceai org fork. Fixes the
check_changes gate (validate_table_providers_commit.sh checks datafusion-contrib).
Same commit, no code change.

* cayenne: adopt DF53 native hash-join dynamic-filter pushdown (replaces forked accumulator seam)

DataFusion 53 removed the forked generic hash-join accumulator seam
(CollectLeftAccumulator/ColumnBounds traits + recreate_with_accumulator).
Its native inner-join dynamic-filter pushdown now builds the probe filter
centrally: min/max bounds (statistics pruning) AND membership (InList within
hash_join_inlist_pushdown_max_size, else hash-table lookup) -- a superset of
the old ExactLeftAccumulator. The CayenneAccelerationExec scan already consumes
pushed dynamic filters via its filter-pushdown hooks, so the custom
CayenneJoinRewriter physical rule (inner-join only, like the native path) is
redundant.

- Delete runtime-datafusion join_accumulator module + bench + lib export.
- Delete CayenneJoinRewriter + its exact-join-filter config/helpers/tests/
  snapshots from cayenne optimizer_rules; keep CayenneDynamicFilterSharing and
  CayenneAntiJoinSortMergeRewriter. Remove now-dead IsCayenneAccelerationExec.
- Rewire runtime builders: drop the exact-accumulator budget clamp; the native
  InList budget (hash_join_inlist_pushdown_max_size) is still sized from
  runtime.query.memory_limit per partition. Accept the historical
  exact_join_filter/join_rewriter/exact_accumulator rule tokens as recognized
  no-ops so existing spicepods do not warn.
- Add regression tests asserting the native inner-join dynamic filter reaches
  the Cayenne scan, and that semi joins (native is inner-only) do not.

* Fix Unity Catalog vending constructors for delta-kernel 0.23

The 2410e63da3 trunk merge brought #11180's from_object_store/with_engine
(written against delta-kernel 0.18.2) into the 0.23-migrated file without
threading the parquet_object_store field — a silent semantic conflict that
broke compilation. with_engine now takes the store explicitly; the vended
store serves as both the engine store and the parquet read store.

* Add Post-Merge Verification section to the DataFusion upgrade issue template

After each fork version-branch PR merges, diff the new version branch against
the previous line and verify every Spice patch is PRESENT / UPSTREAMED /
consciously dropped (tip-tree symbol checks, not SHA reachability), then
re-pin Cargo.toml to the final merged commit.

* cayenne: real per-partition scan statistics for join-side selection

Cayenne's scan already produces exact per-file statistics (row counts +
column min/max, derived from Vortex file footers via collect_scan_file_
statistics) and DataFusion's FileScanConfig::partition_statistics(Some(i))
already surfaces them per partition. Two gaps kept that signal from reaching
the planner on scans that mutate visibility:

- The Int64Pk / KeyBased deletion-filter execs did not implement
  partition_statistics, so they fell back to the ExecutionPlan default
  (Statistics::new_unknown) -- discarding the child's real per-partition
  row counts and bounds entirely whenever any deletion vector was present.

Implement partition_statistics on both deletion-filter execs to pass the
child's per-partition statistics through, demoted to Inexact via to_inexact():

- Row / null counts become upper bounds (correct as Inexact).
- Column min/max stay valid -- deleting rows can only shrink a column's
  range, so the bounds remain conservative for zone/file pruning.
- Absent statistics (collect_stat=false, e.g. position-deletes pending)
  stay Absent through to_inexact() -> unknown input degrades to unknown
  output, never a fabricated value.

This is strictly more useful than Unknown for the new native hash-join
dynamic-filter pass (build/probe side selection), range pruning, and limit
pushdown, while never asserting a wrong *exact* count. Clean/frozen scans
keep their Exact stats: the deletion filter is only inserted when deletions
exist, and the CayenneAccelerationExec wrapper passes inner stats through
unchanged.

Tests: exact passthrough on clean partitions (scan + wrapper), Inexact
downgrade under Int64 and composite-PK deletions, min/max bound
preservation, and unknown-stays-unknown when file metadata is absent.

* vortex: document verified 0.74 stats-layout pruning chain in opener

Code-level verification that spice2's scan path engages Vortex 0.74's
FileStatsLayoutReader / zoned StatFn pruning, including the new native
hash-join dynamic InList filters. No behavior change -- the chain already
fires end to end; this records the trace where a future reader will look.

What fires (evidence):
- Pushed conjuncts are translated to a Vortex Expression and handed to
  ScanBuilder::with_some_filter (opener.rs). Inside Vortex the zoned layout
  reader rewrites that filter into a stats predicate via Expression::falsify
  (the StatFn mechanism) and skips whole zones whose min/max can't satisfy it
  via ZoneMap::prune (vortex-layout zoned/pruning.rs).
- InList is fully pushable: convert() lowers InListExpr to list_contains over
  a Vortex Scalar::list, and can_be_pushed_down accepts it when the value and
  all list elements are pushable (vortex convert/exprs.rs).
- Dynamic hash-join filters reach the layout reader: collect_vortex_pushdown_
  conjunct unwraps DynamicFilterPhysicalExpr via .current() at file-open time
  (open() returns a future polled when the file is actually scanned -- not a
  plan-build-time snapshot), and Vortex's PruningResult::mask() RE-DERIVES the
  zone mask whenever the dynamic expression's version advances. So a hash-join
  build side that populates after the scan starts still prunes zones; it is
  not snapshotted too early.
- VortexAccessPlan only adds a row Selection (e.g. position-delete exclusion);
  it does not replace the scan filter, so stats pruning still engages under
  position-delete scans.

What does not fire (by design, not a gap):
- Filters Vortex can't translate are dropped from the in-file predicate
  (skipped_dynamic) but still feed the coarser per-file FilePruner /
  PrunableStream, which is itself dynamic-aware.

No gap found; no code fix required.

* Re-pin all forked dependencies to their merged canonical branch tips

Post-merge verification complete across every fork PR; pins now reference
the merged spiceai-* branch tips (not personal/pre-merge branches):
datafusion 0b0be50f (spiceai-53-patches incl. #162 patch restorations),
ballista 3bb79af4 (spiceai-53 incl. #48 S3 shuffle restoration),
federation 3c051ece (spiceai-53 = upstream v0.5.3; patches tip follows #78),
iceberg e519b221, arrow 18a40370, duckdb eb4b02bf, delta-kernel 714d64fd,
object_store 55b8f3b3, table-providers 2826b94f (datafusion-contrib),
spice-rs 7bc9e455 (trunk: Arrow 58 SDK + query bindings).
adbc arrow-58 lock edges preserved per the known snowflake-api arrow-57
subtree interaction.

* runtime: document IPC StreamWriter reuse in encode_batches_to_ipc + tests

Audit of encode_batches_to_ipc for the Arrow 58.1/58.2 cheap-StreamWriter
hot path: the implementation already constructs a single StreamWriter once
and reuses it across every batch in the stream (the dictionary-tracking and,
where enabled, compression context live on the writer). It does NOT create a
writer or codec per batch, so no restructuring is needed.

Document the invariant with a code comment so the reuse is not accidentally
regressed (moving the writer construction into the loop would re-emit the
schema/dictionaries and rebuild the codec per batch). Add two tests:
- empty input yields an empty buffer;
- a multi-batch encode round-trips through a single StreamReader back into
  every original batch in order, which exercises the multi-batch path and
  guards the single-writer/single-stream property.

Verification-only change; no behavior change to the encode path.

* cayenne: migrate delta_encoding to the vortex 0.74 Scheme API

PR #11199 was written against the pre-0.74 btrblocks compressor (FloatCode/
IntCode/StringCode + include_* builder methods, removed in 0.74). Levels now
map to explicit Scheme sets built from empty(); levels 4-6 exclude FSST via
SchemeId; compression_strategy=zstd registers schemes::string::ZstdScheme
(vortex-btrblocks 'zstd' feature). Level semantics and the spicepod parameter
surface are unchanged.

* data_components: delta-kernel 0.23 removed DefaultEngine::new — use the builder

* cluster: restore spice-ballista functionality stripped by the clean-upstream adaptation

The upgrade temporarily adapted runtime/cluster to clean upstream ballista 53
(thin metrics trait, no task cancellation, no readiness signalling, renamed
executor-state APIs). With the spice patch set restored on spiceai-53
(PRs #47/#48), trunk's implementations are correct again: full
SchedulerMetricsCollector (executor task gauges, durations, shuffle
read/write metrics), task-cancellation plumbing, executor readiness, and the
fork's get_executor_state surface. heartbeat.rs keeps the object_store-0.13
ObjectStoreExt import; service.rs keeps the IPC writer-reuse work; the
trunk-merged CountingExec mock and cancel_tasks adapt to Arc<PlanProperties>
and QueryHandle::into_stream.

* Pin datafusion-federation to the merged spiceai-53-patches tip (#78)

v0.5.3 + the EXISTS/NOT-EXISTS and DML federation ports — the two patches
the v0.5.3 rewrite did not cover. Completes the dependency-PR cycle: all 16
fork PRs merged; every pin now references a merged canonical branch tip.

* cluster: adapt PartitionResultStream to the fork's BallistaClient surface (path-based fetch_partition)

* fix lint on the DF53 upgrade additions

cargo fmt across the merge-resolution edits; let-else conversions in the
test-framework polling loops (single_match_else); doc-markdown backticks
(federation wrapper header, turso/builder/optimizer_rules docs); expect
instead of unwrap in the filter_exec stats test; drop the unused duckdb
deny-list import. make lint-rust is green.

* test(sqlite-accel): Decimal round-trip regression test

Adds test_sqlite_decimal_round_trip covering the SQLite accelerator
Decimal path that the 'acceleration: sqlite' MySQL/Postgres quickstart
E2E jobs exercise. It accelerates a Decimal128(10,2) column (mirroring
the quickstart col_Decimal: 1.11, NULL, 99.99, 0.00, 2.00) and a
high-precision Decimal256(40,4) column, then reads them back.

Before the table-providers SQLite Decimal fix this reproduced both the
DF53-branch failures: a sea-query 'precision cannot be larger than 16'
panic (Decimal256) and the read-back error 'Invalid column type Text at
index ... name: col_Decimal' (Decimal128). Requires the matching
datafusion-table-providers fix to be pinned to pass.

* Re-pin table-providers to spiceai-53-patches 17084c8 (sqlite Decimal fix)

The v0.11 line lost the spiceai/sea-query fork's decimal(p,s) SQLite
rendering (crates.io sea-query emits real(p,s) — REAL affinity, panics at
precision>16) and rows_to_arrow locked one storage class per column. The
patches branch now emits decimal(p,s), binds Decimal as canonical BigDecimal
text, and decodes string columns tolerantly across storage classes. Fixes
the mysql/postgres quickstart (acceleration: sqlite) E2E failures; covered
by test_sqlite_decimal_round_trip.

* test(sqlite-accel): List<Utf8> round-trip regression test

Mirrors the Postgres/MySQL quickstart E2E failure where a `List<Utf8>`
column accelerated into SQLite read back with `Failed to decode value:
Json error: Encountered unexpected 'e' whilst parsing value`.

SQLite has no array storage class, so the list is persisted as TEXT and
the accelerated read path JSON-parses that text into the declared
`List<Utf8>`. The write side must emit valid JSON (`["expired","active"]`),
which the table-providers SQLite fix now does. The list values include
strings with spaces and embedded quotes to exercise JSON quoting/escaping.

Inserts via the accelerator and scans back through the federation JSON
cast, asserting all rows round-trip.

* Re-pin table-providers to 3311cd72: List/Struct cells written as valid JSON

v0.11's insert path rendered List/Struct cells via ArrayFormatter (bare
tokens, not JSON), so the accelerated read-back's federation schema-cast
JSON decode failed. Cells now serialize through arrow-json make_encoder —
the exact inverse of the read-side decoder. Covers the remaining
postgres/mysql quickstart (acceleration: sqlite) failures; guarded by
test_sqlite_list_round_trip.

* cargo fmt on the sqlite accel regression tests

* docs: backtick SQLite in the accel regression test docs (doc_markdown)

* fix(df53): nsql snapshots → 53.1.0 and test-framework AS-alias under sqlparser 0.61

Build and Test had exactly 8 unit failures from the upgrade:
- 4 nsql snapshots embedded the live DATAFUSION_VERSION (52.5.0); the
  DataFusion 53.1 bump renders 53.1.0. Update all 4 snapshots.
- 4 test-framework query-rewrite tests: sqlparser 0.61's new
  TableAlias.explicit field was false, rendering 'customer customer'
  (no AS); the rewriter's own doc comment + tests expect
  'customer AS customer'. Set explicit: true.

Validated: cargo test -p test-framework --lib test_rewrite_ → 16 passed.

* fix(df53): accept full nsql snapshots (expanded Spark function inventory)

My earlier snapshot commit only bumped the DataFusion version string; the
upgrade also expanded the Spark-compatibility scalar-function inventory
(adds json_tuple, add_months, array_contains, base64, date_diff,
substring, to_utc_timestamp, etc.), so the snapshots still mismatched.
Accept the full rendered output.

Validated: cargo test -p runtime --lib nsql_context_ → 7 passed.

* fix(df53): relabel Delta column-mapping nested fields without a struct cast

DataFusion 53 rejects a struct->struct CastExpr when the source and target
fields share no names (Cannot cast struct with N fields ... no field name
overlap). Delta column mapping renamed physical->logical nested struct/list/map
field names via exactly such a cast, breaking
query_delta_lake_column_mapping_mode_{id,name}.

Replace the CastExpr with a RelabelFieldsExpr that recursively rebuilds the
array's DataType (metadata-only field rename) without touching values, buffers,
or null masks.

Validated: cargo test -p runtime --test integration --features delta_lake
delta_lake::query_delta_lake_column_mapping -> 3 passed; 0 failed.

* deps: re-pin table-providers (spiceai fork 51c56f8a), snowflake-api (946bdbc), drop object_store patch

- datafusion-table-providers -> spiceai/datafusion-table-providers @ 51c56f8a
  (lukim/tp-53-patches-df53-seam): adds DuckDB with_function_support deny-list
  seam, Engine unparser model, util/supported_functions, sqlite between.
- snowflake-api -> spiceai/snowflake-rs @ 946bdbc (arrow58 + object_store 0.13).
- Removed [patch.crates-io] object_store fork entry; object_store now resolves
  to stock crates.io 0.13.2 (single version, no holdout).

* deps(df53): re-pin table-providers (spiceai-53) + snowflake (spiceai-58), unify Arrow 58 + DuckDB, restore #10703 deny-list

Re-pin to the merged canonical fork branches:
- datafusion-table-providers -> spiceai-53 @ 6d01d73 (with_function_support seam
  on both DuckDB factories, Engine unparser model, sqlite BETWEEN/decimal_cmp +
  decimal(p,s) NUMERIC affinity)
- snowflake-api -> spiceai-58 @ d5151079 (arrow 58 + object_store 0.13)

Alignment + consequences:
- Drop the object_store [patch.crates-io]; unify on stock object_store 0.13.2
  (snowflake-api no longer pins the 0.12 line).
- Align everything to Arrow 58: bump the source-arrow/source-arrow-ipc aliases
  57->58 and remove the now-identity source_arrow_compat shim (snowflake-api is
  Arrow 58).
- Retire the spiceai_duckdb_fork (1.4.4) alias; runtime + table-providers share
  one duckdb (spiceai/duckdb-rs eb4b02bf, DuckDB 1.5.2 + Arrow 58).
- Un-no-op connector-duckdb with_spice_deny_list + install the deny-list on the
  DuckDB accelerator factory, so Spice-only UDFs (json_get_str, ...) evaluate
  locally instead of being pushed into DuckDB (fixes #10703).
- Update DuckDB json-function EXPLAIN snapshots (DF53 column widths) + the sqlite
  decimal plan snapshot (redundant Projection elided).

Validated: duckdb_json_functions, duckdb_connector_does_not_push_down_spice_functions,
sqlite::test_sqlite_decimal_memory -> 3 passed.

* fix(df53): Round-1 CI (snowflake ObjectStoreExt, TP guard, advisory) + #10931 typed S3 timeout

Three Round-1 CI failures, plus the #10931 integration:

- Build Test Binary: object_store 0.13 moved get/put off `ObjectStore` onto a new
  `ObjectStoreExt` trait; snowflake-rs spiceai-58 (PR #8) didn't import it, so
  snowflake-api failed to compile (E0599 no method get/put). Re-pin snowflake-api
  to a9828507 (spiceai/snowflake-rs PR #9 adds the import).
- check_changes: validate_table_providers_commit.sh expected the pin on
  datafusion-contrib, but the complete DF53 work lives on the Spice org fork
  (spiceai/datafusion-table-providers @ 6d01d73). Point the guard at the org fork,
  aligning table-providers with the other Spice forks (snowflake-rs, duckdb-rs).
- Check Rust Licenses: ignore RUSTSEC-2026-0173 (proc-macro-error2 unmaintained;
  transitive via mysql_async; verified the sole new failing advisory).

- #10931 (improve S3 response-body-decode error): implement the typed approach
  Phillip asked for, now possible since object_store is 0.13 — walk the error
  source chain and downcast to object_store::client::HttpError, surfacing
  HttpErrorKind::Timeout with an actionable client_timeout message. No string
  matching, no fork backport. + 3 unit tests (compile-verified).

* style: cargo fmt --all (import ordering after duckdb unification + s3 #10931)

* deps(odbc): arrow-odbc 21->22.1.1 (arrow 58) — eliminate arrow-57 subtree + drop IPC bridge

arrow-odbc 21 capped arrow at '<58', forcing a separate Arrow-57 subtree that
adbc_core (which accepts '>=53,<59') was unified onto. That made the
table-providers ADBC connection (adbcconn.rs) mix Arrow 57 with the workspace
Arrow 58 (E0599 as_list, Arc<Schema>: From<Schema>, RecordBatchReader
unsatisfied) — failing Rust Lint (--features adbc) and ADBC Integration Tests.

arrow-odbc 22+ accepts arrow '>=29,<59', so it resolves to the workspace Arrow 58.
Switching the pin (git pacman82@7316b13 -> crates.io 22) drops the Arrow-57
subtree entirely (arrow/arrow-array/.../arrow-ipc 57.3.1 all removed) and lets
adbc_core resolve to Arrow 58 naturally — no lockfile split needed.

With arrow-odbc on the workspace Arrow, the source-arrow-ipc 57->58 IPC bridge is
redundant: odbcconn's odbc_schema_to_arrow / odbc_record_batch_to_arrow become
pass-throughs (Arrow clones are shallow Arc bumps), and the source-arrow-ipc dep
plus the now-dead IoCursor import and ODBCArrow{Serialize,Deserialize} error
variants are removed.

Verified: cargo check -p data_components --features adbc is clean (adbcconn.rs on
Arrow 58).

* fix(odbc): drop Result from no-op bridge fns (clippy::unnecessary_wraps) + canonicalize snowflake pin

- odbc_schema_to_arrow / odbc_record_batch_to_arrow became pass-through clones
  after the arrow-odbc 22 bump, so the Result wrapper tripped
  clippy::unnecessary_wraps (-Dpedantic) in Rust Lint. Return the value directly
  and simplify the 3 call sites. (cargo check didn't catch it; clippy does.)
- Re-pin snowflake-api from my personal branch commit -> the merged spiceai-58
  tip 94bd751f (PR #9), so the dependency tracks the canonical fork branch.

Verified: cargo clippy -p db_connection_pool --features odbc --tests with CI's
pedantic/deny flags is clean.

* deps(round2): rusqlite 0.37->0.40.1 + Rust toolchain/MSRV 1.94.1->1.95.0

rusqlite 0.40 (libsqlite3-sys 0.38) needs Rust >=1.95 (cfg_select! in its
build.rs), so these land together:

- rusqlite -> 0.40.1 (spiceai/rusqlite e39c9c46, bundled-decimal) + tokio-rusqlite
  -> spiceai/tokio-rusqlite b10df82e (raises the rusqlite req to allow 0.40).
- datafusion-table-providers -> spiceai-53 5eb7c5d3 (PR #7 rusqlite-0.40 bump +
  PR #8 UInt64-read-as-i64: rusqlite 0.40 dropped the u64 FromSql impl).
- Rust toolchain + workspace MSRV -> 1.95.0; CI/Docker/codegen refs bumped to match.

Verified: cargo test --no-run -p runtime --test integration (postgres,mysql,
delta_lake,duckdb,sqlite,spark,databricks,dynamodb,oracle,kafka,iceberg-write,
snapshots,mongodb) compiles clean under Rust 1.95.0 + rusqlite 0.40.1.

* style: clippy --fix for Rust 1.95 pedantic lints

Rust 1.95's clippy enforces lints 1.94.1 did not (map_unwrap_or,
unnecessary_trailing_comma, duration_suboptimal_units, useless_conversion).
Machine-applicable auto-fixes across the workspace + tools: `.map(f).unwrap_or(g)`
-> `.map_or(g, f)`, drop redundant `.into_iter()`, larger Duration units, trailing
commas. Behavior-preserving.

* style: clippy --fix for Rust 1.95 (feature-gated tail)

Final 1.95 clippy lint in feature-gated code (search/s3_vectors) not reached by the
reduced-feature pass; full-feature clippy (adbc,models,odbc,mcp,release,spicebench,...)
is now clean under Rust 1.95.

* fixes

* clippy

* move everything to 1.95.0

* update 'datafusion-table-providers'

* fix merge

* clippy

* deps: re-pin datafusion-table-providers to sqlite decimal fix

Pin to spiceai/datafusion-table-providers@2487974 (PR #10) which
decodes decimal columns as Utf8 via get_ref() to handle mixed SQLite
storage classes (INTEGER/REAL/TEXT), then casts to the target Decimal type.

* clippy: fix new Rust 1.95 pedantic lints

- duration-suboptimal-units: from_secs(60) → from_mins(1), etc.
- useless-conversion: remove redundant .into_iter()
- map-unwrap-or: .map(f).unwrap_or(x) → .map_or(x, f)
- unnecessary-trailing-comma: remove trailing commas in macros

* Fix formatting

* Update datafusion-table-providers to spiceai-53 merged tip

PR spiceai/datafusion-table-providers#10 is now merged to spiceai-53 branch.
Update the commit reference from the pending PR tip to the actual merged tip.

* Update

* fix: re-pin table-providers (sqlite OOB-cast guard, #12) + 1.95 clippy

- datafusion-table-providers -> spiceai-53 25ce3658 (PR #12): guards the sqlite
  decimal-cast loop against an out-of-bounds index that panicked on the refresh
  SELECT MAX(ts) probe -> fixes the sqlite-accelerated MySQL/PostgreSQL quickstarts.
- clippy --fix for the one Rust 1.95 pedantic lint the trunk reconcile re-introduced
  (cayenne provider/table.rs).

* chore(deny): ignore RUSTSEC-2026-0174 (headers ASCII-invariant notice); dedupe RUSTSEC-2026-0173

* fix(cayenne): adapt merged trunk code to object_store 0.13 + DF53 fork ExecutionPlan API

The trunk merge brought new cayenne code written against object_store 0.12
and upstream DataFusion. Adapt it to this branch's pinned deps:

- Path::child is deprecated in object_store 0.13 (-> .join()). Use
  prefix.clone().join() for borrowed/reused prefixes; consume via .join()
  where the receiver is final (avoids clippy::redundant_clone).
- StatsOverrideExec test mock: the Spice DF53 fork's ExecutionPlan::properties
  returns &Arc<PlanProperties> (upstream returns &PlanProperties). Arc-wrap the
  field and return type; use Arc::clone to satisfy clippy::clone_on_ref_ptr.

* chore(deps): bump datafusion-table-providers to spiceai-53 tip (0bf2a0b9)

Pulls 6 commits onto the previous pin (25ce3658), including the sqlite
acceleration data-correctness fixes that were failing the MySQL/PostgreSQL
sqlite-accelerated quickstart tests:

- fix(sqlite): round-trip Decimal columns losslessly
- fix(sqlite): serialize List/Struct columns as valid JSON text
- fix(sqlite): guard rows_to_arrow cast loop on empty results

spiceai-53 merged tip after fork PR #13 (Sync upstream spiceai-53).

* feat(cayenne): listing-time pruning, per-file stats, and scan pushdown

Implement Cayenne Vortex audit priorities 1–6:

1. Drop non-overlapping Vortex files at listing time via footer min/max.
2. Persist per-file statistics in the metastore for reuse across scans.
3. Prune inline memtable and RAM-tier segments using batch min/max bounds.
4. Broaden PK selective-scan heuristics (small IN/BETWEEN → target_partitions=1).
5. Return Inexact table statistics during pending deletes instead of None.
6. Push sparse Int64 key deletes into Vortex scan filters as NOT IN predicates.

Adds file_pruning helpers, metastore DDL for cayenne_snapshot_file_statistics,
and an integration test asserting listing-time file pruning reduces files_scanned.

* Revert "feat(cayenne): listing-time pruning, per-file stats, and scan pushdown"

This reverts commit 08125e53d70281d5ab62cfa94472a5abef662c86.

* feat(cayenne): listing-time pruning, per-file stats, and scan pushdown (#11234)

* feat(cayenne): listing-time pruning, per-file stats, and scan pushdown

Implement Cayenne Vortex audit priorities 1–6:

1. Drop non-overlapping Vortex files at listing time via footer min/max.
2. Persist per-file statistics in the metastore for reuse across scans.
3. Prune inline memtable and RAM-tier segments using batch min/max bounds.
4. Broaden PK selective-scan heuristics (small IN/BETWEEN → target_partitions=1).
5. Return Inexact table statistics during pending deletes instead of None.
6. Push sparse Int64 key deletes into Vortex scan filters as NOT IN predicates.

Adds file_pruning helpers, metastore DDL for cayenne_snapshot_file_statistics,
and an integration test asserting listing-time file pruning reduces files_scanned.

* refactor(cayenne): clean up import statements and formatting across multiple files

---------

Co-authored-by: Luke Kim <Luke Kim [email protected]>

* fix(cayenne): satisfy strict clippy on merged pruning code (#11234)

PR #11234's listing-time pruning merged without the workspace's strict
-Dclippy::pedantic lint pass. Conform it (all behavior-preserving):

file_pruning.rs: should_prune_statistics returns bool (never errors —
unnecessary_wraps), dropping ? at call sites and .expect() in tests;
merge_min/merge_max take Precision by reference (needless_pass_by_value)
and collapse identical match arms (match_same_arms); doc_markdown wording.

table.rs: collapse Exact|Inexact num_rows arms (match_same_arms); backtick
ListingTable in a doc comment.

file_pruning_test.rs: Arc::clone the Arc<CayenneCatalog> fixture
(clone_on_ref_ptr), let-bound so it coerces to Arc<dyn MetadataCatalog>,
dropping the now-unused import; hoist the TestAdvancer mock to the test fn
top (items_after_statements).

* Update dependencies

* feat(cayenne): auto-apply inferred sort for CDC tables so compaction clusters for pruning (#11236)

Split the refresh_mode:changes sort gate in extended schema inference by engine. DuckDB/Arrow stay disabled (their inferred sort is a refresh-time sort, a no-op for a change stream). Cayenne is exempt: cayenne_sort_columns sorts the background COMPACTION rewrite, not the change stream, so applying the inferred sort key in changes mode makes the per-file min/max stats (added in #11234) tight and non-overlapping. Listing-time pruning then skips files and collapses the heavy-table read-amp (file fan-out). Lag-neutral (compaction-time only).

Tests: cayenne+changes now sets cayenne_sort_columns; duckdb+changes stays a no-op (test renamed to ..._on_refresh_time_engines).

* Update

* perf(cayenne): fix in-memory CDC durability — bounded checkpoint + O(1) tombstone append (#11232)

* ci(nightly): build spiceai-nightly with the spicebench feature (#11225)

Add the `spicebench` cargo feature to the nightly binary builds (linux
x64, linux aarch64, macos aarch64) so the published nightly CLI binaries
and the ghcr.io/spiceai/spiceai-nightly Docker image include it.

The Docker image jobs reuse the pre-built spiced binary, so enabling the
feature at the cargo build step is sufficient to ship it in the image.

* fix(cluster): distribute accelerated tables wrapped by metadata/index providers (#11226)

The coordinator's partition-distribution rewrite gates on
AcceleratedPartitionProvider::should_partition, which only recognized a bare
AcceleratedTable or FederatedTableProviderAdaptor(AcceleratedTable). Datasets
with column/table metadata (descriptions) get registered wrapped in a
MetadataEnrichedTableProvider, and embedding/full-text-search columns add
IndexedTableProvider/EmbeddingTable wrappers. should_partition then returned
false, so the coordinator skipped distribution and silently federated reads to
the source instead of fanning out to the executors holding the accelerated
partitions.

Route recognition through find_concrete_table_provider, which already unwraps
all of these decorators, so any provider chain containing an AcceleratedTable is
distributed correctly.

Adds hermetic cluster regression tests (column-metadata + federated source) that
run on a local-filesystem cluster state.

* WIP: in-memory CDC fix (A+B) — cayenne lib 497 green; runtime-compile + A1-T3 pending

* Merge develop to trunk — 2026-06-08 (#11215)

* fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192)

* fix(partition): handle DataType::LargeUtf8 in parse_partition_value

`partition_key_to_string` already maps `Utf8 | LargeUtf8 | Utf8View`, but the
reverse `parse_partition_value` only handled `Utf8` and `Utf8View`. Cayenne
represents strings as `LargeUtf8`, so inferring existing partitions on restart
reads the partition value back as `LargeUtf8` and fails with `Unsupported scalar
value type: LargeUtf8` — which forces users to drop `partition_by` on string
columns.

Add the missing `LargeUtf8` arm so `partition_by` on a string column survives a
runtime restart, plus a round-trip regression test.

* fix(kafka): decode JSON payloads directly to Arrow, skip lossy Value round-trip

* build(deps): bump the aws-sdk group with 2 updates (#11210)

Bumps the aws-sdk group with 2 updates: [aws-sdk-dynamodb](https://github.com/awslabs/aws-sdk-rust) and [aws-sdk-glue](https://github.com/awslabs/aws-sdk-rust).


Updates `aws-sdk-dynamodb` from 1.114.0 to 1.115.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-sdk-glue` from 1.148.0 to 1.149.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

---
updated-dependencies:
- dependency-name: aws-sdk-dynamodb
  dependency-version: 1.115.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: aws-sdk
- dependency-name: aws-sdk-glue
  dependency-version: 1.149.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: aws-sdk
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump the github-actions-dependencies group with 2 updates (#11209)

Bumps the github-actions-dependencies group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [github/codeql-action](https://github.com/github/codeql-action).


Updates `actions/checkout` from 6.0.2 to 6.0.3
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/de0fac2e4500dabe0009e67214ff5f5447ce83dd...df4cb1c069e1874edd31b4311f1884172cec0e10)

Updates `github/codeql-action` from 4.36.0 to 4.36.2
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/7211b7c8077ea37d8641b6271f6a365a22a5fbfa...8aad20d150bbac5944a9f9d289da16a4b0d87c1e)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 6.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
- dependency-name: github/codeql-action
  dependency-version: 4.36.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump nvidia/cuda in the docker-dependencies group (#11208)

Bumps the docker-dependencies group with 1 update: nvidia/cuda.


Updates `nvidia/cuda` from 13.2.1-cudnn-runtime-ubuntu24.04 to 13.3.0-cudnn-runtime-ubuntu24.04

---
updated-dependencies:
- dependency-name: nvidia/cuda
  dependency-version: 13.3.0-cudnn-runtime-ubuntu24.04
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: docker-dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ci: upgrade dorny/paths-filter to v4.0.1 and configure Dependabot for composite actions (#11134)

* ci: upgrade dorny/paths-filter to v4.0.1 and configure Dependabot for composite actions

- Upgrade the pinned dorny/paths-filter reference in the check-code-changes
  composite action from v3.0.2 (de90cc6) to latest v4.0.1 (fbd0ab8).
  v4 updates the action to Node 24 runtime (breaking per upstream) and
  adds merge queue support + other improvements.
- Update .github/dependabot.yml to use the 'directories' list for the
  github-actions ecosystem. This ensures Dependabot scans not only
  .github/workflows/ but also the action.yml files inside composite
  actions under .github/actions/*/ (the root '/' entry does not cover
  these by default).
- Added explicit directories for check-code-changes and other composites
  that reference third-party actions (setup-*, build-*, aws-*).

This addresses why Dependabot was not proposing updates for the
paths-filter action (and similar cases in composites).

See: https://github.com/dependabot/dependabot-core/issues/6704

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* security: avoid expression injection risk with untrusted branch names in e2e install test workflow

- github.head_ref and github.ref_name (from pull_request events, including
  forks) were interpolated directly into shell run: scripts to build a
  curl URL for the install script.
- This is a script injection vector (expressions are expanded into the
  executed script source before the shell runs).
- Fix: use step 'env:' for the (trusted, when present) dispatch input,
  and for PR/push-triggered runs use the already-checked-out local
  ./install/install.sh via bash -c "$(cat ...)" (which is the version
  under test anyway).
- This also makes the test work correctly for PRs from forks.
- Added reference to GitHub security hardening docs.

Part of audit for branch handling (target/source refs) in workflows that
can be triggered by untrusted input (related to recent GHA supply-chain
and injection attacks).

* docs: tweak comment on dependabot directories glob for composites

* fix(ci): use pipe invocation for local install.sh in E2E release install test

Use 'cat ./install/install.sh | /bin/bash' instead of 'bash -c "$(cat ...)"'
for the PR/push 'from branch' case.

The -c + command-sub form causes shell quoting breakage (and potential
partial execution of script content) in the step's runner script because
install.sh contains many double-quotes and other metacharacters.

Piping the content matches the 'curl | /bin/bash' style used for the
remote case, ensures [ -t 0 ] is false (non-interactive path in script),
and keeps the security property (no untrusted branch name interpolated
into the executed command line).

This should unblock the E2E install checks for PRs that touch install/.

* fix(ci): normalize line endings when running local install.sh under WSL in E2E test

Windows checkout can produce CRLF for shell scripts (autocrlf). When the
WSL job mounts the workspace, the install.sh has CRLF, and 'cat ... | /bin/bash'
(or previous -c) inside wsl-bash can fail to parse the script.

Pipe through 'tr -d \r' (harmless on LF files) to ensure Unix line endings
for the piped bash invocation. This only affects the local/PR case in the
WSL job; the explicit-branch dispatch case and prod curl get LF from network.

This explains why only the WSL job was failing on PRs that trigger the
install E2E (including this one, via the workflow yml path change).

* fix(ci): use safe remote raw URL (head repo + sha) for 'from branch' local case in E2E install test

Instead of cat local file (which had CRLF issues under WSL from windows checkout),
construct a curl to raw.githubusercontent.com using:
- HEAD_REPO = pull_request.head.repo.full_name (or current repo)  -- supports fork PRs
- HEAD_SHA  = pull_request.head.sha (or current sha) -- always safe characters, no branch-name injection risk

This:
- Tests the exact commit's version of install/install.sh (what the PR actually proposes)
- Gets guaranteed LF content (git blob via raw)
- Avoids any local FS line-ending or mount quirks
- Keeps the 'if INSTALL_SCRIPT_BRANCH' for explicit dispatch (still curls main repo)
- Values are provided via step 'env:' then  inside ... (standard safe pattern)

Removes the previous tr -d workaround.

This should make the WSL (and linux) E2E install checks pass reliably for PRs that touch install/ paths or the workflow itself.

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* Refactor: add explicit ParameterSpec for embedding components (#10853)

* refactor: add explicit ParameterSpec for embedding components

- Create crates/runtime/src/embeddings/params/ with per-provider
  ParameterSpec definitions for all 8 embedding providers
  (openai, azure, google, huggingface, databricks, bedrock, file, model2vec)
- Update try_to_embedding to use Parameters::try_new with explicit specs
- Update all provider function signatures from HashMap to Parameters
- Remove unused extract_secret macro from embed.rs

Closes #6755

* fix: update google error message to reflect new param key

* style: apply rustfmt formatting

* fix: correct ParamLookup API usage for models and bedrock features

* test: add unit tests for embedding ParameterSpec definitions

* fix: resolve clippy::redundant-closure-for-method-calls in embed.rs

Replace closures `|s| s.parse::<T>()` with `str::parse::<T>` at 4 sites
in crates/runtime/src/model/embed.rs to satisfy clippy's pedantic lint.

---------

Co-authored-by: claudespice <[email protected]>

* fix(turso): gate schema changes during writes (#10940)

Co-authored-by: webhop123 <[email protected]>
Co-authored-by: claudespice <[email protected]>

* fix(ci): forward install env vars into WSL via WSLENV in E2E release install test

The E2E Windows job runs install.sh under wsl-bash, but WSL does not inherit
Windows-side env vars by default and runs with 'set -u', so the
INSTALL_SCRIPT_BRANCH/HEAD_REPO/HEAD_SHA vars introduced in #11134 were unbound
inside WSL — failing with 'INSTALL_SCRIPT_BRANCH: unbound variable'. The WSL
job path-gates and never ran on #11134's own PR, so the regression shipped
untested.

Forward the three vars across the interop boundary with WSLENV and default the
branch test to empty so it is robust to a missing var.

* chore(deny): ignore RUSTSEC-2026-0173 (proc-macro-error2 unmaintained via mysql_async)

* fix: apply retention_sql after Arrow refresh writes (#11214)

* refactor: extract retention filter application helper

* fix: apply retention sql after arrow refresh writes

* test: add arrow retention write path coverage

* fix: surface arrow retention write failures

* fix: invalidate cache after committed retention failure

* fix(runtime): resolve clippy pedantic errors in accelerated_table

- Add backticks to PartitionedArrow and DuckDB in refresh.rs doc comment
  (clippy::doc_markdown)
- Drop redundant continue and unnecessary semicolon in retention.rs match
  (clippy::needless_continue, clippy::unnecessary_semicolon)

Fixes the Rust Lint CI failure on the develop->trunk merge PR #11215.

* fix(runtime): satisfy clippy 1.94 pedantic in retention check (needless_continue, unnecessary_semicolon)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: melks <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Luke Kim <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Manish Sharma <[email protected]>
Co-authored-by: Yassine Elyaakoub <[email protected]>
Co-authored-by: webhop123 <[email protected]>
Co-authored-by: Hanbo Wang <[email protected]>

* Update openapi.json (#10918)

Co-authored-by: Spice Schema Bot <[email protected]>
Co-authored-by: claudespice <[email protected]>

* perf(cayenne): finish in-memory CDC durability fix (runtime wiring, O(1) append, periodic checkpoint)

Completes the bounded in-memory CDC tier (`cdc_durability: memory`). Previously
the RAM tier had no mid-run checkpoint and the per-append tombstone-map clone was
O(tier-size), so under sustained ingest the tier grew unbounded — replication lag
and scan starvation. The fix bounds the tier on three axes while preserving the
exactly-once contract (the deferred source slot ack advances only after the
covering checkpoint's Vortex + metastore writes are durable).

- A1: per-table background mem-tier checkpoint task (mirrors the background
  compactor) flushes the RAM tier every `cdc_mem_tier_checkpoint_interval_ms`
  (default 1s) and advances the deferred slot ack on idle / pure-upsert streams
  that never trip a write-path cap or a delete/truncate event trigger. Wired into
  the accelerator next to the compaction spawn (a no-op for file-mode and
  partitioned tables).
- A2: non-zero default caps (`cdc_mem_tier_max_bytes` 64 MiB,
  `cdc_mem_tier_max_age_ms` 2s) so the write path self-spills while the tier is
  still small.
- B: tombstone maps are now structurally-shared persistent maps (`im::HashMap`
  HAMT, XXH3 hasher) so an append shares the corpus root — per-append cost drops
  from O(tier) to O(incoming . log tier); the listing-fence section stays O(1).

Fixes the runtime compile (the new parameter grew the `concat_arrays` length) and
adds the test set: crash-mid-tier exactly-once via the periodic path, idle-source
slot-ack-advances, write-path byte/age cap self-fire, defaults-non-zero, the
append-shares-tombstone-map guard, a bounded-tier (no-O(N^2)) structural-sharing
check, and the runtime checkpoint-vs-push ordering test (a checkpoint racing ahead
of the push only delays an ack, never advances one early or double-acks).

cargo check (cayenne+runtime), cargo test -p cayenne --lib (498 passing), the
runtime ordering test, clippy (both CI gates), and fmt are green.

* refactor(cayenne): remove no-op background memory tier checkpoint calls
chore(deny): update ignored RUSTSEC advisories for unmaintained dependencies

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Phillip LeBlanc <[email protected]>
Co-authored-by: claudespice <[email protected]>
Co-authored-by: melks <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Manish Sharma <[email protected]>
Co-authored-by: Yassine Elyaakoub <[email protected]>
Co-authored-by: webhop123 <[email protected]>
Co-authored-by: Hanbo Wang <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Spice Schema Bot <[email protected]>

* feat(cayenne): cayenne_scan_files_listed/pruned read-amp metric (files considered vs footer-pruned per scan)

* fix(deps): re-pin datafusion-federation/table-providers/duckdb to sgrebnov DF53/arrow58 branches + reconcile federation analyzer API

* fix(connectors): SqlTable::new 4-arg (Option<Engine>) — connector-postgres + postgres tests (table-providers 9955a7b bump)

* fix compile

* fix mongo

* fix brnahc name

* fix validate_table_providers_commit.sh

* fix test

* 3 bugs

* fix 'test_b2_inline_seq_above_delete_seq'

* fix test

* more tests

* Restore federation logic

* fix delta

* fix delta

* Fix s3 metadata columns

* Update snapshots

* fix(runtime): bump Cayenne concat_arrays size 32→33 for trunk #11239 file_pruning param (trunk-merge fixup)

* Fix DuckDB bad merge

* Fix Postgres merge

* Revert flight and flightsql changes

* lint

* Fix build

* Fix lint

* fix(cayenne): restore analytical-join in-list pushdown on DF53 — raise native hash-join InList caps

The DF53 reconciliation dropped the forked ExactLeftAccumulator (exact in-list,
128 MiB budget) that pushed a precise scan-pruning predicate into Cayenne probe
scans for large-probe analytical joins. DF53's native InList pushdown is capped
at 150 distinct values / 128 KiB per partition (point-lookup defaults), so
dimension->fact joins fall back to min/max bounds and the order_line/stock probe
scans run unpruned — a CH-benCH QPH regression.

Raise hash_join_inlist_pushdown_max_distinct_values + _max_size (bounded by the
memory limit) in configure_hash_join_memory_limits so the native exact-membership
in-list fires for analytical build sides; genuinely huge builds still fall back
to hash-table membership.

* feat(cayenne): restore gated ExactLeftAccumulator join rewriter on DF53

Bump the datafusion pin to the re-ported pluggable-accumulator seam
(spiceai/datafusion lukim/datafusion-53-exact-accumulator) and restore the
in-repo join_accumulator (ExactLeftAccumulator) + CayenneJoinRewriter, gated
behind cayenne_optimizer_rules.exact_join_filter (opt-in, off by default).
Coexists with the cap-raise default path and CayenneDynamicFilterSharing.

* Revert "fix(cayenne): restore analytical-join in-list pushdown on DF53 — raise native hash-join InList caps"

This reverts commit 42ce9012e70ef6dc8cba04c6636093202a543887.

* feat(cayenne): two-phase off-fence mem-tier checkpoint (CDC ingest moonshot)

Move the Vortex encode + BEGIN IMMEDIATE metastore commit OUT of the listing
fence in checkpoint_mem_tier (key/Int64 strategies); the fence is now held only
for the cheap in-memory visibility swap. Concurrent CDC appends (which also take
the listing fence) no longer stall on the checkpoint: measured fence-held stall
drops from 12-42ms to ~1us per checkpoint (~17,000-41,000x) at 1-30ms metastore
RTT. The position-based strategy keeps encode+swap under one fence (unchanged).

- table.rs: two-phase checkpoint. clear_mem_tier_up_to_epoch now retains ONLY the
  survivor segments via MemTier::retain_after instead of keeping the whole tier,
  fixing a double-count the off-fence move exposed (an append interleaving the
  off-fence encode/commit made the survivors branch reachable, which re-flushed
  the already-durable prefix into a second file).
- mem_tier.rs: MemSegment carries its per-segment tombstones/bytes/rows/superseded
  so a partial checkpoint rebuilds the survivors' aggregate exactly.
- metadata.rs: raise cdc_mem_tier_max_bytes 64MiB->256MiB and max_age 2s->10s now
  that checkpoints do not hold the fence (the synchronous write-path spill is a
  rare backstop; the non-blocking background checkpointer is the primary flush).
- bench: mem_tier_checkpoint_fence_stall group. test:
  mem_tier_checkpoint_off_fence_preserves_concurrent_append (no vanish/double-count).

* fix(cayenne): pin mem-tier checkpoint sequence under the fence (off-fence over-count)

The off-fence checkpoint reserved its snapshot_sequence OUTSIDE the listing
fence, during/after the encode. A concurrent CDC upsert could then reserve a
sequence BELOW that snapshot_sequence yet append AFTER the checkpoint captured
its flush set — inverting merge-on-read order so the upsert's tombstone
(delete_sequence < snapshot_sequence) could never hide the just-flushed durable
copy. Result: a permanent over-count (SF-100 customer +362, stock +15,132; the
new stress test reproduces a clean 2x locally).

Fix: make append and checkpoint sequence assignment mutually exclusive under the
listing fence. Appends now reserve their (delete,data) pair INSIDE the fence;
the checkpoint reserves its snapshot_sequence atomically with the snapshot
capture under a brief fence at the top, then encodes + commits off the fence
using that pinned sequence. Invariant restored: snapshot_sequence is strictly
above every flushed row and strictly below every later append, so a concurrent
supersede always outranks the durable copy and its tombstone hides it.

Adds mem_tier_checkpoint_no_overcount_under_interleaved_upserts: 64 keys x 40
upsert rounds interleaved with a concurrent checkpoint loop, asserts exactly one
row per key (no over-count, no vanish), last value wins, reload-stable. Reliably
reproduced the 2x over-count before this fix. Full cayenne suite (518 lib + acid/
deletion/on-conflict integration) green; clippy clean.

* fix(cayenne): bound mem-tier checkpoint churn (min-flush gate + orphan snapshot cleanup)

SF-100 forensics on the off-fence runs found the QPH/lag collapse was checkpoint
CHURN, not the checkpoint itself (local apply repro is flat at ~2M rows/s):

- Every 1s periodic tick durably checkpointed even a few-KB delta: a new
  snapshot dir + delete-vector files + a listing refresh per tick. Measured:
  408-676 snapshot dirs per heavy table per 10-minute run.
- checkpoint_mem_tier never called trigger_old_snapshot_cleanup, so every
  checkpoint LEAKED its predecessor's dir: order_line ended with 408 dirs on
  disk while the catalog referenced 2 (~406 orphans; 89G footprint).
- The catalog accumulated 40.3M cayenne_insert_record rows (1.8GB) + 14K
  delete-file rows in-run, slowing every single-writer publish commit as the
  b-tree grew.

Fixes:
- New cdc_mem_tier_min_flush_bytes (default 32MiB): the periodic tick skips a
  tier below this size unless its age reached cdc_mem_tier_max_age_ms (slot-ack
  and crash-replay stay bounded; query freshness is unaffected since RAM rows
  are visible immediately on append). Write-path cap spill + explicit
  checkpoints bypass the gate. ~10-30x less snapshot/delete-file/insert-record
  churn at CH-benCH rates.
- checkpoint_mem_tier phase-2 now triggers the same graceful old-snapshot
  cleanup as every other publish path (orphan leak fixed).
- ParameterSpec descriptions updated for the 256MiB/10s defaults; new param
  cayenne_cdc_mem_tier_min_flush_bytes plumbed.

Tests: mem_tier_periodic_tick_gates_small_young_tier (gate skips small+young,
age cap still flushes); legacy tick tests run with the gate off; plus an
#[ignore]d timed apply-throughput repro. 519 lib tests green; clippy clean.

* fix(runtime): bump cayenne PARAMETERS concat_arrays length for cdc_mem_tier_min_flush_bytes (33->34)

* perf(cayenne): O(1) per-scan mem-tier deletion view (persistent maps + merged-snapshot memo)

Tuned2 SF-100 left one gap: QPH (941 vs file's 7499). Per-query forensics showed
most queries within ~2x of file baseline but q20 (correlated subquery over
stock+order_line, re-scans per outer group) at 322ms -> 74.6s: every scan paid
two O(tier-tombstones) costs on the two biggest resident tiers.

1. InlinedDeletionMaps now holds the SAME persistent (im::HashMap/XXH3) maps as
   the tier's InMemTombstones, so mem_tier_deletion_maps is an O(1) HAMT-root
   clone per scan instead of an O(tier) std-HashMap materialization.
2. The per-scan with_mem_tier_tombstones extend (tier tombstones merged into the
   file-side deletion index, incl. bloom inserts) is memoized on the provider.
   Key = (file-index Arc ptr, NEW MemTier.version content counter, structural
   epoch): a hit requires all three, so any append/clear/publish forces a
   rebuild and a stale pairing can never be served (MemTier.version exists
   because epoch is preserved across clears and is therefore ambiguous).
   Quiescent re-scans - q20's correlated rescans - now pay O(1) load_full.

Test: merged_scan_deletions_memo_reuses_and_invalidates (memo hit on quiescent
re-scan via Arc::ptr_eq, invalidation on append, merge-correct visibility).
520 lib tests green; acid/int64/keybased/on_conflict integration green; clippy
clean. NOTE: inline_with_pending_deletions_test::test_checkpoint_after_inlined_upsert
fails PRE-EXISTING at the branch base 136808ce18 (DF53 line, unrelated).

* refactor(cayenne): simplify mem-tier checkpoint/scan additions (no behavior change)

/simplify pass over the moonshot diff:

- InlinedDeletionMaps is now a type alias of InMemTombstones (it had become a
  field-for-field duplicate incl. the hand-rolled Default); the identity bridge
  in mem_tier_deletion_maps collapses to snapshot.tombstones.clone(), and the
  duplicated int64 key-range fold is gone (int64_deleted_key_range promoted out
  of cfg(test) — one implementation, no hand-tracked parity).
- The triplicated under-fence checkpoint tail (clear prefix + inline-metadata
  clear + row-count rederive) is one helper, and clear_mem_tier_up_to_epoch is
  renamed clear_mem_tier_flushed_prefix (it clears a segment-count prefix; the
  epoch param fed only a tautological debug_assert).
- The merged-deletions memo moved out of the 470-line scan body into a provider
  method next to its struct (one altitude, one place that owns the key proof).
- The age-cap predicate is defined once (mem_tier_age_cap_reached) and shared by
  the write-path spill and the tick churn gate — the slot-ack bound holds only
  while both compute the identical predicate.
- retain_after seeds aggregates from the first survivor (O(1) root clone) and
  folds only the rest — the common 1-survivor case under the phase-2 fence is
  now scalar adds.
- 7 copy-pasted no-op TestAdvancer impls deduped to one NoopSlotAdvancer; the
  bench's duplicate no-op symbol shares the existing one; satellite copies of
  the sequence-ordering invariant comment now point at the canonical capture-
  site statement; epoch's doc cross-references version (the content key).

520 lib tests + acid/int64/keybased/on_conflict integration green; clippy clean
(the 3 bench findings under --benches are pre-existing in untouched benches).

* test(cayenne): pin probe-side dynamic-filter install with a resident mem tier

Adds mem_tier_join_probe_keeps_dynamic_filter_pushdown: with the fact scan a
union(file branch, bare memory branch), DF53's hash-join dynamic filter still
installs on the probe-side DataSourceExec — verified to hold today, guarded
against future routing regressions.

Investigation note: this REFUTES the union-blocks-pushdown hypothesis for the
SF-100 memory-mode dimension-join slowdown (q20 322ms -> 65s; supplier-join set
4-200x). An always-true-FilterExec absorber on the memory branch was prototyped
and changed nothing (A/B via this test), so it was not shipped. The open thread
is the live SF-100 plan: join mode / statistics() with a resident tier (EXPLAIN
q20 under load).

* chore(cayenne): drop two stale #[expect(items_after_statements)] left by the NoopSlotAdvancer dedup

* bench(cayenne): join-shape x tier-state matrix for memory-mode scan tax

Measures {single_scan, q20-shaped semi_join, q21-shaped triple_scan} x
{file_only, overlay_fresh, overlay_upsert} on a memory-mode table, with a
correctness gate (every lane's result must equal file_only's; overlay rows are
out-of-domain or value-ident…