Add example for using CacheFactory with DataFrame::cache #18893

Activity

1. milenkovicm commented on Dec 1, 2025

   

   milenkovicm

   on Dec 1, 2025

   Contributor

   More actions

   It gets a bit complicated when it comes to ballista implementation of cache. There are few things which needed to be implemented:

   • query planner extension which would be able to handle custom logical plan extension
   • there is need for a some kind cache manager which would need to handle cache locations
   • also there is need for job breakdown, a first job which would materialize cache and second which would read the cache ...

   It gets event more complicated when cache reconciliation and task colocation are considered.

   I'm not sure if we can or should implement it all for a showcase
2. jizezhang commented on Dec 2, 2025

   

   jizezhang

   on Dec 2, 2025

   ContributorAuthor

   More actions

   Thanks for the explanation, makes sense. I was just thinking what would be a good meaningful CacheFactory example for users.
3. jizezhang commented on Dec 7, 2025

   

   jizezhang

   on Dec 7, 2025

   ContributorAuthor

   More actions

   Hi @milenkovicm can you review this PR #19139 when you get a chance? If the overall content looks good, I'll work on adding docs and polishing. Thanks!
4. github-merge-queue

   added a commit that references this issue on Dec 21, 2025

   doc: add example for cache factory (#19139)

   ...

   2c3566c
5. milenkovicm

   closed this as completedin #19139on Dec 21, 2025