Implement spark action to compute partition stats #8459

Activity

1. ajantha-bhat

   changed the title [-]Implement parallel stats using metadata tables and a CALL procedure.[/-] [+]Implement spark action to compute partition stats and a CALL procedure.[/+] on Jan 8, 2024
2. ajantha-bhat

   changed the title [-]Implement spark action to compute partition stats and a CALL procedure.[/-] [+]Implement spark action to compute partition stats[/+] on Jan 8, 2024
3. ajantha-bhat commented on Apr 8, 2024

   

   ajantha-bhat

   on Apr 8, 2024

   MemberAuthor

   More actions

   Based on the experiments from #9437, spark action is not effective as the serialization cost of each partition stats entry is expensive.

   Will implement an API in core module to compute stats in a distributed way and a spark procedure to call this API.
4. ajantha-bhat

   closed this as not plannedon Apr 8, 2024