Use MemoryManager in Join operators, and return errors if the memory budget is exceeded #5220
Description
See description on #3941
The idea is to apply the same treatment to Join operators that was done to GroupBy in #3940 and throw an error if their memory budget is exceeded rather than unlimited allocating.
All the operators in https://github.com/apache/arrow-datafusion/tree/master/datafusion/core/src/physical_plan/joins need to be evaluated.
Items to do:
- Apply limits to NLJ Ignoring of memory-pool limits & OOM on large cartesian-product join #5162
- Apply limits to Cross Join
- Apply limits to Hash Join
- Apply limits to Sort Merge Join
cc @DDtKey and @crepererum who have expressed interest in this project in the past