Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13136

Data exchange (shuffle, broadcast) should only be handled by the exchange operator

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      In an ideal architecture, we have a very small number of physical operators that handle data exchanges, and the rest simply declare the input data distribution needed and let the planner inject the right Exchange operators.

      We have almost that, except the following few operators:

      1. Limit: does its own shuffle or collect to get data to a single partition.
      2. Except: does its own shuffle; note that this operator is going away and will be replaced by anti-join (SPARK-12660).
      3. broadcast joins: broadcast joins do its own broadcast, which is a form of data exchange.

      Here are a straw man for limit. Split the current Limit operator into two: a partition-local limit and a terminal limit. Partition-local limit is just a normal unary operator. The terminal limit requires the input data distribution to be a single partition, and then takes its own limit. We then update the planner (strategies) to turn a logical limit into a partition local limit and a terminal limit.

      For broadcast join, it is more involved. We would need to design the interface for the physical operators (e.g. we are no longer taking an iterator as input on the probe side), and allow Exchange to handle data broadcast.

      Note that this is an important step towards creating a clear delineation between distributed query execution and single-threaded query execution.

      Attachments

        Issue Links

          Activity

            People

              hvanhovell Herman van Hövell
              rxin Reynold Xin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: