Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24497

ANSI SQL: Recursive query

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 4.1.0
    • SQL

    Description

      Examples

      Here is an example for WITH RECURSIVE clause usage. Table "department" represents the structure of an organization as an adjacency list.

      CREATE TABLE department (
          id INTEGER PRIMARY KEY,  -- department ID
          parent_department INTEGER REFERENCES department, -- upper department ID
          name TEXT -- department name
      );
      
      INSERT INTO department (id, parent_department, "name")
      VALUES
           (0, NULL, 'ROOT'),
           (1, 0, 'A'),
           (2, 1, 'B'),
           (3, 2, 'C'),
           (4, 2, 'D'),
           (5, 0, 'E'),
           (6, 4, 'F'),
           (7, 5, 'G');
      
      -- department structure represented here is as follows:
      --
      -- ROOT-+->A-+->B-+->C
      --      |         |
      --      |         +->D-+->F
      --      +->E-+->G
      

       
      To extract all departments under A, you can use the following recursive query:

      WITH RECURSIVE subdepartment AS
      (
          -- non-recursive term
          SELECT * FROM department WHERE name = 'A'
      
          UNION ALL
      
          -- recursive term
          SELECT d.*
          FROM
              department AS d
          JOIN
              subdepartment AS sd
              ON (d.parent_department = sd.id)
      )
      SELECT *
      FROM subdepartment
      ORDER BY name;
      

      More details:

      http://wiki.postgresql.org/wiki/CTEReadme

      https://info.teradata.com/htmlpubs/DB_TTU_16_00/index.html#page/SQL_Reference/B035-1141-160K/lqe1472241402390.html

       

      Attachments

        Issue Links

          1.
          Extend CTE logical plan classes with recursion parameter Sub-task Resolved Milan Stefanovic
          2.
          Extend CTESubstitution.scala to make it aware of recursion Sub-task Resolved Milan Cupac
          3.
          Changes in ResolveWithCTE.scala to have the analyzer grok recursive anchors Sub-task Resolved Milan Cupac
          4.
          Addition of InsertLoops to replace recursive Union with UnionLoop and CTERelationRef referencing itself with UnionLoopRef Sub-task Closed Unassigned
          5.
          Implement checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with Unionloop cases. Sub-task Resolved Milan Cupac
          6.
          Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries Sub-task Resolved Milan Cupac
          7.
          Optimize Recursive CTE execution for simple queries Sub-task Resolved Pavle Martinović
          8.
          SQL Metrics Sub-task Resolved Pavle Martinović
          9.
          Add column pruning to Recursive CTEs Sub-task Resolved Unassigned
          10.
          Add Cross Join as legal in recursion of Recursive CTE Sub-task Resolved Pavle Martinović
          11.
          Fix case with multiple rCTEs that reference each other Sub-task Resolved Pavle Martinović
          12.
          Enable rCTE referencing from within a CTE Sub-task Resolved Pavle Martinović
          13.
          Enable multiple self-references and self-references from a Subquery inside rCTEs Sub-task Resolved Pavle Martinović
          14.
          Enable overriding the recursion row limit by adding a LIMIT operator Sub-task Resolved Pavle Martinović
          15.
          Enable overriding the recursion level limit in syntax Sub-task Resolved Pavle Martinović
          16.
          Disable Inline Forcing for all rCTEs Sub-task Resolved Pavle Martinović
          17.
          Fix non-deterministic queries to produce different results at every step Sub-task Resolved Pavle Martinović
          18.
          Fix bug with multiple self-references with UnresolvedSubqueries in rCTEs Sub-task Resolved Pavle Martinović
          19.
          Fix bug with wrong constraints in LogicalRDDs referencing previous iterations Sub-task Resolved Pavle Martinović
          20.
          Add type coercion to UnionLoop Sub-task Resolved Pavle Martinović
          21.
          Resolve bug with nullability of Union in rCTEs Sub-task Resolved Pavle Martinović
          22.
          Disable Recursive CTE self-references from window functions and inside sorts Sub-task Resolved Pavle Martinović
          23.
          Return correct error message for anchor self references in rCTEs Sub-task Resolved Pavle Martinović
          24.
          Ban self references inside topmost CTEs in Recursive CTEs Sub-task Resolved Pavle Martinović
          25.
          Enable LIMIT ALL to override recursion row limit Sub-task Resolved Pavle Martinović
          26.
          Fix bug with eliminated self reference Sub-task Resolved Pavle Martinović
          27.
          Remove useless Casts in type coercion for rCTEs Sub-task Resolved Pavle Martinović
          28.
          Add plan normalization for recursive CTEs Sub-task Resolved Pavle Martinović

          Activity

            People

              pajaraja Pavle Martinović
              yumwang Yuming Wang
              Votes:
              6 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: