Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` by alamb · Pull Request #15166 · apache/datafusion

alamb · 2025-03-11T21:03:47Z

Which issue does this PR close?

Closes Support different EXPLAIN formats via SQL #15021
Closes Graphviz PhysicalPlan #219

Rationale for this change

I would like to be able to see more of the great explain plan code that DataFusion already has (see #15021 for details)

What changes are included in this PR?

Changes:

Here is an example

> explain format text select * from values(1) t(x);
Invalid or Unsupported Configuration: Invalid explain format: text
> explain format indent select * from values(1) t(x);
+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | SubqueryAlias: t                                    |
|               |   Projection: column1 AS x                          |
|               |     Values: (Int64(1))                              |
| physical_plan | ProjectionExec: expr=[column1@0 as x]               |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+
2 row(s) fetched.
Elapsed 0.013 seconds.

> explain format tree select * from values(1) t(x);
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │        x: column1@0       │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │         bytes: 128        │ |
|               | │       format: memory      │ |
|               | │          rows: 1          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.

Are these changes tested?

Are there any user-facing changes?

datafusion/sql/src/parser.rs

waynexia · 2025-03-12T18:11:10Z

datafusion/sql/src/statement.rs

                ..
            } => {
-                self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
+                let format = format.map(|format| format.to_string());


Not very related to this PR, will we change sqlparser part to use a String instead of enum like here? (I'd prefer so, as it gives us more flexibility)

I think that would be a fine idea to propose upstream in sqlparser

I am not aware of any plans at the moment

docs/source/user-guide/sql/explain.md

datafusion/sql/src/parser.rs

Co-authored-by: Ruihang Xia <[email protected]>

alamb · 2025-03-12T18:59:57Z

datafusion/physical-plan/src/display.rs

    TreeRender,
 }

-impl FromStr for DisplayFormatType {


This is superceded by the ExplainFormat

alamb · 2025-03-12T19:00:09Z

datafusion/sqllogictest/test_files/explain.slt

 statement ok
 drop table t2;
+
+## Tests for explain format


Here are the tests showing how this works

alamb · 2025-03-12T20:08:36Z

FYI @irenjj

…o alamb/explan_format

irenjj · 2025-03-13T01:07:49Z

Thanks @alamb !
I did a quick local run. It looks like datafusion.explain.format and explain ... format [type] work together to determine the final output of EXPLAIN. datafusion.explain.format sets the default, while explain ... format [type] determines the final output. The explain ... format [type] option has a higher priority. Maybe we can mention this in explain.md.

irenjj · 2025-03-13T01:09:27Z

I'll take a look at others later today.❤️

irenjj · 2025-03-13T04:33:19Z

datafusion/core/src/physical_planner.rs

+                    displayable(optimized_plan.as_ref()).to_stringified(
+                        e.verbose,
+                        FinalPhysicalPlan,
+                        DisplayFormatType::TreeRender,
+                    ),


Maybe we can add a new function to handle tree explain like display_pg_json, then the third parameter is no longer needed as I found the values passed in below are all default values.

That is a good idea -- I will try it

I did it in a83ec4e and i think it looks much nicer ❤️

irenjj · 2025-03-13T04:37:28Z

Thanks @alamb , others LGTM.

alamb · 2025-03-13T20:10:43Z

Thanks @alamb ! I did a quick local run. It looks like datafusion.explain.format and explain ... format [type] work together to determine the final output of EXPLAIN. datafusion.explain.format sets the default, while explain ... format [type] determines the final output. The explain ... format [type] option has a higher priority. Maybe we can mention this in explain.md.

This is a good idea -- thank you. I did it in 31f402d

irenjj

LGTM, thanks again! @alamb and @waynexia

waynexia

Looking great! Thank you @alamb @irenjj

alamb · 2025-03-15T12:19:14Z

Thank you again for your review @irenjj and @waynexia 🙏

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Mar 11, 2025

alamb force-pushed the alamb/explan_format branch from d2e0d63 to aa53939 Compare March 12, 2025 15:54

alamb commented Mar 12, 2025

View reviewed changes

datafusion/sql/src/parser.rs Outdated Show resolved Hide resolved

alamb force-pushed the alamb/explan_format branch from aa53939 to b23e3cd Compare March 12, 2025 16:37

github-actions bot added the documentation Improvements or additions to documentation label Mar 12, 2025

alamb mentioned this pull request Mar 12, 2025

Minor: split EXPLAIN and ANALYZE planning into different functions #15188

Merged

alamb force-pushed the alamb/explan_format branch from b23e3cd to 187f38e Compare March 12, 2025 16:48

alamb mentioned this pull request Mar 12, 2025

Support different EXPLAIN formats via SQL #15021

Closed

waynexia reviewed Mar 12, 2025

View reviewed changes

Support EXPLAIN FORMAT <format>

c965e8e

alamb force-pushed the alamb/explan_format branch from 187f38e to c965e8e Compare March 12, 2025 18:57

alamb marked this pull request as ready for review March 12, 2025 18:58

Update datafusion/sql/src/parser.rs

9a93ec1

Co-authored-by: Ruihang Xia <[email protected]>

alamb commented Mar 12, 2025

View reviewed changes

Improve documentation

398cf18

Merge branch 'alamb/explan_format' of github.com:alamb/datafusion int…

114bf4b

…o alamb/explan_format

irenjj reviewed Mar 13, 2025

View reviewed changes

alamb added 4 commits March 13, 2025 09:04

Remove to_stringified and simplify code

a83ec4e

mark deprecated rather than remove

f2cbf9d

Merge remote-tracking branch 'apache/main' into alamb/explan_format

fe08421

Add a note about explain format configuration

31f402d

irenjj approved these changes Mar 14, 2025

View reviewed changes

waynexia approved these changes Mar 15, 2025

View reviewed changes

alamb merged commit 41e7aed into apache:main Mar 15, 2025
28 checks passed

alamb deleted the alamb/explan_format branch March 15, 2025 12:19

Jefffrey mentioned this pull request Sep 15, 2025

Provide the option of displaying Explain Plans as JSON #6137

Closed

Comments

Conversation

alamb commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

waynexia Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 12, 2025

Uh oh!

irenjj commented Mar 13, 2025

Uh oh!

irenjj commented Mar 13, 2025

Uh oh!

irenjj Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

irenjj commented Mar 13, 2025

Uh oh!

alamb commented Mar 13, 2025

Uh oh!

irenjj left a comment

Choose a reason for hiding this comment

Uh oh!

waynexia left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alamb commented Mar 11, 2025 •

edited

Loading