[SPARK-55785][UI] Compact SQL plan visualization with detail side panel by yaooqinn · Pull Request #54565 · apache/spark

yaooqinn · 2026-03-01T19:02:04Z

What changes were proposed in this pull request?

Improve the SQL plan visualization on the execution detail page by making node labels compact and adding a clickable detail side panel for metrics.

Key changes:

Compact node labels: Show only operator name as plain text (removed HTML labelType and <b> tags)
Compact cluster labels: Shortened from WholeStageCodegen (1) / duration: total (min, med, max)... to (1) / 9.3s
Detail side panel (col-4 right sidebar): Click any node to see its metrics in a structured table
Cluster click: Shows cluster-level metrics + all child node metrics grouped by operator
Structured metric tables: Uses metricType from server JSON to render Total/Min/Med/Max sub-tables (no regex parsing)
Panel collapses with Plan Visualization toggle
Scrollable panel body (max-height: 80vh) for large metric sets
Layout tuning: Reduced ranksep/nodesep, cluster padding via dagre pre-layout

Why are the changes needed?

The existing SQL plan visualization bakes all metrics into node labels, making them hard to read for large plans with many operators. The compact layout with on-demand detail panel provides a cleaner overview while keeping full metric details accessible.

Does this PR introduce any user-facing change?

Yes. The SQL execution detail page now shows:

Compact plan graph with operator names only
A right-side panel showing full metrics on node click
Cluster labels with just the codegen stage number and total duration

How was this patch tested?

Updated SparkPlanGraphSuite for new label format
Added test for makeNodeDetailsJson verifying cluster prefix and children array
Manual verification with simple and complex queries (multi-way joins, unions, window functions)
Scalastyle passes

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored with GitHub Copilot.

…il side panel Improve the SQL execution detail page plan visualization by making DAG nodes compact and adding a detail side panel for on-demand metric viewing. Changes: - SparkPlanGraph: Node labels now show only the operator name (no inline metrics), making nodes uniformly small and the graph easier to scan for large queries - SparkPlanGraph: New makeNodeDetailsJson() generates a JSON blob with full node metrics, stored in a hidden div for JavaScript consumption - ExecutionPage: Plan viz area split into col-9 (graph) + col-3 (sticky detail panel) using BS5 grid. Panel shows 'Details' initially - spark-sql-viz.js: On node click, side panel populates with operator name, description, and a metrics table. Existing highlight behavior preserved - spark-sql-viz.css: Panel styling consistent with plan viz color scheme Before: All metrics embedded in DAG node labels, making boxes large and the graph hard to navigate for complex queries with many operators. After: Compact uniform nodes with full metrics accessible via click in a persistent side panel. Co-authored-by: Copilot <[email protected]>

yaooqinn · 2026-03-01T19:23:31Z

Screenshots

Plan Overview

Node Click

Cluster Click

Complex Plan

Copilot

Pull request overview

This PR updates Spark SQL’s execution detail “Plan Visualization” to use compact node/cluster labels in the rendered DOT graph, and moves full metric rendering into a clickable right-side details panel driven by server-provided JSON.

Changes:

Make SparkPlan DOT node labels compact (operator name only) and shorten WholeStageCodegen cluster labels.
Add makeNodeDetailsJson to emit node/cluster metric details for a new right-side details panel.
Update the Plan Visualization page layout + JS/CSS to support node/cluster selection and metric table rendering.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala	Generates compact DOT labels and adds node-details JSON for the UI side panel.
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala	Adds a two-column layout with a sticky details panel and embeds node-details JSON in metadata.
sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js	Implements detail panel rendering, cluster click behavior, and tighter layout spacing.
sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.css	Styles compact labels and the new details side panel.
sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SparkPlanGraphSuite.scala	Updates DOT label expectation and adds coverage for node-details JSON prefixes/children.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala

sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/spark-sql-viz.js

sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SparkPlanGraphSuite.scala

sarutak · 2026-03-02T15:04:41Z

@yaooqinn Thank you for the nice proposal! Let me check in details and will leave my comment tomorrow.

yaooqinn · 2026-03-02T15:09:46Z

Thank you @sarutak，take you time

gengliangwang · 2026-03-03T13:48:43Z

@yaooqinn This looks great! I do have one suggestion: with the current change, users can only view metric from one node at a time. What do you think about displaying all nodes connected to the selected node?(highlighted nodes in screenshot) For instance, it would be helpful if users could see the row count right before a filter or join operation.

yaooqinn · 2026-03-03T15:27:38Z

What do you think about displaying all nodes connected to the selected node?

In this case, for multi-way join or union, the side table will be a mess.

However, your suggestion inspires me that we can add some functionality to the Viz that shows only how a single data/time/memory flow view like

gengliangwang · 2026-03-03T21:23:07Z

In this case, for multi-way join or union, the side table will be a mess.

We can either

show the number of rows in the edge, and keep the side table
or, show the detailed metrics of highlighted nodes as a graph instead of a side table

yaooqinn · 2026-03-04T01:02:33Z

@gengliangwang OK, I will consider that. BTW, the current design referenced databricks' query profiler which also show these data on a side panel

yaooqinn · 2026-03-04T17:13:31Z

Updated: Edge labels showing row counts between operators

Compact plan visualization with row count labels on edges. The 5B row join explosion is immediately visible.

Full plan view — edge labels show data flow: 1M + 500K inputs → 5B after BroadcastHashJoin → 400 after aggregation → 100 after filter

Edge labels highlight:

Data volume flowing between operators
Join explosion detection (5B rows from 1M × 500K)
Aggregation effectiveness (5B → 400 rows)

cc @gengliangwang

sarutak · 2026-03-04T17:31:54Z

Thank you @yaooqinn for proposing a nice feature! I'd like to leave some comments and questions.

As @gengliangwang was concerned, I'm wondering some users may prefer the old view where all metrics are visible at a glance. So, how about adding a toggle to switch between compact and detailed (legacy) modes?
When there are multiple operators with the same name (e.g., multiple HashAggregates), the panel header only shows the operator name, making it hard to tell which one is selected without looking at the graph highlight. Would it be worth adding the desc field to makeNodeDetailsJson and displaying it in the panel?
Since labelType: "html" was removed and labels are now plain text, could you verify that operator names containing < or > render correctly in the SVG text nodes?
The old code added empty lines for nodes without metrics to ensure consistent node heights. With compact labels this seems unnecessary, but have you checked that edge-to-node connections still look correct (no gaps) for metric-less nodes?

yaooqinn · 2026-03-04T17:50:35Z

Thanks @sarutak for the thorough review! All 4 points addressed:

Toggle compact/detailed mode — Added a "Show metrics in graph nodes" checkbox that switches between compact labels (name only) and detailed labels (name + all metrics in nodes). State persisted in localStorage.
desc in side panel — Added the full plan description (desc) to the side panel JSON and display. Now when clicking a HashAggregate, you see the full signature which disambiguates duplicates.
Special chars in operator names — Verified: DOT labels use StringEscapeUtils.escapeJava and dagre-d3 renders SVG text elements (not innerHTML), so special characters are safe.
Edge connections for metric-less nodes — Verified: dagre-d3 auto-sizes nodes based on label content. Compact labels produce smaller nodes; edges connect to node borders correctly.

sarutak · 2026-03-04T18:57:22Z

As the appearance was updated by the latest change, could you update the PR description?
Also, should we update web-ui.html (web-ui.md) ?

Replaces verbose metric-laden DOT node labels with compact operator names and a clickable detail side panel. Also adds row count labels on edges to show data flow volume between operators. Changes: - SparkPlanGraph: compact node labels (name only), JSON detail data for side panel, edge labels showing number of output rows - spark-sql-viz.js: side panel with metrics table on node click, edge highlighting, proper XSS escaping via textContent - spark-sql-viz.css: side panel styles, edge label styles Co-authored-by: Copilot <[email protected]>

yaooqinn · 2026-03-05T04:31:02Z

Updated screenshots — all features:

1. Compact mode (default) with edge row counts:

2. Side panel with metrics table on node click:

3. Desc tooltip on panel title hover:

4. Detailed mode — metrics tables embedded in graph nodes (10px font):

5. Detailed mode + Show Stage/Task IDs:

yaooqinn · 2026-03-05T04:40:42Z

@sarutak

Please take a look at the new design,

the metrics table is embedded in the plan nodes when toggle on
the desc as tooltips for the metrics panel
web-ui.html updates, I'd prefer a new JIRA ticket to finalize all UI changes in SPARK-55846, WDYT?

yaooqinn · 2026-03-05T15:46:49Z

Merged to master, thank you @sarutak

gengliangwang

Late LGTM. Thanks for incorporating the suggestions and improvements.

yaooqinn · 2026-03-05T17:24:14Z

Thank you @gengliangwang

yaooqinn force-pushed the SPARK-55785 branch from a0a9091 to 5f06af1 Compare March 1, 2026 19:22

yaooqinn requested a review from Copilot March 2, 2026 03:13

Copilot started reviewing on behalf of yaooqinn March 2, 2026 03:14 View session

yaooqinn requested review from dongjoon-hyun and sarutak March 2, 2026 03:17

Copilot AI reviewed Mar 2, 2026

View reviewed changes

yaooqinn requested review from cloud-fan and gengliangwang March 3, 2026 11:31

yaooqinn changed the title ~~[SPARK-55786][UI] Compact SQL plan visualization with detail side panel~~ [SPARK-55785][UI] Compact SQL plan visualization with detail side panel Mar 3, 2026

yaooqinn force-pushed the SPARK-55785 branch 2 times, most recently from 96a4a1d to b972b72 Compare March 4, 2026 16:59

yaooqinn force-pushed the SPARK-55785 branch from b972b72 to ba7a074 Compare March 4, 2026 17:49

yaooqinn force-pushed the SPARK-55785 branch 5 times, most recently from 3891dfb to fb33fae Compare March 5, 2026 04:21

yaooqinn force-pushed the SPARK-55785 branch from fb33fae to 60ba3b6 Compare March 5, 2026 04:28

sarutak approved these changes Mar 5, 2026

View reviewed changes

yaooqinn closed this in b42b1f2 Mar 5, 2026

yaooqinn deleted the SPARK-55785 branch March 5, 2026 15:46

gengliangwang reviewed Mar 5, 2026

View reviewed changes

Conversation

yaooqinn commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

yaooqinn commented Mar 1, 2026

Screenshots

Plan Overview

Node Click

Cluster Click

Complex Plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarutak commented Mar 2, 2026

Uh oh!

yaooqinn commented Mar 2, 2026

Uh oh!

gengliangwang commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaooqinn commented Mar 3, 2026

Uh oh!

gengliangwang commented Mar 3, 2026

Uh oh!

yaooqinn commented Mar 4, 2026

Uh oh!

yaooqinn commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Mar 4, 2026

Uh oh!

yaooqinn commented Mar 4, 2026

Uh oh!

sarutak commented Mar 4, 2026

Uh oh!

yaooqinn commented Mar 5, 2026

Uh oh!

yaooqinn commented Mar 5, 2026

Uh oh!

yaooqinn commented Mar 5, 2026

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yaooqinn commented Mar 1, 2026 •

edited

Loading

gengliangwang commented Mar 3, 2026 •

edited

Loading

yaooqinn commented Mar 4, 2026 •

edited

Loading