Skip to content

feat: stats panel, heatmap chart, async pipeline, and Go-stage chunk pruning#126

Merged
fahimfaisaal merged 30 commits into
mainfrom
feat/stats-panel
Jun 14, 2026
Merged

feat: stats panel, heatmap chart, async pipeline, and Go-stage chunk pruning#126
fahimfaisaal merged 30 commits into
mainfrom
feat/stats-panel

Conversation

@fahimfaisaal

@fahimfaisaal fahimfaisaal commented Jun 14, 2026

Copy link
Copy Markdown
Member

Summary

  • Stats Panel — new StatsPanel.vue with descriptive stats (20 metrics × all series), Pearson/Spearman correlation heatmap, virtualized table, searchable/sortable series filter, per-tab CSV export (RFC-4180), and lazy off-thread compute via dedicated stats.worker.ts
  • Heatmap chart — new --charts heatmap type; useHeatmapChartOptions with dataZoom (inside + slider) on both axes when label count > 50, y-label clipping fix, visualMap layout below x-slider, added to --charts valid set and Go-stage chunk graph
  • Correlation heatmap — merged ChartCorrelation.vue into ChartHeatmap.vue; user-selectable correlation axis via Selector.vue dropdown; smart auto-pick (smallest axis, ≥2 entities, ≥3 observations); per-axis cache in useStatsWorker; removed CORR_MAX_ENTITIES / CORR_MIN_OBSERVATIONS caps; always-show cell labels
  • Async chart pipeline — transform worker now owns full raw dataset; grouping, arrangement, sort, scale, and labels are compute params posted to the worker; no main-thread data work on setting changes; eliminated proxy clone cost via shallowRef + markRaw (~7300ms → ~700ms for 500k rows)
  • Go-stage chunk pruningSelectChunks (gated BFS) in pkg/template/chunks.go; GenerateUI prunes unreachable JS chunks at generation time based on --charts + needs3D, shrinking output HTML for non-3D / non-heatmap runs
  • Auto-aggregation — Go pipeline auto-aggregates grouped CSV/JSON rows before JSON write (~200k → ~3k rows); UI buildChartForSignature and cellsFor average duplicates instead of last-wins
  • FullscreenuseFullscreen composable extracted; scoped per containerRef so ChartCard and StatsPanel correlation don't cross-trigger; toolbox saveAsImage wired via createToolboxConfig
  • URL router — defer ?g= deep-link param until worker groups are ready via one-shot Vue 3.5 { once: true } watcher
  • Performance — eliminated double-recompute on first ready by switching groupName: Ref<string> to activeGroupId: Ref<number> in useChartPipeline; pre-populated chart slots via listChartSignatures for immediate per-card skeletons

…ine)

Snapshot of in-progress async-worker refactor before reworking swap-axis to
move all per-setting computation (incl. grouping) into the worker. Includes
new ui/src/lib/swap.ts and temporary debug logs (removed during the rework).
Spec: requirement-swap-async.md
Arrangement-aware, non-mutating grouping (identityKeys -> targetKeys) that
generalizes useDataPoint.grouped. Foundation for moving grouping into the
worker. Unused for now.
The worker now owns the full raw dataset and treats arrangement, group,
sort, scale and showLabels all as compute params. After the one-time init
clone (and again only on a dataset switch), no setting change does any
main-thread data work — swap/group/sort/scale just post a param and render
the returned ChartData.

- transform.worker.ts: own raw rows; init/setArrangement project+group via
  projectAndGroup and reply with groupNames; compute reads grouped.get(group).
  Remove the broken swap message and swapAxisFields.
- useChartPipeline.ts: rawData/arrangement/labels/groupName inputs; reinit
  only on dataset change; arrangement posts setArrangement; group/sort/scale/
  labels recompute off the cache; expose groupNames. Drop triggerSwap.
- useDataPoint.ts: drop the 500k main-thread grouping computeds; add per-dataset
  arrangement state + activeArrangement; source groups from the worker via
  setGroupNames; keep resultGroups/activeGroupId/selectGroup working.
- Dashboard.vue: pass raw data + arrangement + derived labels + groupName;
  mirror the pipeline's groupNames into useDataPoint.
- AxisSwapper.vue: set the arrangement key instead of mutating rows/labels.
- swap.ts: drop the now-unused mutating swapAxisFields.
Groups are populated asynchronously by the worker on first `ready`.
Register a one-shot watcher (Vue 3.5 `{ once: true }`) on
`resultGroups.value.length` so the `?g=N` deep-link is applied once
groups arrive, without delaying `?d=` / settings which remain synchronous.
Replace groupName: Ref<string> param with activeGroupId: Ref<number> in
useChartPipeline. Resolve the group name internally via currentGroupName()
(reads from the pipeline's own groupNames ref), so pumpQueue() called inside
the ready handler uses the correct name synchronously — without waiting for a
downstream watcher to propagate it.

Previously, groupNames [] → [...] on first ready caused activeGroupName to
transition '' → groupNames[0] in a later microtask, which tripped the params
watch and triggered a full second recompute on every page load and every swap.
Now the params watch tracks activeGroupId (an integer), which does not change
on the first ready, so only one compute batch is queued. User group-select
(activeGroupId change) still triggers a single recompute as expected.

Also removes the now-unused activeGroupName computed from useDataPoint, and
collapses a double blank line in AxisSwapper.vue.
Switch dataSets to shallowRef+markRaw so rows stay plain objects;
postMessage clones them natively instead of the JSON round-trip
(~7300ms → ~700ms for 500k rows). Drop console.info(e) in the worker
that serialized the full message to devtools on every init.

Pre-populate chart slots in reinit() via listChartSignatures so per-card
skeletons appear immediately while the worker clones and groups data.
Remove the full-page LoadingSkeleton from the init phase — header and
metadata are visible from the moment data loads; only chart slots pulse.

Fix toStatSignature to handle absent per/unit fields without generating
misleading signature strings.
- Off-thread stats worker (stats.worker.ts + useStatsWorker.ts) computes
  descriptive stats and correlation matrices lazily; WeakMap cache makes
  reopening the same chart instant
- StatsPanel.vue: virtualized descriptive table (series × 20 metrics),
  diverging-heatmap correlation matrix (Pearson/Spearman), skeleton while
  loading, tab-aware CSV download button (RFC-4180, full precision)
- ChartCard.vue: Sigma toggle button shows panel only when series exist
- lib/stats.ts: mean/median/quantile/mode/variance/skew/kurtosis/mad/
  describe/pearson/spearman/correlationMatrix/computeProfiles
- lib/csv.ts: toCsvCell + descriptiveCsv + correlationCsv builders
- Raise tsconfig lib to ES2023; replace slice().sort() with toSorted()
  in stats.ts (×2) and transform.ts (×1)
- Add vitest 3.x; stats.test.ts (40 tests) + csv.test.ts (11 tests)
…raging

Go layer: AggregateDataPoints (shared/aggregate.go) sums DataPoints sharing
the same (name,xAxis,yAxis,zAxis) key. Fires automatically when parser is
csv/json and --group is active — 200k rows collapse to ~3k before JSON write.
Progress logs bracket the heavy passes ("Parsing...", "Aggregating N rows...").

UI layer: buildChartForSignature (2D) and cellsFor (3D) now average duplicates
instead of last-wins overwrite. Correct for benchmark count=N repeats; no-op
for CSV/JSON already pre-summed by Go.

Adds transform.test.ts (+20 tests) covering listChartSignatures, build3DRender,
projectAndGroup, and additional buildChartForSignature branches.
The dataZoom slider's left/right boundary labels rendered in ECharts'
default gray, too dim against the dark canvas. Pass ChartStyling into
createDataZoomConfig and set slider textStyle.color to styling.textColor
so the labels match axis/legend text (#e5e7eb dark, #374151 light).
Click any column header in the stats panel's descriptive table to sort
series rows by that metric. Tri-state cycle: descending -> ascending ->
original series order, with an arrow indicator. Non-finite cells always
sink to the bottom so a mostly-missing column still surfaces real
min/max. Sort runs main-thread over already-computed profiles (no worker
round-trip); virtualization and CSV export follow the sorted order.
Shows a debounced (300 ms) search input above the table when series
count exceeds 20. Filters rows by case-insensitive name match on top
of the existing column sort. Virtualization and CSV export follow the
filtered result; search resets on chart change.
Cap search input at max-w-xs (20rem). Derive the series column header
and search placeholder from chartData.axisLabels.x, falling back to
"Series". Show a filtered/total count beside the input while a query
is active.
Replace gap-2 with ml-auto so the filtered/total count sits at the
right edge of the row rather than immediately beside the input.
Move the stats panel off the chart's critical path and make the
correlation matrix pick the best axis to correlate along instead of
always using the series axis.

Non-blocking compute:
- Dedicated stats worker now serves kinded requests (descriptive eager,
  correlation lazy) instead of one eager computeProfiles call, so opening
  a panel never blocks the chart and a tab computes only when first opened.
- Per-ChartData piece cache in useStatsWorker; each tab shows its own
  loading skeleton.

Smart correlation axis (x -> y -> z cascade):
- selectCorrelationAxis prefers the series (x) axis, falls back to the
  category (y) axis, then the z axis (3D), taking the first whose entity
  count fits the 200 cap and which has >=3 observations. A chart with
  thousands of series but few categories now correlates the categories
  instead of showing nothing.
- Generic buildCorrelationColumns fits the same correlationMatrix over any
  entity axis (y/z are transposes over the other two axes' tuples).
- Correlation renders as an echarts canvas heatmap (ChartCorrelation +
  useCorrelationOption), with a caption naming the picked axis.

CorrelationMatrix gains `axis`; worker/composable plumb zAxis through.
…sable

Extract inline fullscreen logic from ChartCard into a shared useFullscreen
composable (containerRef, isFullscreen, withFullscreenToolbox), then reuse
it in ChartCorrelation to give the heatmap the same toolbox fullscreen toggle.
…on bottom-right

- Add icon slot and ClassValue-typed class prop to Button component
- Use Button component for Stats toggle
- Move axis badges to header row (top-right)
- Move Stats button below chart (bottom-right)
- StatsPanel renders below the Stats button
… layout

- Enable inside + slider dataZoom on both axes when length > 50 (LARGE_X_THRESHOLD)
- x slider at bottom:55, visualMap horizontal centered below at bottom:5
- Fix y-axis label clipping: left:100px when largeX+containLabel:false
- Reduce top padding for 2D heatmap (no legend): 5% instead of 15%
- Axis name (nameGap:41) centered within dataZoom slider band
…ove entity cap

- Merge ChartCorrelation.vue into ChartHeatmap.vue (single dumb renderer)
- StatsPanel hosts fullscreen + saveAsImage toolbox via useFullscreen +
  createToolboxConfig (shared with all charts via getBaseOptions)
- Add axis-selector dropdown (Selector.vue) to correlation tab; user can
  switch which axis supplies matrix entities; defaults to auto-pick (smallest)
- Remove CORR_MAX_ENTITIES / CORR_MIN_OBSERVATIONS caps: any axis with ≥2
  entities is usable; auto-pick = smallest (cheapest) axis, user can switch
  to any including large ones
- Add dataZoom (inside + slider) to correlation heatmap when label count
  exceeds LARGE_X_THRESHOLD, matching regular heatmap behavior
- Per-axis cache in useStatsWorker so switching axes recomputes and caches
  independently
- Update stats.test.ts to reflect new uncapped axis-selection logic
Multiple useFullscreen instances (ChartCard + StatsPanel correlation) all
responded to any fullscreenchange event via !!document.fullscreenElement.
When ChartCard entered fullscreen, corrIsFullscreen also became true,
overlaying the correlation div. Now each instance only tracks whether its
own containerRef is the fullscreen element.
…ank on tab switch

When user scrolled the descriptive table then switched tabs, scrollTop.value
kept its old non-zero value while the remounted DOM element reset to 0.
Returning to the descriptive tab produced a large empty topPad spacer until
the next scroll event synced the state. measure() now reads scrollTop from
the DOM so the watch([view, activeLoading]) call on remount corrects both
scrollTop and viewportH together.
… log

- Add SelectChunks (gated BFS) in pkg/template/chunks.go; GenerateUI now
  accepts charts+needs3D and prunes unreachable chunks at generation time
- Add shared/chart_selection.go with DatasetNeeds3D for 3D engine gating
- Extend --charts flag to include heatmap (ValidSet + SliceDefault)
- Update vite.config.ts compress-bundle log to show raw → encoded (kB)
  instead of duplicating Vite's own gzip output
- Docs: add Output File Size section to ui/index.mdx, Reducing Output Size
  to commands/root.mdx + ui.mdx, Prune step to internals/how-it-works.mdx
- StatsPanel.vue and useSettingsStore.ts updates for stats panel branch
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.24309% with 43 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
cmd/ui.go 60.00% 14 Missing and 2 partials ⚠️
pkg/template/generate-ui.go 40.00% 10 Missing and 2 partials ⚠️
cmd/root.go 64.70% 5 Missing and 1 partial ⚠️
pkg/template/chunks.go 90.38% 4 Missing and 1 partial ⚠️
shared/aggregate.go 90.00% 3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@fahimfaisaal
fahimfaisaal merged commit 29ff383 into main Jun 14, 2026
1 check passed
@fahimfaisaal
fahimfaisaal deleted the feat/stats-panel branch June 14, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants