feat: stats panel, heatmap chart, async pipeline, and Go-stage chunk pruning#126
Merged
Conversation
…ine) Snapshot of in-progress async-worker refactor before reworking swap-axis to move all per-setting computation (incl. grouping) into the worker. Includes new ui/src/lib/swap.ts and temporary debug logs (removed during the rework). Spec: requirement-swap-async.md
Arrangement-aware, non-mutating grouping (identityKeys -> targetKeys) that generalizes useDataPoint.grouped. Foundation for moving grouping into the worker. Unused for now.
The worker now owns the full raw dataset and treats arrangement, group, sort, scale and showLabels all as compute params. After the one-time init clone (and again only on a dataset switch), no setting change does any main-thread data work — swap/group/sort/scale just post a param and render the returned ChartData. - transform.worker.ts: own raw rows; init/setArrangement project+group via projectAndGroup and reply with groupNames; compute reads grouped.get(group). Remove the broken swap message and swapAxisFields. - useChartPipeline.ts: rawData/arrangement/labels/groupName inputs; reinit only on dataset change; arrangement posts setArrangement; group/sort/scale/ labels recompute off the cache; expose groupNames. Drop triggerSwap. - useDataPoint.ts: drop the 500k main-thread grouping computeds; add per-dataset arrangement state + activeArrangement; source groups from the worker via setGroupNames; keep resultGroups/activeGroupId/selectGroup working. - Dashboard.vue: pass raw data + arrangement + derived labels + groupName; mirror the pipeline's groupNames into useDataPoint. - AxisSwapper.vue: set the arrangement key instead of mutating rows/labels. - swap.ts: drop the now-unused mutating swapAxisFields.
Groups are populated asynchronously by the worker on first `ready`.
Register a one-shot watcher (Vue 3.5 `{ once: true }`) on
`resultGroups.value.length` so the `?g=N` deep-link is applied once
groups arrive, without delaying `?d=` / settings which remain synchronous.
Replace groupName: Ref<string> param with activeGroupId: Ref<number> in useChartPipeline. Resolve the group name internally via currentGroupName() (reads from the pipeline's own groupNames ref), so pumpQueue() called inside the ready handler uses the correct name synchronously — without waiting for a downstream watcher to propagate it. Previously, groupNames [] → [...] on first ready caused activeGroupName to transition '' → groupNames[0] in a later microtask, which tripped the params watch and triggered a full second recompute on every page load and every swap. Now the params watch tracks activeGroupId (an integer), which does not change on the first ready, so only one compute batch is queued. User group-select (activeGroupId change) still triggers a single recompute as expected. Also removes the now-unused activeGroupName computed from useDataPoint, and collapses a double blank line in AxisSwapper.vue.
Switch dataSets to shallowRef+markRaw so rows stay plain objects; postMessage clones them natively instead of the JSON round-trip (~7300ms → ~700ms for 500k rows). Drop console.info(e) in the worker that serialized the full message to devtools on every init. Pre-populate chart slots in reinit() via listChartSignatures so per-card skeletons appear immediately while the worker clones and groups data. Remove the full-page LoadingSkeleton from the init phase — header and metadata are visible from the moment data loads; only chart slots pulse. Fix toStatSignature to handle absent per/unit fields without generating misleading signature strings.
- Off-thread stats worker (stats.worker.ts + useStatsWorker.ts) computes descriptive stats and correlation matrices lazily; WeakMap cache makes reopening the same chart instant - StatsPanel.vue: virtualized descriptive table (series × 20 metrics), diverging-heatmap correlation matrix (Pearson/Spearman), skeleton while loading, tab-aware CSV download button (RFC-4180, full precision) - ChartCard.vue: Sigma toggle button shows panel only when series exist - lib/stats.ts: mean/median/quantile/mode/variance/skew/kurtosis/mad/ describe/pearson/spearman/correlationMatrix/computeProfiles - lib/csv.ts: toCsvCell + descriptiveCsv + correlationCsv builders - Raise tsconfig lib to ES2023; replace slice().sort() with toSorted() in stats.ts (×2) and transform.ts (×1) - Add vitest 3.x; stats.test.ts (40 tests) + csv.test.ts (11 tests)
…raging
Go layer: AggregateDataPoints (shared/aggregate.go) sums DataPoints sharing
the same (name,xAxis,yAxis,zAxis) key. Fires automatically when parser is
csv/json and --group is active — 200k rows collapse to ~3k before JSON write.
Progress logs bracket the heavy passes ("Parsing...", "Aggregating N rows...").
UI layer: buildChartForSignature (2D) and cellsFor (3D) now average duplicates
instead of last-wins overwrite. Correct for benchmark count=N repeats; no-op
for CSV/JSON already pre-summed by Go.
Adds transform.test.ts (+20 tests) covering listChartSignatures, build3DRender,
projectAndGroup, and additional buildChartForSignature branches.
The dataZoom slider's left/right boundary labels rendered in ECharts' default gray, too dim against the dark canvas. Pass ChartStyling into createDataZoomConfig and set slider textStyle.color to styling.textColor so the labels match axis/legend text (#e5e7eb dark, #374151 light).
Click any column header in the stats panel's descriptive table to sort series rows by that metric. Tri-state cycle: descending -> ascending -> original series order, with an arrow indicator. Non-finite cells always sink to the bottom so a mostly-missing column still surfaces real min/max. Sort runs main-thread over already-computed profiles (no worker round-trip); virtualization and CSV export follow the sorted order.
Shows a debounced (300 ms) search input above the table when series count exceeds 20. Filters rows by case-insensitive name match on top of the existing column sort. Virtualization and CSV export follow the filtered result; search resets on chart change.
Cap search input at max-w-xs (20rem). Derive the series column header and search placeholder from chartData.axisLabels.x, falling back to "Series". Show a filtered/total count beside the input while a query is active.
Replace gap-2 with ml-auto so the filtered/total count sits at the right edge of the row rather than immediately beside the input.
Move the stats panel off the chart's critical path and make the correlation matrix pick the best axis to correlate along instead of always using the series axis. Non-blocking compute: - Dedicated stats worker now serves kinded requests (descriptive eager, correlation lazy) instead of one eager computeProfiles call, so opening a panel never blocks the chart and a tab computes only when first opened. - Per-ChartData piece cache in useStatsWorker; each tab shows its own loading skeleton. Smart correlation axis (x -> y -> z cascade): - selectCorrelationAxis prefers the series (x) axis, falls back to the category (y) axis, then the z axis (3D), taking the first whose entity count fits the 200 cap and which has >=3 observations. A chart with thousands of series but few categories now correlates the categories instead of showing nothing. - Generic buildCorrelationColumns fits the same correlationMatrix over any entity axis (y/z are transposes over the other two axes' tuples). - Correlation renders as an echarts canvas heatmap (ChartCorrelation + useCorrelationOption), with a caption naming the picked axis. CorrelationMatrix gains `axis`; worker/composable plumb zAxis through.
…sable Extract inline fullscreen logic from ChartCard into a shared useFullscreen composable (containerRef, isFullscreen, withFullscreenToolbox), then reuse it in ChartCorrelation to give the heatmap the same toolbox fullscreen toggle.
…on bottom-right - Add icon slot and ClassValue-typed class prop to Button component - Use Button component for Stats toggle - Move axis badges to header row (top-right) - Move Stats button below chart (bottom-right) - StatsPanel renders below the Stats button
… layout - Enable inside + slider dataZoom on both axes when length > 50 (LARGE_X_THRESHOLD) - x slider at bottom:55, visualMap horizontal centered below at bottom:5 - Fix y-axis label clipping: left:100px when largeX+containLabel:false - Reduce top padding for 2D heatmap (no legend): 5% instead of 15% - Axis name (nameGap:41) centered within dataZoom slider band
…ove entity cap - Merge ChartCorrelation.vue into ChartHeatmap.vue (single dumb renderer) - StatsPanel hosts fullscreen + saveAsImage toolbox via useFullscreen + createToolboxConfig (shared with all charts via getBaseOptions) - Add axis-selector dropdown (Selector.vue) to correlation tab; user can switch which axis supplies matrix entities; defaults to auto-pick (smallest) - Remove CORR_MAX_ENTITIES / CORR_MIN_OBSERVATIONS caps: any axis with ≥2 entities is usable; auto-pick = smallest (cheapest) axis, user can switch to any including large ones - Add dataZoom (inside + slider) to correlation heatmap when label count exceeds LARGE_X_THRESHOLD, matching regular heatmap behavior - Per-axis cache in useStatsWorker so switching axes recomputes and caches independently - Update stats.test.ts to reflect new uncapped axis-selection logic
Multiple useFullscreen instances (ChartCard + StatsPanel correlation) all responded to any fullscreenchange event via !!document.fullscreenElement. When ChartCard entered fullscreen, corrIsFullscreen also became true, overlaying the correlation div. Now each instance only tracks whether its own containerRef is the fullscreen element.
…ank on tab switch When user scrolled the descriptive table then switched tabs, scrollTop.value kept its old non-zero value while the remounted DOM element reset to 0. Returning to the descriptive tab produced a large empty topPad spacer until the next scroll event synced the state. measure() now reads scrollTop from the DOM so the watch([view, activeLoading]) call on remount corrects both scrollTop and viewportH together.
…_LABEL_MAX threshold
… log - Add SelectChunks (gated BFS) in pkg/template/chunks.go; GenerateUI now accepts charts+needs3D and prunes unreachable chunks at generation time - Add shared/chart_selection.go with DatasetNeeds3D for 3D engine gating - Extend --charts flag to include heatmap (ValidSet + SliceDefault) - Update vite.config.ts compress-bundle log to show raw → encoded (kB) instead of duplicating Vite's own gzip output - Docs: add Output File Size section to ui/index.mdx, Reducing Output Size to commands/root.mdx + ui.mdx, Prune step to internals/how-it-works.mdx - StatsPanel.vue and useSettingsStore.ts updates for stats panel branch
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StatsPanel.vuewith descriptive stats (20 metrics × all series), Pearson/Spearman correlation heatmap, virtualized table, searchable/sortable series filter, per-tab CSV export (RFC-4180), and lazy off-thread compute via dedicatedstats.worker.ts--charts heatmaptype;useHeatmapChartOptionswith dataZoom (inside + slider) on both axes when label count > 50, y-label clipping fix, visualMap layout below x-slider, added to--chartsvalid set and Go-stage chunk graphChartCorrelation.vueintoChartHeatmap.vue; user-selectable correlation axis viaSelector.vuedropdown; smart auto-pick (smallest axis, ≥2 entities, ≥3 observations); per-axis cache inuseStatsWorker; removedCORR_MAX_ENTITIES/CORR_MIN_OBSERVATIONScaps; always-show cell labelsshallowRef+markRaw(~7300ms → ~700ms for 500k rows)SelectChunks(gated BFS) inpkg/template/chunks.go;GenerateUIprunes unreachable JS chunks at generation time based on--charts+needs3D, shrinking output HTML for non-3D / non-heatmap runsbuildChartForSignatureandcellsForaverage duplicates instead of last-winsuseFullscreencomposable extracted; scoped percontainerRefso ChartCard and StatsPanel correlation don't cross-trigger; toolboxsaveAsImagewired viacreateToolboxConfig?g=deep-link param until worker groups are ready via one-shot Vue 3.5{ once: true }watchergroupName: Ref<string>toactiveGroupId: Ref<number>inuseChartPipeline; pre-populated chart slots vialistChartSignaturesfor immediate per-card skeletons