Skip to content

feat: async chart pipeline with worker-owned grouping and perf improvements#125

Merged
fahimfaisaal merged 9 commits into
mainfrom
feat/data-zoom-large-x
Jun 11, 2026
Merged

feat: async chart pipeline with worker-owned grouping and perf improvements#125
fahimfaisaal merged 9 commits into
mainfrom
feat/data-zoom-large-x

Conversation

@fahimfaisaal

@fahimfaisaal fahimfaisaal commented Jun 11, 2026

Copy link
Copy Markdown
Member

Summary

  • Add x-axis data zoom for large-scale datasets
  • Move grouping, arrangement, and all per-setting computation into the transform worker — swap/group/sort/scale post a param and render the returned ChartData with no main-thread data work
  • Eliminate proxy clone cost: shallowRef+markRaw for datasets, native postMessage clone replaces JSON round-trip (~7300ms → ~700ms for 500k rows)
  • Eliminate double-recompute on first ready: switch groupName: Ref<string>activeGroupId: Ref<number> so the params watcher doesn't fire on group-name propagation
  • Fix ?g= URL param to defer until worker groups are ready (Vue 3.5 { once: true } watcher)
  • Show per-card init skeletons immediately; drop full-page skeleton during init

Key changes

  • useChartPipeline.ts — dual epoch (dataEpoch/jobEpoch), reinit/setArrangement/recompute paths, expose groupNames
  • transform.worker.ts — owns raw rows; init/setArrangement reply with groupNames; compute reads from internal cache
  • useDataPoint.ts — drop 500k main-thread grouping computeds; source groups from worker via setGroupNames
  • transform.ts — add pure projectAndGroup helper
  • useUrlRouter.ts — one-shot watcher for ?g= param
  • AxisSwapper.vue — set arrangement key, no row mutation
  • chartConfig.ts — x data zoom threshold config

Test Plan

  • Load a large dataset (500k+ rows) and verify zoom controls appear on x-axis
  • Swap axes — charts rerender without full-page flash
  • Change group — single recompute (no double-fire in devtools)
  • Load page with ?g=1 — correct group selected after worker ready
  • Per-card skeletons visible during init, no full-page loading spinner
  • go test ./... passes

…ine)

Snapshot of in-progress async-worker refactor before reworking swap-axis to
move all per-setting computation (incl. grouping) into the worker. Includes
new ui/src/lib/swap.ts and temporary debug logs (removed during the rework).
Spec: requirement-swap-async.md
Arrangement-aware, non-mutating grouping (identityKeys -> targetKeys) that
generalizes useDataPoint.grouped. Foundation for moving grouping into the
worker. Unused for now.
The worker now owns the full raw dataset and treats arrangement, group,
sort, scale and showLabels all as compute params. After the one-time init
clone (and again only on a dataset switch), no setting change does any
main-thread data work — swap/group/sort/scale just post a param and render
the returned ChartData.

- transform.worker.ts: own raw rows; init/setArrangement project+group via
  projectAndGroup and reply with groupNames; compute reads grouped.get(group).
  Remove the broken swap message and swapAxisFields.
- useChartPipeline.ts: rawData/arrangement/labels/groupName inputs; reinit
  only on dataset change; arrangement posts setArrangement; group/sort/scale/
  labels recompute off the cache; expose groupNames. Drop triggerSwap.
- useDataPoint.ts: drop the 500k main-thread grouping computeds; add per-dataset
  arrangement state + activeArrangement; source groups from the worker via
  setGroupNames; keep resultGroups/activeGroupId/selectGroup working.
- Dashboard.vue: pass raw data + arrangement + derived labels + groupName;
  mirror the pipeline's groupNames into useDataPoint.
- AxisSwapper.vue: set the arrangement key instead of mutating rows/labels.
- swap.ts: drop the now-unused mutating swapAxisFields.
Groups are populated asynchronously by the worker on first `ready`.
Register a one-shot watcher (Vue 3.5 `{ once: true }`) on
`resultGroups.value.length` so the `?g=N` deep-link is applied once
groups arrive, without delaying `?d=` / settings which remain synchronous.
Replace groupName: Ref<string> param with activeGroupId: Ref<number> in
useChartPipeline. Resolve the group name internally via currentGroupName()
(reads from the pipeline's own groupNames ref), so pumpQueue() called inside
the ready handler uses the correct name synchronously — without waiting for a
downstream watcher to propagate it.

Previously, groupNames [] → [...] on first ready caused activeGroupName to
transition '' → groupNames[0] in a later microtask, which tripped the params
watch and triggered a full second recompute on every page load and every swap.
Now the params watch tracks activeGroupId (an integer), which does not change
on the first ready, so only one compute batch is queued. User group-select
(activeGroupId change) still triggers a single recompute as expected.

Also removes the now-unused activeGroupName computed from useDataPoint, and
collapses a double blank line in AxisSwapper.vue.
Switch dataSets to shallowRef+markRaw so rows stay plain objects;
postMessage clones them natively instead of the JSON round-trip
(~7300ms → ~700ms for 500k rows). Drop console.info(e) in the worker
that serialized the full message to devtools on every init.

Pre-populate chart slots in reinit() via listChartSignatures so per-card
skeletons appear immediately while the worker clones and groups data.
Remove the full-page LoadingSkeleton from the init phase — header and
metadata are visible from the moment data loads; only chart slots pulse.

Fix toStatSignature to handle absent per/unit fields without generating
misleading signature strings.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fahimfaisaal
fahimfaisaal merged commit 1013c16 into main Jun 11, 2026
1 check passed
@fahimfaisaal
fahimfaisaal deleted the feat/data-zoom-large-x branch June 11, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants