Skip to content

feat(cli): track which registry items add installs - #3099

Merged
miguel-heygen merged 2 commits into
mainfrom
feat-add-item-telemetry
Aug 7, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
feat-add-item-telemetry

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Aug 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

Emits one registry_item_added event per registry item hyperframes add writes into a project, so the catalog can be ranked by what people actually install.

Why

cli_command records that add ran and nothing about what it installed:

trackEvent("cli_command", { command, ...runIdField(runId) })

And the registry is served from raw.githubusercontent.com, so there is no server-side counter either. The result is that "which block or component is most installed" is unanswerable today — which also means there is no signal for what is worth building more of, or what is dead weight in a 50+ item catalog.

How

The event is emitted from runAdd, immediately after installAll returns. That is the single choke point for an item reaching a project:

  • The bulk add <tag> path re-enters runAdd once per item rather than installing separately, so tag installs are covered without a second call site.
  • Every failure — unknown item, wrong type, compatibility gate, a write that fails — throws before this line, so a refused install is never counted as a download.

requested distinguishes the item the user named from the transitive registryDependencies pulled in behind it. Collapsing the two would rank a popular dependency above everything that depends on it, which inverts exactly the question the event exists to answer.

Properties are item, item_type, requested.

Privacy

Item names are public registry identifiers — never project names, paths, or file content. The event goes through trackEvent, which returns early unless shouldTrack(), so an install that opted out via hyperframes telemetry disable, HYPERFRAMES_NO_TELEMETRY, or DO_NOT_TRACK sends nothing. No new opt-out surface, and none needed. The telemetry docs section is updated to name what is now collected.

Test plan

  • Unit tests added/updated
  • Manual testing performed
  • Documentation updated (if applicable) — the CLI telemetry section is updated; no new page

Two assertions in add.test.ts, both against the emitted payload rather than the transport (shouldTrack() is already false under test, so a real call and no call are indistinguishable):

  • Installing dep-block, which depends on base-component, emits both, with requested true only for the item that was asked for.
  • A compatibility-refused install emits nothing, folded into the existing gate test so the setup is not duplicated.

packages/cli suite green (315 tests across add + telemetry).

Verified end to end through the built CLI, with globalThis.fetch wrapped to capture the PostHog batch instead of sending it. Running from source proves nothing here: isDevMode() is true for a .ts entry, so shouldTrack() short-circuits and a real call looks exactly like no call. Faking a 200 in the hook keeps it off production analytics, since only a failed flush leaves events queued for the detached flushSync child that would bypass it.

  • add code-snippet-visual-studio-dark emitted {"item":"code-snippet-visual-studio-dark","item_type":"hyperframes:block","requested":true}.
  • add captions installed 16/16 and emitted exactly 16 events, one per item, item_type correct across 15 components and 1 block. This is the case worth seeing, since the tag path re-enters runAdd per item instead of having its own call site.
  • HYPERFRAMES_NO_TELEMETRY=1 and DO_NOT_TRACK=1: zero batches captured, not merely zero registry_item_added — no HTTP request at all.
  • An unknown item emitted nothing, exercising the refused-install branch in production code rather than a mock.

Not covered: no registry item declares registryDependencies today (grep -rl registryDependencies registry/ is empty), so requested: false cannot be exercised against the real registry and rests on the fixture-based unit test. Worth watching that the first item to take a dependency produces the expected split.

Also here: run in add.ts carries a bare fallow-ignore-next-line complexity. It is 28 cyclomatic, predates this change, and is untouched by it — Fallow scores it as new only because the file changed. Splitting its tag-fallback branch into a helper would fix it honestly, and is worth a follow-up, but not inside a telemetry change.

`cli_command` records that `add` ran and nothing about what it installed, and
the registry is served from raw.githubusercontent.com, which gives no per-item
counter either — so there is no way to tell which block or component people
actually pull, and no way to know what is worth building more of.

Emit one `registry_item_added` event per item written into a project, from
`runAdd` after the install succeeds. That is the single choke point: the bulk
`add <tag>` path re-enters it per item, and a failed or compatibility-refused
install throws before it, so a refused install is never counted as a download.

`requested` separates the item the user named from the transitive
`registryDependencies` pulled in behind it; without it a popular dependency
outranks everything that depends on it.

Item names are public registry identifiers, never user content or project data,
and the event goes through `trackEvent` — an install that opted out via
`hyperframes telemetry disable`, `HYPERFRAMES_NO_TELEMETRY` or `DO_NOT_TRACK`
sends nothing.
@mintlify

mintlify Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
hyperframes 🟢 Ready View Preview Aug 7, 2026, 10:10 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Closing the gap the test plan admitted: the event is now proven end to end through the built CLI, not just at the emit seam.

Running from source could never show this — isDevMode() returns true for a .ts entry, so shouldTrack() short-circuits and a real call is indistinguishable from no call. So I built the CLI and drove node dist/cli.js add with globalThis.fetch wrapped to capture the PostHog batch. Faking a 200 is what keeps it off production analytics: flush() only leaves events queued when the request fails, and only a non-empty queue makes the exit handler spawn the detached flushSync child that would bypass the hook. Nothing escaped.

Single item — add code-snippet-visual-studio-dark, 6 events in the batch, one of them:

{"item":"code-snippet-visual-studio-dark","item_type":"hyperframes:block","requested":true}

Bulk tag — add captions installed 16/16 and produced exactly 16 registry_item_added events, one per item, with item_type tracking correctly across the 15 components and 1 block:

caption-pill-karaoke        | hyperframes:component | requested: true
... 14 more components ...
mk-callout-highlight        | hyperframes:block     | requested: true

This is the case I most wanted to see, since the tag path re-enters runAdd per item rather than having its own call site.

Opt-out — same command with HYPERFRAMES_NO_TELEMETRY=1, and again with DO_NOT_TRACK=1: zero batches captured. Not "no registry_item_added" — no HTTP request at all.

Unknown item — the run that kicked this off was add dep-block (a test fixture, not a real registry entry). It failed as unknown-item and emitted nothing, which is the refused-install branch behaving correctly in production code rather than in a mock.

One honest gap this surfaced: no registry item currently declares registryDependencies (grep -rl registryDependencies registry/ is empty), so requested: false cannot be exercised against the real registry. It is covered only by the fixture-based unit test, where dep-block depends on base-component. Worth knowing that the field is correct-by-test but unobserved in the wild until the first item takes a dependency — and worth watching that the first one that does produces the split we expect.

The unit tests assert the emit seam and nothing past it. `shouldTrack()`
short-circuits whenever `isDevMode()` is true, and that is true for any `.ts`
entry, so under vitest a real event and no event are indistinguishable and the
transport is never exercised at all.

Drive the built CLI instead and assert on the HTTP body it actually produces:
one event per installed item, the dependency reported with `requested: false`,
an opted-out install sending no request at all (not merely one without this
event), and a refused install counting nothing.

Two fixtures, because neither case is reachable through the real registry. The
registry origin is a first-class project setting, so a local one supplies the
`registryDependencies` edge that no shipped catalog item declares today; and
`globalThis.fetch` is wrapped to capture the batch rather than send it. The
faked 200 is load-bearing: only a failed flush leaves events queued, and only a
non-empty queue spawns the detached `flushSync` child that would bypass the
hook and reach production analytics.

Verified the check can fail — forcing `requested: true` for every item turns it
red on exactly the dependency assertion.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Pushed af7d97db4 — the E2E coverage is now a checked-in test, not a run I did once.

scripts/ci/cli-telemetry-e2e.mjs, wired into the CLI smoke (required) job, which already builds the monorepo. It has to live there rather than in vitest: shouldTrack() short-circuits whenever isDevMode() is true, and that is true for any .ts entry, so under vitest a real event and no event are identical and nothing past trackEvent runs at all.

It closes the gap I flagged earlier. The requested: false branch is unreachable through the public registry because no catalog item declares registryDependencies — but the registry origin is a first-class project setting (hyperframes.json#registry), so the script serves a fixture registry with that edge and points a real project at it. Real CLI binary, real resolver, real installer, real transport; only the origin moves.

add e2e-dep-block (installs a dependency behind the requested item)
  ✓ install succeeded
  ✓ requested item written
  ✓ dependency written
  ✓ one event per installed item
  ✓ dependency reported, not as a request
  ✓ requested item reported as a request
opt out
  ✓ HYPERFRAMES_NO_TELEMETRY=1 installs
  ✓ HYPERFRAMES_NO_TELEMETRY=1 sends nothing
  ✓ DO_NOT_TRACK=1 installs
  ✓ DO_NOT_TRACK=1 sends nothing
unknown item
  ✓ install fails
  ✓ a refused install is not counted

The opt-out assertions are on zero batches, not zero matching events: an opted-out install must make no request at all, not a request that happens to omit this one event.

Nothing reaches PostHog. globalThis.fetch is wrapped to capture the batch, and the faked 200 is load-bearing rather than cosmetic — flush() only leaves events queued when the request fails, and only a non-empty queue makes the exit handler spawn the detached flushSync child that would bypass the hook.

And the check can actually fail. Forcing requested: true for every item, rebuilding, and re-running turns it red on exactly one assertion:

  ✗ dependency reported, not as a request
      got: {"item":"e2e-base-component","item_type":"hyperframes:component","requested":true}

Reverted and green again.

@miguel-heygen
miguel-heygen merged commit 0bda6b5 into main Aug 7, 2026
60 checks passed
@miguel-heygen
miguel-heygen deleted the feat-add-item-telemetry branch August 7, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant