Conversation
Adds the foundation for v0.14.2's bulk-action progress streaming work: - src/core/progress.ts: dependency-free reporter with auto/human/json/quiet modes, TTY-aware rendering, time+item rate gating, heartbeat helper for slow single queries, dot-composed child phases, EPIPE defense (both sync throw and async 'error' event), and a singleton module-level signal coordinator so SIGINT/SIGTERM emits abort events for all live phases without leaking per-instance listeners. - src/core/cli-options.ts: parseGlobalFlags() for --quiet / --progress-json / --progress-interval=<ms> (both space and = forms), plus cliOptsToProgressOptions() that resolves to the right mode. Non-TTY default is human-plain one-line-per-event; JSON is explicit opt-in so shell pipelines don't suddenly see structured noise. - test/progress.test.ts (17 cases): mode resolution, rate gating, no-fake- totals on heartbeat paths, EPIPE paths, SIGINT singleton, child phase composition. - test/cli-options.test.ts (14 cases): flag parsing, invalid values, interleaved flags, mode resolution. Follow-ups wire doctor/embed/files/export/extract/import/sync/migrate/ repair-jsonb/backlinks/orphans/lint/integrity/eval/autopilot/jobs plus the apply-migrations orchestrators through this reporter, and route Minion handlers to job.updateProgress instead of stderr. See the plan in ~/.claude/plans/. 1682 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Parse --quiet / --progress-json / --progress-interval from argv BEFORE command dispatch, strip them, stash resolved CliOptions on a module-level singleton (same pattern as Commander's program.opts()) and on every OperationContext created for shared-op dispatch. - src/cli.ts: parseGlobalFlags(rawArgs) at the top of main(); setCliOptions once; dispatch sees only the stripped argv. Fixes the "gbrain --progress-json doctor" unknown-command case that Codex flagged. - src/core/cli-options.ts: expose setCliOptions/getCliOptions/ _resetCliOptionsForTest singleton. Commands that want progress call getCliOptions() to construct their reporter. - src/core/operations.ts: OperationContext gains optional cliOpts field so shared-op handlers (and MCP-invoked ops that need a reporter) can read the same settings. MCP callers leave it undefined and consumers default to quiet. - test/cli-options.test.ts: +4 cases covering singleton round-trip and an integration smoke spawning `bun src/cli.ts --progress-json --version` to prove the global flag survives dispatch. 45 relevant unit tests pass (progress + cli-options + cli.test.ts). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Doctor on a 52K-page brain used to sit silent for 10+ minutes while the DB checks ran, then get killed by an agent timeout. Wired through the new reporter so agents see which check is running and the slow ones heartbeat every second. doctor.ts: - Start a single `doctor.db_checks` phase around the DB section, with a per-check heartbeat before each step (connection, pgvector, rls, schema_version, embeddings, graph_coverage, integrity, jsonb_integrity, markdown_body_completeness). - jsonb_integrity now scans 5 targets, not 4: added page_versions. frontmatter so the check surface matches `repair-jsonb` (per Codex review of the plan — the old 4-target scan missed a known repair site). Per-target heartbeat so 50K-row scans show incremental progress. - markdown_body_completeness: wrap the existing query in a 1s heartbeat timer. The regex scan over rd.data ->> 'content' can't be paginated usefully; this just lets agents see life during the sequential scan. No fake totals — the LIMIT 100 query has no meaningful total count. - integrity sample: same heartbeat pattern around the 500-page scan. orphans.ts: - findOrphans() wraps the NOT EXISTS anti-join in a 1s heartbeat. Keyset pagination was considered and rejected: without an index on links.to_page_id it's no faster than the full scan, and may re-plan the anti-join per batch. A schema migration adding that index is the right fix and is queued for v0.14.3. Follow-ups: - Step 3b: wire embed/files/export (the \r-only stdout offenders). - Step 5: end-to-end progress test spawning `gbrain doctor --progress-json` against a fixture brain, asserting stderr events and clean stdout. All existing unit tests continue to pass (76/76 in doctor + orphans + progress + cli-options). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Replaces the \r-on-stdout progress pattern in the three worst offenders
(embed, files sync, export) with the shared reporter on stderr. Stdout
now carries only final summaries, so scripts and tests that grep for
counts ("Embedded N chunks", "Files sync complete", "Exported N pages")
still work when output is piped.
- embed.ts: runEmbedCore accepts an optional onProgress callback. The
CLI wrapper builds a reporter and passes reporter.tick(); Minion
handlers will pass job.updateProgress in Step 4. Worker-pool is
single-threaded JS so no rate-gate race (per Codex review #18).
- files.ts syncFiles(): tick per file; summary preserved on stdout.
- export.ts: tick per page; summary preserved on stdout.
Also fixes a --quiet flag collision. `skillpack-check` has its own
--quiet mode (suppress all stdout). parseGlobalFlags strips --quiet
globally now, and skillpack-check reads the resolved CliOptions
singleton via getCliOptions() instead of re-parsing argv. Test updated
to match the stripping behavior.
1686 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extract, import, and sync now stream per-file progress to stderr through
the shared reporter. All three kept their stdout summaries + JSON
action-events intact so existing tests + agent scripts are unaffected.
- extract.ts (4 paths: links/timeline × fs/db): replaced the ad-hoc
`process.stderr.write({event:"progress"...})` lines with reporter
ticks. Same channel (stderr), canonical schema now, visible in both
text and --json modes. Stdout action-events (`add_link` /
`add_timeline`) untouched — tests grep them.
- import.ts: the logProgress() function that printed every 100 files to
stdout is now a progress.tick() call per file. Rate-gated by the
reporter. Stdout still gets the final "Import complete (Xs)" summary
and the --json payload.
- sync.ts: three new phases (`sync.deletes`, `sync.renames`,
`sync.imports`) tick per file, so big syncs show each step rather than
a single end-of-run summary. Phase hierarchy ready to be child()-chained
into runImport / runEmbed later, per Codex review #26.
Updated the #132 nested-transaction regression test in test/sync.test.ts
to also accept the new hoisted-loop shape — the guarantee (this loop is
not wrapped in engine.transaction) still holds.
1686 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires the remaining bulk commands through the reporter: - migrate-engine: phase starts (migrate.copy_pages, migrate.copy_links), per-page tick. Old \"Progress: N/total\" stdout logs replaced by stderr ticks; final stdout summary preserved. - repair-jsonb: per-column start + a heartbeat timer while each UPDATE runs (minutes on 50K-row tables). CRITICAL: stdout stays clean so migrations/v0_12_2.ts's JSON.parse(child.stdout) still works. Per Codex review #12. - backlinks: 1s heartbeat around findBacklinkGaps() (sync double-walk of the brain dir). - lint: tick per page; per-issue stdout output preserved. - integrity auto: tick per page in the main resolver loop. The separate ~/.gbrain/integrity-progress.jsonl resume marker is untouched (its role shifts from live progress reporting to resume-only). - eval: add an onProgress option to core's runEval(), CLI wraps with a reporter. Phases: eval.single / eval.ab. Tick per query. core/search/eval.ts gains a RunEvalOptions type so future callers (MCP eval op, Minion handlers) can also hook in without the reporter. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- src/core/embedding.ts: embedBatch() gains an optional EmbedBatchOptions.onBatchComplete callback, fired after each 100-item sub-batch. CLI wrappers pass reporter.tick; Minion handlers can pass job.updateProgress. - src/core/enrichment-service.ts: enrichEntities() config gains onProgress(done, total, name) fired after each entity. Same split: CLI -> reporter, Minion -> DB-backed progress. No CLI behavior change on its own. Wiring these callbacks into the Minion handlers is Step 4. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- cli-options.ts: childGlobalFlags() returns the flag suffix to append to child gbrain subprocesses. Empty string by default, " --quiet --progress-json" when the parent has them set, so child behavior inherits the parent's progress-mode without scattering string-concat logic across every execSync site. - migrations/v0_12_2.ts: each execSync inherits the parent's global flags. Phase C (repair-jsonb --dry-run --json) pins explicit stdio to ['ignore','pipe','inherit'] so child stderr streams straight through while stdout stays captured for JSON.parse. Per Codex review #12. - migrations/v0_12_0.ts + v0_11_0.ts: same childGlobalFlags wiring at each gbrain-subcommand execSync. - upgrade.ts: post-upgrade timeout bumped 300s → 30min (1_800_000 ms) with GBRAIN_POST_UPGRADE_TIMEOUT_MS override. The old 300s cap killed v0.12.0 graph-backfill migrations on 50K+ brains; the heartbeat wiring added in v0.14.2 makes long waits observable, so a generous ceiling no longer means users stare at a silent terminal. - jobs.ts: the embed Minion handler passes job.updateProgress as the onProgress callback, so per-job progress is durable in minion_jobs and readable via `gbrain jobs get <id>`. Primary Minion progress channel is DB-backed — stderr from `jobs work` stays coarse for daemon liveness only. Per Codex review #20. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/check-progress-to-stdout.sh greps src/ for the banned
`process.stdout.write('\r…')` pattern that v0.14.2 removed from the
bulk-action codepaths. Wired into the `bun run test` script so any
future regression that puts progress back on stdout fails fast. An
empty allowlist documents the position: every known call site was
migrated; new exceptions need a rationale in the allowlist.
test/e2e/doctor-progress.test.ts (Tier 1, needs Postgres + pgvector):
- `gbrain --progress-json doctor --json`: stderr carries JSONL progress
events with the canonical {event, phase, ts} shape, starts + finishes
for `doctor.db_checks`. Stdout stays parseable JSON — no progress
pollution.
- `gbrain doctor` (no flag): human-plain progress goes to stderr only,
stdout stays free of `[doctor.db_checks]`.
- `gbrain --quiet doctor`: reporter emits nothing; doctor still runs to
completion.
test/cli-options.test.ts: +2 spawning integration tests. One verifies
`gbrain --progress-json --version` keeps stdout clean of progress events
(single-shot commands that don't use a reporter aren't affected). One
guards the skillpack-check --quiet regression — --quiet suppresses
stdout by reading the resolved CliOptions singleton, not re-parsing argv.
Full test matrix:
bun run test -> 1726 pass / 184 skipped (no DB) / 0 fail
bun run test:e2e -> 136 pass / 13 skipped / 0 fail
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- VERSION + package.json bumped to 0.14.2. - docs/progress-events.md (new): canonical JSON event schema reference. Stable from v0.14.2, additive only. Lists every phase name shipped in this release, the five event types (start/tick/heartbeat/finish/ abort), the TTY/non-TTY rendering rules, subprocess inheritance semantics, and the Minion DB-backed progress model. - CLAUDE.md: "Bulk-action progress reporting" section under the build instructions; Key files entries for src/core/progress.ts, src/core/cli-options.ts, scripts/check-progress-to-stdout.sh, and docs/progress-events.md; doctor.ts entry updated to note the v0.14.2 5-target jsonb_integrity scan + heartbeat wiring. - CHANGELOG.md v0.14.2: full release summary per project voice rules. The "numbers that matter" table, per-command before/after grid, backward-compat warnings for stdout→stderr moves, and an itemized changes section covering reporter/CLI plumbing/schema/Minion handlers/doctor fixes/upgrade timeout/CI guard/tests. No em dashes. Real file paths, real commands, real numbers. - skills/migrations/v0.14.2.md (new): agent migration note. Mechanical step is "nothing" since v0.14.2 is purely additive. Walks agents through the three new global flags, the 14 wired commands, the event schema cheat sheet, Minion progress via job.updateProgress, and scripts/verification commands. Full test matrix: bun run test (unit + guards) -> 1726 pass / 184 skipped / 0 fail bun run test:e2e (Postgres) -> 141 pass / 8 skipped / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Conflicts resolved:
- src/commands/doctor.ts: kept both progress imports (HEAD) + the
DbUrlSource import (master). Also kept HEAD's progress.heartbeat('graph_coverage')
line that master had dropped.
- src/commands/migrations/v0_12_0.ts + v0_12_2.ts: took master's
removal of appendCompletedMigration (Bug 3 moved ledger writes to the
runner). Kept HEAD's childGlobalFlags import so child subprocesses
still inherit the parent's --progress-json / --quiet mode.
- src/commands/sync.ts: merged HEAD's per-file reporter ticks
(sync.deletes / sync.renames / sync.imports phases) with master's
Bug 9 failedFiles tracking. Every failure path also pushes to
failedFiles; reporter.tick fires once per file regardless of outcome.
- CHANGELOG.md: combined both v0.14.2 sections into one coherent entry
covering the progress wave AND the 8 reliability bug fixes. Merged
"numbers that matter" tables, reconciled "To take advantage"
instructions into one ordered list, merged itemized changes into
unified sections.
No changes needed from linter-style adjustments on:
- CLAUDE.md (master edits + HEAD's Bulk-action progress section;
git auto-merged cleanly)
- package.json (version bump 0.14.2 came from both sides)
- src/cli.ts (HEAD's parseGlobalFlags + getCliOptions; master's edits
compatible, no conflict)
- src/commands/import.ts (HEAD's reporter + master's Bug 9 failedFiles
merged cleanly as modify-modify; progress stays, failure tracking stays)
- src/commands/jobs.ts (HEAD's embed handler onProgress callback and
master's edits to unrelated lines)
- src/commands/migrations/v0_11_0.ts (HEAD's childGlobalFlags import
and master's Bug 3 comment merged cleanly)
Test matrix after merge:
bun run test -> 1780 pass / 179 skipped (no DB) / 0 fail
bun run test:e2e -> 141 pass / 8 skipped / 0 fail
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Master sits at 0.14.2 (reliability wave). This PR lands on top as 0.15.2 (progress streaming wave). Splits the merge-time combined CHANGELOG entry back into two discrete release sections so history stays honest: - [0.15.2] = progress reporter, CliOptions, 14 wired commands, Minion embed handler, doctor jsonb_integrity 5-target fix, upgrade timeout bump, CI guard, progress unit+E2E tests. - [0.14.2] = master's eight root-cause bug fixes, restored verbatim from origin/master. Touched files: - VERSION + package.json: 0.14.2 -> 0.15.2 (next patch off master). - skills/migrations/v0.14.2.md -> skills/migrations/v0.15.2.md (rename + rewrite frontmatter + body to v0.15.2). - CHANGELOG.md: split into two entries; progress-wave refs renamed v0.14.2 -> v0.15.2; reliability-wave entry restored from master. - src/core/progress.ts, src/commands/doctor.ts, src/commands/sync.ts, src/commands/upgrade.ts, docs/progress-events.md, test/sync.test.ts: progress-wave v0.14.2 references -> v0.15.2. The remaining v0.14.2 references in test/e2e/migration-flow.test.ts (Bug 3 context) and CLAUDE.md (reliability-wave key commands, Bug 3 ledger move) correctly point at master's 0.14.2 release. Test matrix after version bump: bun run test -> 1780 pass / 179 skipped / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Conflicts resolved: - VERSION + package.json: kept 0.15.2 (our locked target). Merged master's additions: build:llms script, scripts/run-e2e.sh for test:e2e, new postinstall message + trustedDependencies whitelist, pglite pinned to 0.4.3 exactly. Kept our check-progress-to-stdout CI guard in test. - src/commands/doctor.ts: merged master's schema_version loud-fail check (#218 — reports when migrations never ran) with our progress.heartbeat scaffolding. Added a progress.heartbeat('index_audit') call to master's new --index-audit check so the new surface streams status like the rest. progress.finish() now runs after all 12 checks (was 11 before). - CLAUDE.md: kept both doctor.ts descriptions merged into one (progress + schema_version + --index-audit). Kept all our progress.ts / cli-options.ts / check-progress-to-stdout.sh / progress-events.md entries. Retagged the progress reporter as "Introduced in v0.15.2" and docs as "Stable from v0.15.2". - CHANGELOG.md: stacked releases in chronological order — [0.15.2] (us), [0.15.1] (master fix wave), [0.15.0] (master llms.txt + AGENTS.md), [0.14.2] (master reliability wave), [0.14.1] and below. Master additions auto-merged cleanly: - AGENTS.md, INSTALL_FOR_AGENTS.md, README.md, llms.txt, llms-full.txt, scripts/build-llms.ts, scripts/llms-config.ts (all new). - scripts/run-e2e.sh (sequential E2E driver, replaces parallel bun test). - scripts/check-jsonb-pattern.sh (extended to also guard max_stalled=1). - src/commands/jobs.ts (new --max-stalled / --backoff-* / --timeout-ms / --idempotency-key flags on `jobs submit`). - src/core/engine.ts, migrate.ts, minions/queue.ts, minions/types.ts, pglite-engine.ts, pglite-schema.ts, postgres-engine.ts, schema-embedded.ts, schema.sql (kind discriminator, migration v12/v13, max_stalled=5 schema default + backfill). - test/migrate.test.ts, minions.test.ts, migrations-v0_14_0.test.ts, pglite-engine.test.ts, build-llms.test.ts. Regenerated llms.txt + llms-full.txt (master's build-llms.test drift guard would fail otherwise) to pick up our CLAUDE.md edits. Test matrix after merge: bun run test -> 1806 pass / 179 skipped / 0 fail bun run test:e2e (Postgres) -> 141 pass / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gbrain doctoron a 52K-page brain sits silent for 10+ min then gets killed by agent timeouts; 14 bulk commands either write\rprogress to stdout (invisible under pipes) or emit nothing at all.src/core/progress.tsreporter +src/core/cli-options.tsglobal flags. Every bulk command streams progress to stderr via a canonical schema. Stdout stays clean for data output.job.updateProgress. CI guard wired intobun run test.How it works
--quiet,--progress-json,--progress-interval=<ms>) parsed before command dispatch, stored in a module-levelCliOptionssingleton, reachable by every command without parameter threading.\r-rewrite on TTY, plain one-line-per-event on non-TTY, JSONL when--progress-json). Rate-gated by both time (1s default) and item count. Handles EPIPE on both sync throw and stream'error'event. Singleton SIGINT/SIGTERM coordinator emitsabortevents for every live phase.docs/progress-events.mddocuments the{event, phase, ts}envelope + optional fields. Stable from v0.14.2, additive only.childGlobalFlags()helper propagates flags throughexecSync('gbrain ...')calls in the migration orchestrators.v0_12_2.tsuses explicit stdio so child stderr passes through while captured stdout stays JSON-parseable.job.updateProgressas theonProgresscallback; DB-backed progress is the primary channel for Minion work. Stderr fromjobs workstays coarse.Numbers
\r)jsonb_integrityscan targets (doctor)page_versions.frontmatter)repair-jsonbgbrain doctorsilence on 52K pagesTest plan
bun run test(unit + guard): 1731 pass / 179 skipped (no DB) / 0 failbun run test:e2e(Postgres+pgvector via Docker): 141 pass / 8 skipped / 0 failscripts/check-progress-to-stdout.shpasses (no\r-on-stdout anywhere in src/)gbrain --progress-json doctor --json 2>/tmp/p.log >/tmp/d.json;jq . /tmp/d.jsonparses cleanly,/tmp/p.loghas one JSON event per lineCommits (per-step, bisectable)
ea8170estep 1 — shared ProgressReporter + CliOptions (+ 31 unit tests)a3ad647step 2 — wire global flags into cli.ts88b2479step 3a — doctor + orphans heartbeat streaming3e5daeastep 3b — embed + files + export stderr progress3d77513step 3c — extract + import + sync reporter streaming4fa46cfstep 3d — migrate/repair/backlinks/lint/integrity/evale7be5d6step 3e — onProgress callbacks on core libs607c894step 4 — orchestrators + upgrade + minion handlers1459040step 5 — E2E doctor-progress test + CI guardb845ccfstep 6 — docs + v0.14.2 release bumpBackward-compat warnings (in CHANGELOG)
Progress for
embed,files,export,extract,import,migrate-enginemoved from stdout to stderr. Stdout now carries only final summaries and--jsonpayloads. Scripts that parsed stdout for progress lines need to read stderr instead.🤖 Generated with Claude Code