Skip to content

Add filesystem hardening and evidence-aware usage reporting#6

Merged
thousandflowers merged 15 commits into
thousandflowers:mainfrom
brianluby:worktree-fix-usage-counting
Jun 18, 2026
Merged

Add filesystem hardening and evidence-aware usage reporting#6
thousandflowers merged 15 commits into
thousandflowers:mainfrom
brianluby:worktree-fix-usage-counting

Conversation

@brianluby

Copy link
Copy Markdown
Contributor
  • fix: correct skill usage counting and per-session dead-tool display
  • fix: harden mute, hook management, and low-activity cost estimate
  • refactor: drop unused frontmatter name in skill/agent scan
  • fix: confirm bulk mute and resolve named agents
  • fix: make destructive prune/mute operations recoverable
  • feat: write shared config files atomically
  • fix: harden the sqlite3 subprocess
  • fix: confine restore paths and tighten state-file permissions
  • fix: cap file sizes read during scan
  • fix: ignore rules symlinks that escape the rules tree
  • Harden pruning and release install paths
  • fix: harden mute/prune writes and centralize path-safety helpers
  • Address security review path-safety findings
  • fix: address automated PR review findings

brianluby added 14 commits June 17, 2026 09:48
- usage: classify a skill invocation as broken only when is_error is set.
  The substring scan for "error" in tool_result content misfired on
  successful output that merely mentioned the word (e.g. a linter reporting
  "no errors found"), wrongly producing a REAP(broken) verdict.
- report: DeadToolChars is a total summed across every session; divide it by
  the session count before the "per session" line, which had been inflating
  the figure by the number of sessions.
- report: round SessionsPerMonth and floor an active user to 1 so a
  low-frequency user (e.g. 1 session in 31 days) no longer gets a silent
  $0.00/month estimate from integer truncation.
- mute: strip a multi-line YAML description's continuation lines (block and
  folded scalars) instead of leaving malformed frontmatter behind; key each
  backup by a hash of the skill name so names that sanitize to the same
  string no longer clobber each other's backup.
- hook: match our marker as a trailing shell comment rather than a substring
  so a foreign hook that merely mentions the marker survives install and
  uninstall; shell-quote the executable path in the installed command.
- cmd: guard state-mutating commands (keep/restore/mute/unmute/install-hook/
  uninstall-hook) when no Claude directory is resolved, instead of writing
  state into the current working directory.
The invocation key has always been the directory/file name (plus any plugin
prefix) — that is how skills and agents are addressed in transcripts. The
frontmatter name was parsed into a variable that was then discarded
(staticcheck SA4006). Remove the dead read and document that the name field
is metadata, not the invocation key.
- mute: `reap mute <name>` now resolves agents as well as skills (findItem);
  previously the named path filtered to skills only, so `reap mute <agent>`
  always failed with "no skill found" even though `--all` muted agents.
- mute: bare `reap mute` and `reap mute --all` now preview the candidates and
  prompt [Y/n] (honoring --yes), matching prune. Before, a bare `reap mute`
  with a forgotten name silently stripped every eligible skill and agent.
- mute: skip already-muted items in bulk via errors.Is(ErrAlreadyMuted)
  instead of comparing the formatted error string.
- factor the prune/mute confirmation prompt into a shared confirm helper.
Each destructive op mutated a file before recording the action, so a failure
to write the manifest/state left the change with no way to undo it.

- QuarantineItem: roll the moved item back to its original location if the
  manifest write fails.
- RemoveMCP: read the manifest before editing the config, and restore the
  original config if the manifest write fails.
- Mute: restore the original SKILL.md if the mute-state write fails, so a
  skill is never left stripped with no recorded way to unmute it.
- RestoreAll / UnmuteAll: persist the items already restored before returning
  a mid-iteration error, instead of discarding that progress. UnmuteAll now
  restores in sorted order for deterministic behavior.
Add internal/atomicfile, which writes via a temp file in the same directory
followed by a rename, so a crash mid-write leaves the complete old or new file
rather than a truncated one. Use it for the externally-shared configs whose
corruption would break the user's Claude Code setup: settings.json (hook
install/uninstall) and the MCP config file (RemoveMCP edit, its rollback, and
restore). Internal reaped/ metadata keeps direct writes — it is small and
already protected by the prune/mute rollback paths.
The sqlite3 CLI was invoked with the parent environment and no time bound.

- Bound it with a 60s context timeout so a huge or pathological database
  cannot hang reap indefinitely.
- Pass -init os.DevNull so a hostile ~/.sqliterc cannot run shell commands or
  load extensions when the CLI starts.
- Run with a minimal environment (only TMPDIR) so an inherited PATH or HOME
  cannot influence which binary runs or which config it reads. The binary is
  already resolved to an absolute path via LookPath.
- prune: validate paths from the manifest before acting on them. File moves
  must stay within the Claude directory and MCP config writes within the home
  tree, so a tampered manifest cannot turn `reap restore` into an arbitrary
  file move or overwrite (e.g. ~/.ssh/authorized_keys).
- write reap's state files (manifest.json, mute state.json, nudge-state.json,
  overrides.json) with 0600 instead of world-readable 0644.
- override: write overrides.json atomically.
The scanners read each config/markdown file whole before parsing; the line
scanners only bound per-line size. A crafted multi-gigabyte file placed under
the scan root could OOM the process. Route every scan read through readCapped,
which refuses files larger than 10 MiB (far above any real skill, agent, hook,
plugin, or CLAUDE.md file) and is treated like any other read error.
ScanProse walked ~/.claude/rules and stat'd every *.md, following symlinks. A
planted symlink could make reap surface a file outside the rules tree (its path
and size) in the report. Resolve each entry and skip any that resolves outside
the rules directory.
- New internal/safepath package dedupes WithinDir (was 3 copies) and
  Sanitize (was 2 copies) across scan/prune/mute so the confinement
  boundary can't drift between packages.
- mute: write the live SKILL.md and backup via atomicfile.Write
  (preserving the original file mode, backup owner-only) so a crash no
  longer leaves a corrupt skill file; rollback now restores the original
  and removes the orphan backup, surfacing any rollback failure.
- prune: RemoveMCP now surfaces a rollback failure instead of swallowing
  it, so a modified config is never left without a manifest record.
- atomicfile: fsync the parent directory after rename for crash durability.
- usage: Parse sets IncompleteEvidence on mid-walk read/stat errors and
  continues, instead of silently swallowing the failure.
- report: truncate() is now rune-aware so multibyte skill names don't
  emit invalid UTF-8; document the plugin/personal suffix collision in
  lookupUses.
Copilot AI review requested due to automatic review settings June 18, 2026 07:59
@brianluby

Copy link
Copy Markdown
Contributor Author

I really like the tool but before I recommended it to my org and others I did a code review and found several things that needed to be fixed. The PR has been reviewed by GH Copilot and Coderabbit. skillreaper really opened my eyes to just how much tokens skills can eat up. Thanks for this.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens skillreaper’s filesystem interactions (path confinement, symlink handling, atomic writes, and permission tightening) and makes usage-based decisions evidence-aware (so partial/failed transcript parsing can’t justify REAP/MUTE). It also strengthens the npm postinstall flow by pinning downloads to the package version and verifying release assets via checksums.

Changes:

  • Add shared atomic write + path-safety helpers, apply them across prune/mute/hook/state/config writes, and tighten file permissions.
  • Make transcript parsing (JSONL + OpenCode SQLite) bounded/streamed and propagate “incomplete evidence” so absence-of-use decisions become REVIEW.
  • Pin npm release downloads to the package version tag and verify sha256 via checksums.txt, with new Node test coverage.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
npm/test/release.test.js Adds tests for version-pinned URLs, checksum parsing, and sha256 verification.
npm/package.json Adds node --test script for the npm package.
npm/lib/release.js Implements pinned GitHub release URLs + checksum-based asset verification helpers.
npm/install.js Uses verified release asset download + safer subprocess invocation for install/fallback build.
npm/bin/skillreaper.js Updates runtime downloader to use verified assets + safer command execution.
internal/usage/usage.go Adds IncompleteEvidence and avoids false error detection from "error" substring in output.
internal/usage/usage_test.go Tests overlong line handling + “mentions error but succeeds” behavior.
internal/usage/sqlite.go Streams sqlite3 output with timeouts/output caps and returns partial stats on bounded failures.
internal/usage/sqlite_test.go Adds tests for multiline content, overlong rows, and output-limit partial stats.
internal/scan/skills.go Stops using frontmatter name as the invocation key; adds path confinement checks and capped reads.
internal/scan/skills_test.go Adds tests for plugin install path escapes and symlink escape scenarios.
internal/scan/prose.go Skips rules roots/entries that escape the rules tree via symlinks; adds warnings.
internal/scan/prose_test.go Tests rule inclusion and symlink-escape skipping behaviors.
internal/scan/plugins.go Hardens plugin registry reading (non-symlink root, capped read, installPath confinement).
internal/scan/mcp.go Uses capped reads for config and plugin .mcp.json.
internal/scan/io.go Introduces readCapped and resolveWithin helpers to bound reads and confine paths.
internal/scan/io_test.go Tests capped reads and oversized skill skip behavior.
internal/scan/hooks.go Uses capped reads for hook settings files.
internal/scan/claudemd.go Uses capped reads for CLAUDE.md discovery.
internal/scan/agents.go Stops using frontmatter name as the invocation key; adds path confinement checks and capped reads.
internal/safepath/safepath.go Adds shared confinement + safe atomic write helpers used by mutating operations.
internal/safepath/safepath_test.go Adds tests for confinement/sanitization and write/read rejection cases.
internal/report/report.go Rounds sessions/month and holds REAP/MUTE at REVIEW for evidence-blind platforms; adds documented lookup tradeoff.
internal/report/report_test.go Adds tests for low-activity sessions/month and evidence-blind MUTE/REAP behavior.
internal/report/render.go Fixes per-session dead-tool display and makes truncation UTF-8-safe.
internal/report/render_test.go Tests per-session dead-tool display and UTF-8-safe truncation.
internal/prune/prune.go Makes prune operations recoverable via atomic writes + rollback and adds restore path confinement.
internal/prune/prune_test.go Adds rollback, confinement, and partial-progress persistence tests.
internal/override/override.go Switches override file writes to atomic writes with tighter permissions.
internal/mute/mute.go Adds path validation, unique backup naming, rollback on state failure, deterministic unmute-all ordering.
internal/mute/mute_test.go Adds extensive hardening and rollback/confinement test coverage for mute/unmute.
internal/hook/hook.go Quotes executable path in hook command, avoids marker substring false-positives, uses atomic writes for state.
internal/hook/hook_test.go Adds tests for quoting and foreign-marker substring handling.
internal/atomicfile/atomicfile.go Introduces atomic file writer with fsync of file + directory.
internal/atomicfile/atomicfile_test.go Tests atomic writer behavior, perms, and failure semantics.
cmd/reap/main.go Adds requireClaudeDir, evidence-blind improvements, bulk mute confirmation, and safer item lookup.
cmd/reap/main_test.go Adds tests for state-command guardrails, mute UX, incomplete evidence behavior, and suffix ambiguity.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/usage/sqlite.go Outdated
@thousandflowers thousandflowers merged commit 797b879 into thousandflowers:main Jun 18, 2026
@thousandflowers

Copy link
Copy Markdown
Owner

Thank you so much for this.
Doing a full code review before adopting it, and running it through Copilot and CodeRabbit on top, is way beyond what I'd expect from a PR. The hardening work here is substantial: path confinement, symlink handling, atomic writes, the sqlite subprocess sandboxing, checksum verification on releases these are real improvements, not cosmetic ones.

Really glad skillreaper has been useful to you, and the note about token usage made my day. Merged. Thanks again, and feel free to open an issue if anything comes up as you roll it out to your org.

@brianluby

Copy link
Copy Markdown
Contributor Author

Thank you so much for this. Doing a full code review before adopting it, and running it through Copilot and CodeRabbit on top, is way beyond what I'd expect from a PR. The hardening work here is substantial: path confinement, symlink handling, atomic writes, the sqlite subprocess sandboxing, checksum verification on releases these are real improvements, not cosmetic ones.

Really glad skillreaper has been useful to you, and the note about token usage made my day. Merged. Thanks again, and feel free to open an issue if anything comes up as you roll it out to your org.

Thanks for the response. I genuinely am excited for this project. It hits a nail on the head that I don't think most teams understand till they are hitting budgets. I look forward to contributing more towards this project. I had some feature ideas I came up with while doing these updates and with projects I have. I'll create feature ideas in issues for us to review.

@thousandflowers

Copy link
Copy Markdown
Owner

Love it please do open those issues, I'm keen to see what you've got in mind.

Happy to talk through any of them before you start building if that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants