Add filesystem hardening and evidence-aware usage reporting#6
Conversation
brianluby
commented
Jun 18, 2026
- fix: correct skill usage counting and per-session dead-tool display
- fix: harden mute, hook management, and low-activity cost estimate
- refactor: drop unused frontmatter name in skill/agent scan
- fix: confirm bulk mute and resolve named agents
- fix: make destructive prune/mute operations recoverable
- feat: write shared config files atomically
- fix: harden the sqlite3 subprocess
- fix: confine restore paths and tighten state-file permissions
- fix: cap file sizes read during scan
- fix: ignore rules symlinks that escape the rules tree
- Harden pruning and release install paths
- fix: harden mute/prune writes and centralize path-safety helpers
- Address security review path-safety findings
- fix: address automated PR review findings
- usage: classify a skill invocation as broken only when is_error is set. The substring scan for "error" in tool_result content misfired on successful output that merely mentioned the word (e.g. a linter reporting "no errors found"), wrongly producing a REAP(broken) verdict. - report: DeadToolChars is a total summed across every session; divide it by the session count before the "per session" line, which had been inflating the figure by the number of sessions.
- report: round SessionsPerMonth and floor an active user to 1 so a low-frequency user (e.g. 1 session in 31 days) no longer gets a silent $0.00/month estimate from integer truncation. - mute: strip a multi-line YAML description's continuation lines (block and folded scalars) instead of leaving malformed frontmatter behind; key each backup by a hash of the skill name so names that sanitize to the same string no longer clobber each other's backup. - hook: match our marker as a trailing shell comment rather than a substring so a foreign hook that merely mentions the marker survives install and uninstall; shell-quote the executable path in the installed command. - cmd: guard state-mutating commands (keep/restore/mute/unmute/install-hook/ uninstall-hook) when no Claude directory is resolved, instead of writing state into the current working directory.
The invocation key has always been the directory/file name (plus any plugin prefix) — that is how skills and agents are addressed in transcripts. The frontmatter name was parsed into a variable that was then discarded (staticcheck SA4006). Remove the dead read and document that the name field is metadata, not the invocation key.
- mute: `reap mute <name>` now resolves agents as well as skills (findItem); previously the named path filtered to skills only, so `reap mute <agent>` always failed with "no skill found" even though `--all` muted agents. - mute: bare `reap mute` and `reap mute --all` now preview the candidates and prompt [Y/n] (honoring --yes), matching prune. Before, a bare `reap mute` with a forgotten name silently stripped every eligible skill and agent. - mute: skip already-muted items in bulk via errors.Is(ErrAlreadyMuted) instead of comparing the formatted error string. - factor the prune/mute confirmation prompt into a shared confirm helper.
Each destructive op mutated a file before recording the action, so a failure to write the manifest/state left the change with no way to undo it. - QuarantineItem: roll the moved item back to its original location if the manifest write fails. - RemoveMCP: read the manifest before editing the config, and restore the original config if the manifest write fails. - Mute: restore the original SKILL.md if the mute-state write fails, so a skill is never left stripped with no recorded way to unmute it. - RestoreAll / UnmuteAll: persist the items already restored before returning a mid-iteration error, instead of discarding that progress. UnmuteAll now restores in sorted order for deterministic behavior.
Add internal/atomicfile, which writes via a temp file in the same directory followed by a rename, so a crash mid-write leaves the complete old or new file rather than a truncated one. Use it for the externally-shared configs whose corruption would break the user's Claude Code setup: settings.json (hook install/uninstall) and the MCP config file (RemoveMCP edit, its rollback, and restore). Internal reaped/ metadata keeps direct writes — it is small and already protected by the prune/mute rollback paths.
The sqlite3 CLI was invoked with the parent environment and no time bound. - Bound it with a 60s context timeout so a huge or pathological database cannot hang reap indefinitely. - Pass -init os.DevNull so a hostile ~/.sqliterc cannot run shell commands or load extensions when the CLI starts. - Run with a minimal environment (only TMPDIR) so an inherited PATH or HOME cannot influence which binary runs or which config it reads. The binary is already resolved to an absolute path via LookPath.
- prune: validate paths from the manifest before acting on them. File moves must stay within the Claude directory and MCP config writes within the home tree, so a tampered manifest cannot turn `reap restore` into an arbitrary file move or overwrite (e.g. ~/.ssh/authorized_keys). - write reap's state files (manifest.json, mute state.json, nudge-state.json, overrides.json) with 0600 instead of world-readable 0644. - override: write overrides.json atomically.
The scanners read each config/markdown file whole before parsing; the line scanners only bound per-line size. A crafted multi-gigabyte file placed under the scan root could OOM the process. Route every scan read through readCapped, which refuses files larger than 10 MiB (far above any real skill, agent, hook, plugin, or CLAUDE.md file) and is treated like any other read error.
ScanProse walked ~/.claude/rules and stat'd every *.md, following symlinks. A planted symlink could make reap surface a file outside the rules tree (its path and size) in the report. Resolve each entry and skip any that resolves outside the rules directory.
- New internal/safepath package dedupes WithinDir (was 3 copies) and Sanitize (was 2 copies) across scan/prune/mute so the confinement boundary can't drift between packages. - mute: write the live SKILL.md and backup via atomicfile.Write (preserving the original file mode, backup owner-only) so a crash no longer leaves a corrupt skill file; rollback now restores the original and removes the orphan backup, surfacing any rollback failure. - prune: RemoveMCP now surfaces a rollback failure instead of swallowing it, so a modified config is never left without a manifest record. - atomicfile: fsync the parent directory after rename for crash durability. - usage: Parse sets IncompleteEvidence on mid-walk read/stat errors and continues, instead of silently swallowing the failure. - report: truncate() is now rune-aware so multibyte skill names don't emit invalid UTF-8; document the plugin/personal suffix collision in lookupUses.
|
I really like the tool but before I recommended it to my org and others I did a code review and found several things that needed to be fixed. The PR has been reviewed by GH Copilot and Coderabbit. skillreaper really opened my eyes to just how much tokens skills can eat up. Thanks for this. |
There was a problem hiding this comment.
Pull request overview
This PR hardens skillreaper’s filesystem interactions (path confinement, symlink handling, atomic writes, and permission tightening) and makes usage-based decisions evidence-aware (so partial/failed transcript parsing can’t justify REAP/MUTE). It also strengthens the npm postinstall flow by pinning downloads to the package version and verifying release assets via checksums.
Changes:
- Add shared atomic write + path-safety helpers, apply them across prune/mute/hook/state/config writes, and tighten file permissions.
- Make transcript parsing (JSONL + OpenCode SQLite) bounded/streamed and propagate “incomplete evidence” so absence-of-use decisions become REVIEW.
- Pin npm release downloads to the package version tag and verify sha256 via
checksums.txt, with new Node test coverage.
Reviewed changes
Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| npm/test/release.test.js | Adds tests for version-pinned URLs, checksum parsing, and sha256 verification. |
| npm/package.json | Adds node --test script for the npm package. |
| npm/lib/release.js | Implements pinned GitHub release URLs + checksum-based asset verification helpers. |
| npm/install.js | Uses verified release asset download + safer subprocess invocation for install/fallback build. |
| npm/bin/skillreaper.js | Updates runtime downloader to use verified assets + safer command execution. |
| internal/usage/usage.go | Adds IncompleteEvidence and avoids false error detection from "error" substring in output. |
| internal/usage/usage_test.go | Tests overlong line handling + “mentions error but succeeds” behavior. |
| internal/usage/sqlite.go | Streams sqlite3 output with timeouts/output caps and returns partial stats on bounded failures. |
| internal/usage/sqlite_test.go | Adds tests for multiline content, overlong rows, and output-limit partial stats. |
| internal/scan/skills.go | Stops using frontmatter name as the invocation key; adds path confinement checks and capped reads. |
| internal/scan/skills_test.go | Adds tests for plugin install path escapes and symlink escape scenarios. |
| internal/scan/prose.go | Skips rules roots/entries that escape the rules tree via symlinks; adds warnings. |
| internal/scan/prose_test.go | Tests rule inclusion and symlink-escape skipping behaviors. |
| internal/scan/plugins.go | Hardens plugin registry reading (non-symlink root, capped read, installPath confinement). |
| internal/scan/mcp.go | Uses capped reads for config and plugin .mcp.json. |
| internal/scan/io.go | Introduces readCapped and resolveWithin helpers to bound reads and confine paths. |
| internal/scan/io_test.go | Tests capped reads and oversized skill skip behavior. |
| internal/scan/hooks.go | Uses capped reads for hook settings files. |
| internal/scan/claudemd.go | Uses capped reads for CLAUDE.md discovery. |
| internal/scan/agents.go | Stops using frontmatter name as the invocation key; adds path confinement checks and capped reads. |
| internal/safepath/safepath.go | Adds shared confinement + safe atomic write helpers used by mutating operations. |
| internal/safepath/safepath_test.go | Adds tests for confinement/sanitization and write/read rejection cases. |
| internal/report/report.go | Rounds sessions/month and holds REAP/MUTE at REVIEW for evidence-blind platforms; adds documented lookup tradeoff. |
| internal/report/report_test.go | Adds tests for low-activity sessions/month and evidence-blind MUTE/REAP behavior. |
| internal/report/render.go | Fixes per-session dead-tool display and makes truncation UTF-8-safe. |
| internal/report/render_test.go | Tests per-session dead-tool display and UTF-8-safe truncation. |
| internal/prune/prune.go | Makes prune operations recoverable via atomic writes + rollback and adds restore path confinement. |
| internal/prune/prune_test.go | Adds rollback, confinement, and partial-progress persistence tests. |
| internal/override/override.go | Switches override file writes to atomic writes with tighter permissions. |
| internal/mute/mute.go | Adds path validation, unique backup naming, rollback on state failure, deterministic unmute-all ordering. |
| internal/mute/mute_test.go | Adds extensive hardening and rollback/confinement test coverage for mute/unmute. |
| internal/hook/hook.go | Quotes executable path in hook command, avoids marker substring false-positives, uses atomic writes for state. |
| internal/hook/hook_test.go | Adds tests for quoting and foreign-marker substring handling. |
| internal/atomicfile/atomicfile.go | Introduces atomic file writer with fsync of file + directory. |
| internal/atomicfile/atomicfile_test.go | Tests atomic writer behavior, perms, and failure semantics. |
| cmd/reap/main.go | Adds requireClaudeDir, evidence-blind improvements, bulk mute confirmation, and safer item lookup. |
| cmd/reap/main_test.go | Adds tests for state-command guardrails, mute UX, incomplete evidence behavior, and suffix ambiguity. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Thank you so much for this. Really glad skillreaper has been useful to you, and the note about token usage made my day. Merged. Thanks again, and feel free to open an issue if anything comes up as you roll it out to your org. |
Thanks for the response. I genuinely am excited for this project. It hits a nail on the head that I don't think most teams understand till they are hitting budgets. I look forward to contributing more towards this project. I had some feature ideas I came up with while doing these updates and with projects I have. I'll create feature ideas in issues for us to review. |
|
Love it please do open those issues, I'm keen to see what you've got in mind. Happy to talk through any of them before you start building if that's useful. |