Skip to content

Auto-discover a shipped baseline so distributed skills carry their own suppressions #278

Closed

Description

@airadier

Summary

Please consider auto-discovering a baseline that ships alongside a skill, so a distributed skill can carry its own vetted false-positive/by-design suppressions and downstream scans surface only genuinely new findings — without every consumer needing to know and pass --baseline.

Context

We publish a set of Agent Skills and run SkillSpector both in our CI and as a recommended check for the people who install the skills. A large share of the findings are inherent to what the skills legitimately do and are documented, e.g.:

  • Threat-detection / incident-response docs that reference attacker techniques as material under investigation (IMDS 169.254.169.254, /etc/shadow, curl … | bash sample data, MITRE ATT&CK tactic tables).
  • An intentionally privileged host agent (--privileged, docker.sock) that host monitoring requires.
  • Credential auto-discovery from documented SYSDIG_* environment variables.

We maintain a reviewed .skillspector-baseline.yaml (rules: with a reason: on each) that suppresses these with explanations, so only new/actionable findings remain.

Problem

skillspector scan <skill> does not load a baseline unless --baseline <path> is passed explicitly (confirmed: a .skillspector-baseline.yaml placed next to a skill has no effect without the flag). When skills are distributed, the person scanning them generally won't know a curated baseline exists — and for a single-skill install there's no obvious repo-root file to point at. So consumers see all the known false positives raw, which erodes trust in the tool's output.

Proposal

Auto-discover a shipped baseline, e.g. one or more of:

  1. If <scanned-dir>/.skillspector-baseline.yaml (and/or a repo-root one) exists, load it by default — with a flag to disable (--no-baseline) and a clear report of which suppressions were applied and their reason: (so suppression stays transparent, not silent).
  2. A SKILL.md frontmatter field that points at or inlines the skill's suppressions.

This mirrors how most linters auto-load a config from the target directory, and it lets skill authors ship explained suppressions rather than asking each user to supply a flag.

Related (possible bug)

skillspector scan <dir> --recursive --baseline <file> appears to not apply the baseline (suppressed count stays 0 across the multi-skill run), whereas a single-skill skillspector scan <skill> --baseline <file> applies it correctly. If intended, worth documenting; if not, honoring --baseline in --recursive mode would help multi-skill repos.

Why it helps the ecosystem

Skill authors can ship vetted, human-explained suppressions with the skill, so any downstream SkillSpector run — not just the author's CI — surfaces only new or genuinely actionable findings. That improves signal for everyone and keeps the "suppressed with a documented reason" audit trail intact.

Happy to help test or contribute a PR if this direction is welcome.

Activity

  1. Spectorian commented on Jul 17, 2026

    @Spectorian
    Collaborator

    I’d separate the two asks. Recursive scans should honor an explicitly supplied baseline; that bug is already tracked in #201/#205. Auto-discovery raises a different trust question: a baseline shipped by the skill author should not silently suppress findings in a consumer’s scan. A safer direction may be to detect and surface the baseline without applying it, require explicit opt-in before using it, and clearly report that an author-supplied baseline was applied, including what it suppressed and why. Would that explicit-trust model work for your downstream users and non-interactive CI flows?

  2. airadier commented on Jul 18, 2026

    @airadier
    Author

    I’d separate the two asks. Recursive scans should honor an explicitly supplied baseline; that bug is already tracked in #201/#205. Auto-discovery raises a different trust question: a baseline shipped by the skill author should not silently suppress findings in a consumer’s scan. A safer direction may be to detect and surface the baseline without applying it, require explicit opt-in before using it, and clearly report that an author-supplied baseline was applied, including what it suppressed and why. Would that explicit-trust model work for your downstream users and non-interactive CI flows?

    Definitely that would work. So either either detect the baseline and apply it, but be verbose about an existing baseline has been applied, then add an option to opt-out, or clearly indicate that a author-provided baseline exists, and offer the option to opt-in.

    My main use case is that we are publishing a set of skills, and the scan is usually quite verbose and detects several FPs. The number of FPs could refrain a user from adopting the skills based on a scan result until the user can clearly find out understand why some of those findings should be acceptable per the skill purpose.

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions