Skip to main content
Glama

litmus

The open behavioral litmus harness for MCP servers — grade A–F, reproducible.

Grade a server in one command

# -p is required: the package ships three bins, so npx must be told which to run
npx -y -p @polygraphso/litmus polygraphso-litmus litmus npm/@modelcontextprotocol/server-filesystem

Point it at an npm ref, a pypi ref, a github/owner/repo ref (cloned, built, and run sandboxed — Docker required; the grade pins the commit SHA), an https:// MCP endpoint, or a local entry file. The harness connects the way an agent would, fingerprints the exact tool surface, runs the four probe categories, and prints the grade with the findings behind it — plus a deterministic evidence bundle on disk. It runs the target's code (egress is Docker-sandboxed; without Docker, C-02 is skipped and the grade caps at B), takes ~20–60s, and exits non-zero on D/F so it scripts anywhere. To dispute any published grade, re-run this same command against the same server — open and deterministic means a re-run reproduces the grade, or refutes it.

Looking up a grade someone already published takes under a second and runs nothing: the polygraph.so index, or check_server from the MCP tools below.

litmus grading npm/@modelcontextprotocol/server-filesystem — five probe steps, then grade: A

Related MCP server: mcp-shield

Use it from your agent — MCP server + plugin

For grade lookups, point any MCP client at polygraph's hosted endpoint, no install:

claude mcp add --transport http polygraph https://polygraph.so/api/mcp

or the raw config:

{ "mcpServers": { "polygraph": { "url": "https://polygraph.so/api/mcp" } } }

This serves the lookup tools only (check_server, list_servers, request_grade); grading a server yourself (run_litmus, run_skill_litmus) executes its code, so it needs the local stdio install below.

The package also ships a stdio MCP server (polygraphso-litmus-mcp) with the full toolset, for any MCP-capable client:

  • check_server — read a server's published grade in under a second (no execution); the pre-flight check before recommending or installing a server.

  • list_servers — servers with a published grade, A first; paged (default 25 per call, with grade/limit/offset filters and a full-corpus summary).

  • request_grade — record a grade request with polygraph.so ($1 one-time fee; graded within 48h of payment — the response carries the payment link).

  • run_litmus — grade a server now: the full harness, grade + evidence returned to the agent.

  • run_skill_litmus — grade a Claude Code / Agent Skill (static scan, A/B/D/F).

  • verify_attestation — read the onchain proof behind a published grade (EAS on Base).

In Claude Code, the plugin wires the server plus two commands in one step:

/plugin marketplace add polygraphso/litmus
/plugin install polygraph@polygraphso

then /polygraph:grade <server> and /polygraph:check <server>. Cursor and manual JSON setups are on polygraph.so; full tool docs in packages/litmus/README.md.

Gate your CI on MCP grades — GitHub Action

Fail a build when an MCP server or an Agent Skill it ships grades D/F under the open behavioral litmus. For servers it is hybrid — a fast lookup of the published grade, then the harness when ungraded; for skills it is a fast static scan. Un-gradeable targets warn unless strict.

It's on the GitHub Marketplace as polygraphso/litmus@v1. For a security gate, pin to a commit SHA rather than the mutable @v1 tag:

# .github/workflows/mcp-gate.yml
name: mcp-gate
on: [pull_request]            # NOT pull_request_target — that exposes secrets to fork PRs
permissions:
  contents: read
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: polygraphso/litmus@<commit-sha>   # pin to a SHA; resolve from the v1 release
        with:
          # Name the targets explicitly (recommended). Grading runs a server's code,
          # so on a public repo prefer an allowlist over discovering PR-controlled config:
          servers: |
            npm/@modelcontextprotocol/server-filesystem
          skills: |
            ./my-skill
          # discover: "true"  # opt in to auto-discovery (.mcp.json/.vscode/.cursor) — trusted repos only
          # min-grade: B      # stricter than the default D/F gate
          # strict: "true"    # also fail on targets that cannot be graded

Inputs: servers · skills · discover (default false) · min-grade · strict · working-directory · version · bearer. Outputs: result · failed · report.

Security. Grading a server runs its code (egress is Docker-sandboxed, but it still executes). Trigger on pull_request, never pull_request_target. Keep discover off on public repos and name targets explicitly — auto-discovered config is pull-request-controllable. bearer is sent as an Authorization header to the target, so pass it only for an explicitly trusted, pinned remote — never with discovery or on untrusted PRs, and keep it scoped and short-lived.

Not on GitHub? The gate is a plain command — npx @polygraphso/[email protected] ci (pin the version) — so it runs in any CI or as a pre-commit hook. A grade is a measurement, not a guarantee: re-run the open harness to reproduce any result.

What litmus is

This is the source for @polygraphso/litmus, the open behavioral litmus harness for MCP servers from polygraph.so.

The harness connects to an MCP server the way an agent would, fingerprints its exact tool surface, and runs four probe categories — C-01 tool-output injection (static, dynamic, and second-order — one tool's output weaponized as another's input), C-02 permission/egress (in a hardened default-deny Docker sandbox, matched host and port), C-03 sensitive-data handling (planted canaries), C-04 adversarial-input handling (malformed/oversized and jailbreak inputs) — then grades the server A–F. A passing grade is a measurement, not a guarantee; the methodology and its disclosed limits are at polygraph.so (the open source here is the ground truth).

Alongside the grade, an npm target's dependency tree is checked against the osv.dev vulnerability database and any vulnerable dependencies are reported as dependency advisories. This is a separate, point-in-time signal — it is advisory only: it never affects the A–F grade and is not part of the reproducible evidence (vulnerability data changes over time, so folding it into the grade would break re-run reproducibility). It applies to npm targets only; other target kinds report it as skipped. Resolution runs npm install --package-lock-only --ignore-scripts, which resolves the tree without downloading tarballs or running any package code. Opt out with --no-deps-audit (or LITMUS_DEPS_AUDIT=0).

The same package also grades Claude Code / Agent Skills (a SKILL.md + bundle) under a separate static litmus (litmus-skill-v3): a deterministic byte-scan — S-01 prompt injection, S-03 data-exfiltration instructions, S-04 dangerous commands in the SKILL.md body or bundled scripts (incl. base64-obfuscated curl | bash) — graded A/B/D/F and anchored by a whole-directory content hash, plus a separate advisory quality signal. It is static (no execution): an A is static-clean, not behavioral proof. See packages/litmus/README.md.

The hosted, operator-run grading service is not in this repo — it lives in a separate private repo and consumes this package from npm like any other client.

Layout

This is a pnpm monorepo. Only @polygraphso/litmus is published; the @polygraph/* packages are private building blocks that tsup bundles into it.

packages/
  litmus/          # @polygraphso/litmus — the only published package (lib + 3 bins: CLI, skill CLI, MCP)
  core/            # contract types, canonical JSON, identity helpers
  probes/          # the harness: connect, fingerprint, grade, probe runners, sandbox
  onchain/         # EAS attestation read + encode/decode (Base) — read-only, no minting
  agent/           # agent-gate decision logic + live-fingerprint recheck
  mcp/             # MCP server wrapper
  cli/             # CLI commands + target/auth resolution
  demo-*-mcp/      # demo MCP servers used as test fixtures

See packages/litmus/README.md for the npm-facing usage docs, and polygraph.so for the methodology and proof format.

Develop

pnpm install
pnpm -r typecheck
pnpm -r test
pnpm --filter @polygraphso/litmus build   # → packages/litmus/dist

Release

@polygraphso/litmus is versioned in packages/litmus/package.json. Tag to publish:

git tag litmus-v<x.y.z> && git push origin litmus-v<x.y.z>

The Publish @polygraphso/litmus workflow builds, typechecks, tests, and publishes with npm provenance. See CONTRIBUTING.md for the full process and the local-development workflow for downstream consumers.

License

Apache-2.0 — © polygraph.so.

Available Tools

2 tools
run_skill_litmusRun a safety litmus on a Claude Code skillA
Read-only
Inspect

Grade a Claude Code / Agent Skill A/B/D/F against the open static safety litmus (litmus-skill-v1). A skill is a SKILL.md (instructions + frontmatter) plus an optional bundle. The litmus scans the bytes for S-01 prompt-injection / context-poisoning in the body, S-03 data-exfiltration instructions, and S-04 dangerous commands in bundled executable scripts. It content-hashes the whole directory (the anti-tamper anchor).

The SAFETY letter is a STATIC read: it does NOT execute the skill or its scripts and is fast — therefore NOT behavioral proof. An A means the static checks found no injection, exfil instruction, or dangerous bundled command, not that the skill is safe to run unsupervised. A command a skill constructs or fetches at runtime is not visible to static scanning (a disclosed limit).

It also returns a SEPARATE, advisory quality signal (well-formed / issues / malformed) — never an A–F letter, never minted, never affecting the safety letter. Its deterministic checks always run; its optional LLM-judged axes (honesty, coherence) run only when a judge is available — the host agent's own model via MCP sampling (no key), or a user-provided OpenAI-compatible key — and are skipped otherwise.

skill_ref (v1): a LOCAL path to a skill directory containing SKILL.md, e.g. ./skills/my-skill. Remote refs (github//#path, marketplace//) are not yet supported.

Inputs: skill_ref (required).

verify_attestationVerify a server's polygraph attestationA
Read-onlyIdempotent
Inspect

Read a server's already-published polygraph (litmus) grade — without running anything — before an agent trusts or, in agentic commerce, pays it.

When a grade is published it returns the behavioral grade (A–F), the attestation UID, the evidence CID, and the graded tool-surface fingerprint. The caller must still recompute the LIVE fingerprint and require it to equal the attested one before paying — a passing attestation can otherwise front for a tool surface the server no longer serves (rug pull).

Grade publishing is still rolling out, so this commonly returns not_available today: that means UNEVALUATED (neither safe nor unsafe), not a failing grade — to grade the server yourself right now, use run_litmus. A lookup_failed result means the lookup itself failed (the index or chain was unreachable); the grade is unknown, which is not the same as unevaluated.

Input: server_ref — e.g. npm/@modelcontextprotocol/server-filesystem.

Inputs: server_ref (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.3/5.0

    Scored across 2 tools

    Disambiguation5/5

    The two tools target distinctly different objects: one reads published attestations for servers, the other grades skills. Their descriptions clearly differentiate their purposes and use cases, leaving no ambiguity.

    Naming Consistency4/5

    Both tool names follow an underscore-separated verb_noun pattern, but the verbs differ (verify vs. run) and the nouns are not parallel (attestation vs. skill_litmus). Minor inconsistency, but overall predictable.

    Tool Count3/5

    With only 2 tools, the server is at the lower end of what is reasonable. While the scope is focused, the inclusion of a third tool (e.g., run_litmus for servers) would improve coverage without bloat.

    Completeness2/5

    The description of verify_attestation mentions a run_litmus tool for server grading that is not present, and there is no tool for publishing attestations. These gaps mean agents cannot perform the full workflow implied by the server's purpose.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      An MCP server that enables easy integration with coding assistants, providing security context to AI agents. This runs locally using the Snyk CLI.
      62
      Apache 2.0
    • A
      license
      A
      quality
      D
      maintenance
      Drop-in security primitives for MCP servers, addressing isError compliance, audit trails, input validation, output sanitization, and OAuth scoping.
      7
      MIT
    • A
      license
      B
      quality
      D
      maintenance
      Enables deterministic security testing of AI agents that use tools by serving synthetic MCP environments with poisoned data, fake secrets, and privileged actions. Records agent tool calls and evaluates security invariants (e.g., canary leaks, forbidden access, approval binding) without an LLM judge or real systems.
      8
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      A transparent MCP proxy that independently re-verifies tool-call claims instead of trusting them, paired with devmcp — the git/CI server it's proven against.
      1
      MIT