🦀 crabpot

Goto: Latest Published | Latest Beta | Main Development

Compatibility trap for OpenClaw plugin contracts. crabpot keeps a curated set of real community plugins pinned under plugins/ and runs seam-focused compatibility checks against OpenClaw plugin APIs. The goal is to catch contract drift before external plugin authors do. Built on top of plugin-inspector, the testing harness for OpenClaw.

Reporting Data

main follows the latest published npm package and npm latest plugin artifacts, with bundled OpenClaw fixtures source-packed from the matching checkout. crab-beta follows beta npm dist-tags for externalized packages and source-packs bundled fixtures. crab-development checks openclaw/openclaw main against source-packed official plugin artifacts from that same OpenClaw checkout.

• Last dashboard update: Jul 16, 2026, 01:44 UTC

• Source: github-main
• OpenClaw version: 2026.7.2
• OpenClaw SHA: 6f22ef08a37d
• Dashboard target: openclaw/openclaw@main + source-packed @openclaw/*
• Plugin artifacts: source-packed from OpenClaw checkout
• GitHub report run: 29464722008

Dashboard

Metric	Result
Fixtures	8 -52 vs main
Hard breakages	0 0 vs main
Warnings	20 -121 vs main
Suggestions	28 -120 vs main
Issues	48 -241 vs main
P0 issues	🔴 P0 0 -1 vs main
P1 issues	🟠 P1 13 -3 vs main
Live issues	0 total -1 vs main / 0 P0 -1 vs main
Compat gaps	11 +11 vs main
Deprecation warnings	1 -47 vs main
Inspector gaps	21 -136 vs main
Upstream metadata	15 -68 vs main
Contract probes	40 -235 vs main
Policy failures	0 0 vs main
Policy warnings	20 -288 vs main
Ref diff failures	0 0 vs main
Profile failures	0 0 vs main
Execution probes	78 pass -42 vs main / 0 fail 0 vs main / 20 blocked -286 vs main
Synthetic probes	66 ready -436 vs main / 2 blocked -7 vs main / 68 total -443 vs main
Cold import	0 ready -7 vs main / 10 blocked -91 vs main / 10 entrypoints -98 vs main
Workspace plan	10 entrypoints -98 vs main / 8 installs -60 vs main / 0 builds -15 vs main
Platform risks	0 Windows -17 vs main / 0 container -17 vs main
Jiti loader candidates	10 -10 vs main
Import loop	p50 67ms -2519 vs main / p95 68ms -2520 vs main / plugin delta RSS 0.1MB -13.9 vs main / plugin delta CPU 16ms -187 vs main
Runtime profile	p50 826ms -5512 vs main / command p95 849ms -5585 vs main / max RSS 98.5MB -346.9 vs main / 3 samples/command

Top Discovered Issues

Severity	Class	Fixture	Code	Decision	Title
🟠 P1	compat-gap	codex	missing-compat-record	core-compat-adapter	codex: compat-dependent behavior lacks registry coverage
🟠 P1	upstream-metadata	codex	reserved-sdk-import	plugin-upstream-fix	codex: plugin imports reserved bundled-plugin SDK compatibility subpaths
🟠 P1	compat-gap	diagnostics-prometheus	missing-compat-record	core-compat-adapter	diagnostics-prometheus: compat-dependent behavior lacks registry coverage
🟠 P1	compat-gap	diffs	missing-compat-record	core-compat-adapter	diffs: compat-dependent behavior lacks registry coverage
🟠 P1	compat-gap	google-meet	missing-compat-record	core-compat-adapter	google-meet: compat-dependent behavior lacks registry coverage
🟠 P1	compat-gap	google-meet	sdk-export-missing	core-compat-adapter	google-meet: plugin SDK import aliases are missing from target package exports
🟠 P1	inspector-gap	memory-lancedb	conversation-access-hook	inspector-follow-up	memory-lancedb: conversation-access hooks need privacy-boundary probes
🟠 P1	compat-gap	memory-lancedb	missing-compat-record	core-compat-adapter	memory-lancedb: compat-dependent behavior lacks registry coverage
🟠 P1	compat-gap	memory-lancedb	missing-compat-record	core-compat-adapter	memory-lancedb: compat-dependent behavior lacks registry coverage
🟠 P1	compat-gap	openclaw-qqbot	missing-compat-record	core-compat-adapter	openclaw-qqbot: compat-dependent behavior lacks registry coverage

What this tests

• plugin manifests and install metadata
• native tool registration and dynamic tool schemas
• channel registration and message delivery seams
• lifecycle hooks such as gateway_start, gateway_stop, and before_install
• agent hooks such as before_tool_call, before_prompt_build, llm_input, llm_output, and agent_end
• provider capability registration such as speech/TTS
• plugin-owned services, routes, subprocesses, and async job patterns

Layout

crabpot/
  crabpot.config.json        fixture manifest and seam tags
  plugins/                   external plugin repositories as git submodules
  reports/                   generated compatibility report artifacts
  scripts/                   manifest and fixture helpers
  test/                      repo-level checks
  docs/                      operating notes and seam matrix

Quick start

npm test
node scripts/list-fixtures.mjs
node scripts/sync-fixtures.mjs --check
npm run report
npm run contract:capture
npm run contract:synthetic
npm run cold-import
npm run workspace:plan
npm run platform:probes
npm run import:profile
npm run execution:report
npm run profile
npm run contract:coverage
npm run readme:summary

To materialize the fixture repos as submodules:

node scripts/sync-fixtures.mjs --materialize
git submodule update --init --recursive

That command mutates .gitmodules and plugins/*. Commit those changes when you intentionally pin or update fixture revisions.

Compatibility report

Start with the dashboard at the top of this README. It is the condensed view of the generated reports: fixture count, breakages, warnings, issue backlog, probe coverage, cold-import blockers, workspace execution shape, and runtime profile.

For deeper review, open the reports in this order:

Need	Command	Primary report
Main compatibility triage, decision matrix, issue backlog	npm run report	reports/crabpot-report.md
Stable issue list for compat-layer planning	npm run report	reports/crabpot-issues.md
Hooks, registrars, SDK imports, and entrypoints that need capture	npm run contract:capture	reports/crabpot-capture.md
Executable synthetic hook/registration probe plan	npm run contract:synthetic	reports/crabpot-synthetic-probes.md
Why plugin entrypoints cannot be safely cold-imported yet	npm run cold-import	reports/crabpot-cold-import.md
Isolated install/build/capture commands Crabpot would run	npm run workspace:plan	reports/crabpot-workspace-plan.md
Results from opt-in isolated fixture execution	npm run execution:report	reports/crabpot-execution-results.md
Behavioral eval planning for plugin categories	npm run eval:behavior	stdout / .crabpot/results/behavior/ when executed
Boot time and RSS against the target OpenClaw registry surface	npm run profile	reports/crabpot-runtime-profile.md
China and adjacent external plugin monitor candidates	manual live discovery pass	reports/crabpot-external-plugin-monitor.md
README dashboard refresh from all generated JSON reports	npm run readme:summary	README.md, reports/crabpot-dashboard-data.json

Each Markdown report has a matching JSON file beside it for CI, dashboards, and future inspector tooling. The JSON is the contract; the Markdown is the review surface. reports/crabpot-dashboard-data.json is the compact machine-readable dashboard card used to compare crab-beta and crab-development against main.

Behavioral eval POC

Behavior evals are profile-driven, default to a dry plan, and stay credential-free unless execution is explicitly enabled. The default profile is the forward LCM tracking gate against latest OpenClaw and latest @martian-engineering/lossless-claw. It verifies recall inside one stable session-key family after /lossless rotate, with the seed pushed behind the fresh tail so raw transcript replay is not enough to pass:

npm run eval:behavior

Run the recent historical repro target from Discord chatter with:

npm run eval:behavior -- --profile recent-lcm-2026-5-22 --json

The companion quarantine gate installs a synthetic malformed context-engine plugin, selects it in the active slot, requires the agent turn to continue through downgrade/default behavior, and tracks the gateway health quarantine signal as an expected failure until that OpenClaw health contract ships in latest:

npm run eval:behavior -- --profile forward-context-engine-quarantine-gate

Execution is opt-in and isolated. The current POC stages a QA-lab-style mock model provider, starts an isolated gateway, drives chat.send -> agent.wait -> chat.history, requires the /lossless rotate command response, and checks the latest assistant response on the recall turn using the same stable session key:

CRABPOT_EXECUTE_BEHAVIOR=1 npm run eval:behavior -- --execute --profile recent-lcm-2026-5-22 --runner local --timeout-ms 120000

The historical LCM target is intentionally red: [email protected] with @martian-engineering/[email protected] may fail before final recall as a behavior-turn-mismatch, or at final recall with memory-recall-mismatch. The forward LCM target is stricter and only expects the final summary-backed recall failure until latest OpenClaw plus LCM can recall after rotate from summarized context rather than raw transcript history. To test a candidate fix, pin a newer OpenClaw package while keeping the same LCM plugin and override the expectation:

CRABPOT_EXECUTE_BEHAVIOR=1 npm run eval:behavior -- --execute --profile forward-lcm-release-gate --openclaw-version 2026.5.26 --plugin npm:@martian-engineering/[email protected] --expect must-pass --runner local --timeout-ms 240000

Behavior eval execution writes an empty npm user config inside the temp workspace so local npm policy, such as a before cutoff, does not silently move latest back to an older release.

Use the main compatibility report like this:

Section	What to do with it
Hard Breakages	Treat as release-blocking contract drift.
Warnings	Review for target OpenClaw compatibility gaps or plugin metadata drift.
Suggestions To OpenClaw Compat Layer	Convert into compat-layer work, inspector follow-ups, or upstream plugin requests.
Issue Findings	Use stable CRABPOT-* ids for tracking and comparison across runs.
Contract Probe Backlog	Turn into tests before changing a plugin-facing seam.
Decision Matrix	Decide whether the fix belongs in core compat, the future inspector, or the plugin upstream.

By default, reports target the OpenClaw checkout configured in crabpot.config.json. Point a run at a branch, tag, SHA checkout, or local fork with --openclaw:

node scripts/generate-report.mjs --openclaw ../openclaw
node scripts/generate-report.mjs --check --openclaw ../openclaw

Crabpot does not execute third-party plugin code during default checks. The workspace plan is dry planning unless you explicitly opt into isolated execution. Preview a fixture lane first:

npm run workspace:execute -- --fixture wecom --dry-run

Then run isolated execution only when you want install/build/import side effects inside Crabpot's generated workspace:

CRABPOT_EXECUTE_ISOLATED=1 npm run workspace:execute -- --fixture wecom
npm run execution:report

Manual OpenClaw ref CI

The OpenClaw Ref Compatibility workflow can be run from GitHub Actions with an OpenClaw branch, tag, or SHA. Set openclaw_repository when testing a fork, and openclaw_ref to the exact ref under review.

The default job runs the static contract suite against that checkout and uploads the generated reports. The optional isolated job runs one fixture lane when run_isolated_fixture is enabled and fixture is set, then uploads .crabpot/results/ plus the execution summary report.

Fixture policy

Fixtures should earn their spot by covering a distinct seam. Popularity is a useful signal, but a small plugin that exercises a rare hook is more valuable than the fourth web-search wrapper.

The first fixture set intentionally covers channels, dynamic tools, LLM observation, diagnostics, gateway-owned services, async jobs, provider capabilities, and security/policy hooks.