Epic: Teammates — AI teammates that investigate on your own runtime #116

Goal

Let a team mention an AI teammate in chat (or start it from the CLI or API) and have it investigate a real problem across their systems, then report in the thread or file a card. Waymark composes the pipeline: teammate definition, playbooks, context from the knowledge base, an allow-listed and audited tool gateway, run state, routing, reports and cards. The team's own runtime (Claude Code, Hermes Agent or any CLI agent) runs the agent loop on the team's own machines, with the team's own model keys. Decision record: docs/adr/0004-teammates.md (PR #129).

Phase 1 is observe-only: teammates read, reason and write up. Writing code, merging and deploying are later phases of this epic.

The scenario that drives phase 1

1. A manager sees blank currency values for one day in a spreadsheet add-in and mentions the bot in chat.
2. The teammate looks up the system map: the add-in reads an internal API, the API reads a service database, and that database is a replica fed by CDC from another system's database through a separate CDC project.
3. It calls the API (the value is null), queries the service database (the rows are missing) and the source database (the rows exist), so the gap is in the replication.
4. It searches the CDC project's logs (an overflow error at the time of the daily import) and its code and commits, and names the commit that narrowed the column type.
5. It replies in the thread with the evidence, or files a card with the full write-up when a human has to act.

Every hop is proven with an audited tool call, and the final verdict cites those calls.

Issues

Order: #117 ∥ #124 → #118 → #119 → #120 ∥ #121 → #122 ∥ #125 ∥ #126 → #123 → #127. #124 adds the index schema version after #84's v3 (v4 unless another schema bump lands first; whichever merges second takes the next number).

Conventions for every issue

• ADR-0004 is the source of truth. An issue may add detail it leaves open (error texts, CLI output, file paths); a change to a contract it fixes (playbook format version: 1, verdict schema, runner protocol, wmr_ tokens, waymark.dev/* annotations) needs an ADR update first.
• Waymark never runs model-generated code or shell commands and never holds the runtime's model keys. Runtime secrets stay in the runner's environment.
• Source-level access control: groups decide which sources each user can see #97 is optional for every issue: visibility rules apply in full once Source-level access control: groups decide which sources each user can see #97 is merged; until then all authenticated users see all sources.
• autonomy other than observe is rejected with an actionable message until its phase ships.
• Tools are authorized by allow lists and stage tools, never by MCP tool annotations. Read-only is guaranteed by the credentials the operator grants; docs say so wherever tools are configured.
• Every downstream tool call and every write (report, card) is audited (Audit log: append-only record of who searched, read and wrote what #94). Tool results and anything a teammate writes pass through redact().
• Run store wiring: Run store: runs.db, run life cycle, routing, event log and waymark runs #118 exports RunStoreDeps ({ runsDb, authDb, db, config, paths, audit, onTerminal? }). waymark serve and local mode build one such object and pass the same one to the gateway (Run gateway: per-stage MCP endpoint, wmr_ run tokens, waymark__* tools and finish_stage #119), the runner protocol handlers and the sweeper (Runners: pull protocol, waymark runner and waymark run #121); card action: file a GitHub issue with the full investigation write-up #126 plugs cardOnTerminal into its onTerminal.
• Visibility of the new documents: agent/run/* is decided by canSeeRunDoc (src/runs/store.ts, Run store: runs.db, run life cycle, routing, event log and waymark runs #118); catalog/* is visible like its document's sourceId (the git source, System map: ingest Backstage catalog-info.yaml, system_map tool and waymark map #124), never decided by the id prefix. On filing, Source-level access control: groups decide which sources each user can see #97 gets a comment with these two rows.
• Run state is state, not cache: runs/ is never touched by reindex and is backed up like auth/.
• Tests: no network, no model, no real tokens; runtimes, GitHub and MCP servers are fakes injected as parameters; loopback only.
• No new runtime dependency unless named in the issue (Tool gateway: allow-listed downstream MCP servers on the run endpoint, with audit and redaction #120 moves @modelcontextprotocol/client from dev to runtime dependencies).
• User-facing behaviour is documented in docs/teammates.md (created by Teammates and playbooks: config sections, playbook YAML format, waymark playbook check and JSON Schema #117) and in the README command table.

Later phases (not yet issues)

• Claude Agent SDK adapter: an in-process adapter on @anthropic-ai/claude-agent-sdk as an optional dependency, alongside the claude-code CLI adapter.
• propose: a code stage that opens branches and PRs (never merges), and a verifier gate (a separate stage or runtime that checks the change) before a PR is proposed.
• Scheduling recipe: docs showing how to start playbooks on a schedule from the OS scheduler or CI by calling waymark run (no built-in scheduler in the core).
• Observability: OpenTelemetry traces for runs, stages and tool calls; cost and turns per run and per stage.
• Evals: recorded scenarios replayed against playbook and runtime changes to measure investigation quality over time (builds on End-to-end harness: scripted test runtime and the "data gap" scenario in CI #127's harness).
• Run scorers and benchmark replay: score finished runs (evidence coverage, correct root cause, cost, turns) and replay a benchmark set of past runs against a new playbook or runtime version.
• context_for_paths: for the files a change touches, return past reviews, conventions, experts and superseded decisions in one call (for the propose code stage; builds on Path facts: index which files each commit, PR and repo doc touches, and who touched them #84, Expertise map: waymark experts <path> and MCP tool who_knows #86, Give agents the team's conventions: MCP tool conventions and start-up context #88, Supersession signals: superseded ADRs, reverted commits and notes that replace notes #91).
• Intake dedup: before starting a run, match the request against past runs and open cards, and link or reuse instead of investigating the same gap twice.
• Gateway exposure of who_knows / conventions if Expertise map: waymark experts <path> and MCP tool who_knows #86 / Give agents the team's conventions: MCP tool conventions and start-up context #88 land after Run gateway: per-stage MCP endpoint, wmr_ run tokens, waymark__* tools and finish_stage #119.
• Integration docs: Sourcegraph MCP as a downstream tool server behind the gateway; Waymark's MCP server used inside other agent factories as their knowledge layer.

Definition of done

• End-to-end harness: scripted test runtime and the "data gap" scenario in CI #127's report-path case runs through the same waymark run --local entry point (adapter registry injected by the test, not a CLI flag) and prints the report and the Run: line.
• End-to-end harness: scripted test runtime and the "data gap" scenario in CI #127's scenario passes in CI: every downstream call is audited as tool.call, the verdict cites call ids, a bad ref is rejected, routing follows the playbook, agent/run/<id> is saved and the card body is rendered.
• A mention through waymark bot test with an allowed user starts a run, posts progress and the final report (Chat trigger: start teammate runs from a bot mention and post progress in the thread #125).
• Manual check (not CI): one real investigation with the claude-code adapter and one with hermes on a test setup chosen by the maintainer, each ending in a report or a card; links in the closing PR, no data.
• A playbook with autonomy: propose is rejected with the documented message.
• Docs: docs/teammates.md covers teammates, playbooks, runners, tools and read-only credentials, the system map, chat triggers and cards.

Evidence

• Decision and contracts: docs/adr/0004-teammates.md (PR docs: ADR-0004 teammates and README rewrite #129).
• Pull-only runners need outbound HTTPS only; GitHub's self-hosted runners use an HTTP(S) long poll and need no inbound connection: https://docs.github.com/en/actions/reference/runners/self-hosted-runners and https://docs.github.com/en/[email protected]/actions/concepts/runners/communicating-with-self-hosted-runners
• Headless Claude Code (claude -p, --output-format stream-json, --mcp-config, --strict-mcp-config, --max-turns, --max-budget-usd): https://code.claude.com/docs/en/cli-reference and https://code.claude.com/docs/en/headless
• Hermes Agent API server (POST /v1/runs, bearer API_SERVER_KEY) and MCP config: https://hermes-agent.nousresearch.com/docs/user-guide/features/api-server and https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp
• Claude Agent SDK (later phase): https://code.claude.com/docs/en/agent-sdk/typescript
• MCP: tool annotations are untrusted hints, aggregators should prefix tool names with a server id: https://modelcontextprotocol.io/specification/latest/server/tools
• Backstage descriptor format (catalog-info.yaml, kinds, dependsOn/providesApis/consumesApis): https://backstage.io/docs/features/software-catalog/descriptor-format
• Builds on: users, groups, linked identities and tokens (ADR-0003 team server model + users, groups, linked identities and API tokens #93, merged; ADR-0003), audit (Audit log: append-only record of who searched, read and wrote what #94, src/audit.ts on origin/main), team server (waymark serve: MCP over Streamable HTTP with API-token auth #95), API v1 (HTTP JSON API v1 on the team server (search, documents, notes, handoffs) #96), source access control (Source-level access control: groups decide which sources each user can see #97), remote mode (Remote mode: CLI search/show and agent hooks talk to a team server (WAYMARK_URL) #98), answer engine (Answer engine: waymark ask composes a cited answer from the knowledge base (LLM optional) #101), bot runtime (Bot runtime: one adapter contract, user mapping, access policy and rate limits for all chat bots #102, merged: src/bots/runtime.ts, src/bots/types.ts) and platform bots (Slack bot: answer @mentions and DMs over Socket Mode (no public URL) #103-Linear agent: answer when Waymark is @mentioned or delegated an issue (agent sessions) #107), path facts (Path facts: index which files each commit, PR and repo doc touches, and who touched them #84, merged), experts (Expertise map: waymark experts <path> and MCP tool who_knows #86) and conventions (Give agents the team's conventions: MCP tool conventions and start-up context #88).