Validate connectors against real accounts: opt-in live smoke tests + checklist #77

Context

Every connector merged so far was built and tested against fake APIs only (CI has no network and no credentials, by design). None has been run against a real account using the Waymark identity yet: GitHub App (#61), Entra provider (#62), Atlassian provider (#63), repo docs (#52), Linear (#55), Notion (#57), Discord (#58). Real APIs drift from docs (field names, pagination, rate-limit headers, auth quirks), so we need a repeatable way to prove each connector works end to end, read-only, with its own identity.

Part of epic #20.

Goal

1. Opt-in live smoke tests that run against real accounts when credentials are present and are skipped otherwise.
2. A manual GitHub Actions workflow to run them on demand with repository secrets.
3. A validation checklist per connector, filled once by the maintainer, recorded in this issue.

Evidence

• AGENTS.md: tests must not hit the network; so live tests live outside npm test.
• Vitest can select files by pattern and skip tests conditionally (describe.skipIf): https://vitest.dev/api/#describe-skipif
• GitHub Actions workflow_dispatch with environment secrets: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflow_dispatch and https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment#environment-secrets
• Identity setup per source: docs/identities.md, ADR-0002 (docs/adr/0002-own-identity.md).

Exact decisions

Live smoke tests

• Files test-live/*.live.test.ts, run with npm run test:live = vitest run --dir test-live (separate from npm test/npm run check; CI never runs them).
• One file per connector. Each describe.skipIf(!<required env vars>) so a missing credential means "skipped", never "failed".
• Each test creates a temp WAYMARK_HOME, writes a config with one source in auth: 'app' mode (plus a second run with auth: 'personal' where that mode exists), runs syncAll, and asserts:
  • no source error; count > 0 on the first run; a second run is incremental (count ≤ first, and the cursor changed or stayed);
  • every stored doc parses, has the expected kind, a non-empty title, a url on the source's host, and tags as the issue specified;
  • no secret leaks: the wiki and db/ contain none of the credential values (scan files for the env var values), and stderr has none of them;
  • identity check where the API allows it (GitHub App: installation token, not user; Linear: viewer is the app actor; Atlassian: /myself is the service account; Entra: token appid/oid claims match the configured client id — decode the JWT payload only for this assertion).
• Tests print counts only, never content.
• Targets (read-only, chosen by the maintainer) come from env, e.g. LIVE_GITHUB_REPO, LIVE_LINEAR_TEAM, LIVE_DISCORD_CHANNEL, LIVE_ADO_ORG/LIVE_ADO_PROJECT, LIVE_ATLASSIAN_SITE. Document them in test-live/README.md.
• Coverage now: git repo docs (no credentials; uses this repo), GitHub App + personal, Linear app + personal, Notion, Discord, Entra token for ADO and Graph scopes, Atlassian token + cloud id. When Azure DevOps connector: work items and comments #53/Azure DevOps connector: pull requests and review threads #54/Jira Cloud connector: issues and comments (search/jql, ADF) #56/Confluence connector: pages of configured spaces #64/Azure DevOps connector: project and code wikis #65/Slack connector: configured channels via the Waymark Slack app #66/Microsoft Teams connector: channel messages via RSC (Waymark Teams app) #67 merge, each adds its live test in the same PR.

Manual workflow

.github/workflows/live.yml: on: workflow_dispatch with an input connectors (comma list or all), environment: live (secrets live there, not at repo level), permissions: contents: read, Node 24, npm ci, npm run test:live. Uses SHA-pinned actions like ci.yml (#60).

Checklist (maintainer, once per connector; tick here with date)

For each source: identity created as in docs/identities.md with read-only permissions → waymark sync succeeds → waymark search <a known word> finds a doc with the right link → change something at the source and resync (update shows up) → delete/trash something where supported (removed after resync) → the source's audit log shows Waymark as the actor → auth: 'personal' (if supported) works and logs the warning.

• Repo docs (Git source: ingest repository markdown docs (README, ADRs, docs/) #52)
• GitHub via GitHub App (ADR-0002: Waymark uses its own identity at every source (+ GitHub App auth) #61) — 2026-10-31 (audit log: N/A on personal accounts, see comment)
• Linear via OAuth app (Linear connector: issues and comments #55)
• Notion integration (Notion connector: pages shared with the integration #57)
• Discord bot (Discord connector: messages of configured channels #58)
• Entra service principal: Azure DevOps token (Entra token provider for Waymark's service principal / managed identity (Azure DevOps, Teams) #62)
• Atlassian service account token + cloud id (Atlassian service account auth for Jira and Confluence #63)
• Azure DevOps work items (Azure DevOps connector: work items and comments #53, shipped in feat: path facts (#84), team server model (#93), Azure DevOps work items (#53), Jira Cloud (#56) #113)
• Jira Cloud issues and comments (Jira Cloud connector: issues and comments (search/jql, ADF) #56, shipped in feat: path facts (#84), team server model (#93), Azure DevOps work items (#53), Jira Cloud (#56) #113)

Deferred manual checks from #113

#113 merges without these; they were moved here so they are not lost. Counts only, never content.

• Jira (Jira Cloud connector: issues and comments (search/jql, ADF) #56): sync a real site with the service account (auth: "app"). Run a second sync to confirm the −24h updated >= window in the profile's time zone. Include an issue with more than 20 comments to exercise the overflow fetch.
• Azure DevOps work items (Azure DevOps connector: work items and comments #53): read-only sync of a real project with auth: "app" (a service principal with Basic access). Confirm errorPolicy: "Omit" returns null for a deleted item, comment paging returns continuationToken, and WIQL accepts the fractional-second cursor.
• Path facts (Path facts: index which files each commit, PR and repo doc touches, and who touched them #84): on an existing v2 install, open the index once and confirm commit facts were backfilled and the one-time log line appeared.

How to run

The maintainer runs the checklist above by hand for each connector: add the source to config.json, set the env vars from docs/identities.md, run waymark sync, then check counts and search. npm run test:live / live.yml stay the automated path. This issue stays open until the checklist is done for every connector listed.

Acceptance criteria

• npm run test:live with no env vars: every live test skipped, exit 0; npm run check unchanged (does not include test-live/).
• With credentials for at least one source (maintainer), the corresponding live test passes; result (counts only) pasted in this issue.
• Secret-leak scan helper unit-tested with a synthetic secret.
• live.yml present, manual-only, uses the live environment, pinned actions.
• test-live/README.md lists every env var per connector and links docs/identities.md.

Out of scope

Write operations of any kind, load/performance testing, running live tests on PRs.

Pending test coverage and live validation from PR #130

Tracks the remaining test-only work for #54, #64 and #65, implemented in PR #130. Automated fake-API and local integration validation passed: npm run check (494 passed, 1 skipped). This is not evidence of successful authentication or synchronization against real Azure DevOps/Confluence accounts. Keep this issue open until the following checks have recorded results.

• Add opt-in live smoke coverage for Azure DevOps PRs (Azure DevOps connector: pull requests and review threads #54), Azure DevOps wikis (Azure DevOps connector: project and code wikis #65) and Confluence (Confluence connector: pages of configured spaces #64), following the existing test-live/ harness. Document target environment variables in test-live/README.md; keep live tests outside npm run check and skip cleanly without credentials.
• Azure DevOps PRs: use a maintainer-selected project and Waymark service principal with Basic access and repository Read permissions. Confirm repository filtering, full PR descriptions, reviewer votes, file threads/replies, and incremental resync. Exercise pagination with suitable existing data and record any scenario not exercised.
• Azure DevOps wikis: confirm wiki filtering, markdown content and page pagination. Run a second unchanged sync and verify no new wiki commit. Using maintainer-prepared fixture changes, verify updated pages and removal after a page disappears. Do not modify remote content from the connector or live tests.
• Confluence: use a maintainer-selected space and Waymark service account. Confirm space-key resolution, ADF panel/table rendering, Link pagination, incremental resync and search links. Using maintainer-prepared fixture changes, verify archived/deleted pages disappear after resync.
• For each connector, verify read-only app identity/permissions and that credentials do not appear in stored files or logs. Record identity evidence where the platform exposes it; otherwise mark the limitation explicitly.
• Record execution date, tested commit SHA, commands or workflow-run link, document counts for first/second sync, pass/fail for each scenario and remaining limitations here. Never paste tokens, document content or personal data. Skipped tests do not count as live validation.

No live execution has been performed for these three connectors yet. This checklist is follow-up validation; it does not claim PR #130 has passed real-account tests.