Skip to content

Latest commit

 

History

History
275 lines (206 loc) · 17.1 KB

CONTRIBUTING.md

File metadata and controls

275 lines (206 loc) · 17.1 KB

Contributing to DocGuard

Thank you for your interest in contributing to DocGuard! This document provides guidelines for contributing.

Getting Started

  1. Fork the repository
  2. Clone your fork: git clone https://github.com/YOUR_USERNAME/docguard.git
  3. Install: npm install (one runtime dependency, @babel/parser, exact-pinned; no dev dependencies)
  4. Run tests: npm test
  5. Run DocGuard on itself: node cli/docguard.mjs guard
  6. Spec Kit (optional, needed to run the /speckit-* workflow): install the specify CLI, then npm run speckit:dev to register this repository's own extension and its hooks. The Spec Kit core skills for Claude Code (.claude/skills/speckit-*) and Codex (.agents/skills/speckit-*) are already tracked.

Development Workflow

DocGuard follows Canonical-Driven Development (CDD). Before making changes:

# 1. Check current compliance
node cli/docguard.mjs guard

# 2. Make your changes

# 3. Run tests
npm test

# 4. Verify docs still pass
node cli/docguard.mjs guard

# 5. Update CHANGELOG.md with your changes

Project Structure

cli/
  docguard.mjs         ← Entry point, config loading, command routing
  commands/             ← user-facing commands (see README for the governed list)
  validators/           ← independent validation modules (one file each)
  scanners/             ← code-truth extraction (routes, schemas, claims, …)
templates/              ← CDD document templates + slash commands
tests/                  ← Integration tests
docs-canonical/         ← DocGuard's own CDD documentation

(Counts are deliberately not stated here — they drift. The README's counts are machine-governed by the canonical-sync validator.)

Architecture Rules

  • Minimal dependencies: exactly one runtime dep (@babel/parser, exact-pinned, optional-load). New deps require constitution-level justification + exact pinning + supply-chain vetting. Default to Node.js built-ins.
  • Validators are pure: Each validator receives (projectDir, config) and returns results. No side effects.
  • Commands don't cross-import: Commands import from validators, never from other commands.
  • Writes are explicit and bounded: The CLI previews first. Deterministic writes require a write flag, preserve recovery evidence, and fail closed on ambiguous ownership. Human-authored intent is never rewritten to match code automatically.

Adding a New Command

Surface rule: the v0.20 consolidation cut 21 commands to 13 because the surface had outgrown its users; it has been creeping back since. A PR adding a new user-facing command must either retire/absorb an existing one or include an explicit justification for why the surface must grow. Internal helper modules under cli/commands/ (not registered in the dispatcher) are exempt — but add them to surfaceSync.commands.ignore in .docguard.json or guard will flag them.

  1. Create cli/commands/your-command.mjs with an exported runYourCommand(projectDir, config, flags) function
  2. Import it in cli/docguard.mjs
  3. Add it to the help text, command routing switch, and argument parsing
  4. Update the README commands table + count (Canonical-Sync/Surface-Sync enforce these)
  5. Add focused tests in tests/<command>.test.mjs or the nearest existing suite
  6. Update CHANGELOG.md

Contributing a detection case

Search open and closed issues and pull requests before filing. Include a minimal synthetic reproduction, a neighboring valid control, the expected and actual finding codes, and the smallest relevant configuration. Classify the report as a suspected false positive, false negative, unsupported syntax, or policy disagreement. Test-only contributions are welcome; maintainers can own the detector change after the behavior is reproduced.

Contributing an agent-context case

Run npm run benchmark:agent-context to validate the frozen synthetic fixtures without invoking a model. New task classes belong in a new predeclared protocol, not in the completed v1 matrix. Include an agent-visible fixture, evaluator-only fail-to-pass and pass-to-pass checks, reviewed reference behavior, an exact changed-file allowlist, model/harness identity, repetitions, and promotion thresholds before collecting observations. Keep private repositories, prompts, source patches, credentials, and raw model prose out of committed results.

Adding a New Validator

Findings rule: new validators MUST emit structured findings (cli/findings.mjs: stable code, severity, confidence, location, → suggestion) and return via resultFromFindings() — not hand-built error/warning strings. The legacy string shape is being migrated out (a few validators per release); don't add to the backlog. Codes are append-only public surface: register them in the CODES registry.

  1. Create cli/validators/your-validator.mjs
  2. Import it in cli/commands/guard.mjs
  3. Add enable/disable support in .docguard.json validators config
  4. Add tests
  5. Update docs-canonical/ARCHITECTURE.md with the new validator

Commit Messages

Follow Conventional Commits:

feat: add new validator for API docs
fix: handle missing package.json gracefully
docs: update ARCHITECTURE.md with new component
refactor: extract scoring logic into shared function
test: add edge case tests for score command

Pull Request Process

  1. Ensure npm test passes with no failures
  2. Ensure node cli/docguard.mjs guard passes
  3. Update CHANGELOG.md under [Unreleased]
  4. Update relevant docs in docs-canonical/ if architecture changed
  5. Request review

Closing a Spec

A change that implements a spec closes it in the same pull request, in one commit, with no hand edit of .docguard-specs.json (specs/101-completion-one-commit):

# tick the last task in specs/<feature>/tasks.md, then:
docguard specs --write
docguard specs complete --id <spec-id> --since <ref> \
  --canonical-doc docs-canonical/ARCHITECTURE.md --persistence living \
  --reason "<what was reviewed>" --write
git add -A && git commit

specs approve takes the same --canonical-doc (repeatable) and --persistence flags. If your change edits the evidence of specs that are already closed, guard reports SPR011 for each; docguard specs complete --stale --reason "<text>" --write closes them all again in one transaction (specs/103-completion-upkeep). Ticking boxes in the completing spec's own tasks.md does not block complete; any other change to it does. The record carries the git blob of every evidence file, so a squash merge or a rebase does not strand it: there is no second "reanchor" commit. Run docguard specs reanchor --id <spec-id> --write only when guard reports SPR008 (a record written before blobs were recorded, or evidence that changed between review and landing).

Code Style

  • ES Modules (import/export) throughout
  • Node.js built-ins only (node:fs, node:path, node:child_process, node:test)
  • ANSI colors via the shared c object from docguard.mjs
  • No TypeScript — plain JavaScript for maximum portability

Tests That Create Temporary Directories

A temporary directory comes from makeTempDir(prefix) in tests/helpers/temp-dir.mjs, never from mkdtempSync(join(tmpdir(), …)) (spec 098). The helper removes the directory when the test it was given (makeTempDir(prefix, t)) ends, when the file's tests end, and, as a backstop, when the process exits or is interrupted. It works without the test runner, so a benchmark or tool uses it too and also removes its directories in a finally. tests/test-temp-dirs.test.mjs fails on a mkdtemp call outside the helper, and npm run test:temp-leaks runs the suite with an empty temporary folder and fails if anything is left in it.

Tests That Create Git Repositories

Fixture git goes through tests/helpers/git.mjs; product git does not.

  • A test that builds or inspects a throwaway repository passes FIXTURE_GIT on every git call (spawnSync('git', [...FIXTURE_GIT, ...args])), or FIXTURE_GIT_SHELL in a git … command line. The options turn hooks off (core.hooksPath set to the null device) and commit and tag signing off, for that one call.
  • Prefer fixtureGit(cwd, args) for setup steps. It throws with git's stderr when the step fails, so a fixture that was never committed cannot let the test pass against an empty repository.
  • Pin the branch when the test depends on it: git init -b main.
  • Never set the options in the environment, and never add them to cli/. DocGuard's own git calls must keep meeting the user's real configuration. The exception is a scratch repository DocGuard itself creates, such as the demo fixture.
  • tests/test-git-hermeticity.test.mjs fails on a git spawn in tests/ that bypasses the helper. A call that must bypass it (a control that proves a config bites) carries a fixture-git-exempt: comment with the reason.

CI's hermeticity job runs the suite under a global config whose hooks all fail, whose signing always fails and whose default branch is trunk. To reproduce it locally without touching your own config:

H=$(mktemp -d); mkdir "$H/hooks"
for h in pre-commit prepare-commit-msg commit-msg post-commit pre-push post-checkout; do
  printf '#!/bin/sh\nexit 1\n' > "$H/hooks/$h"; chmod +x "$H/hooks/$h"
done
printf '[core]\n\thooksPath = %s\n[commit]\n\tgpgsign = true\n[tag]\n\tgpgsign = true\n[gpg]\n\tprogram = false\n[init]\n\tdefaultBranch = trunk\n' "$H/hooks" > "$H/gitconfig"
GIT_CONFIG_GLOBAL="$H/gitconfig" npm test

Reporting Bugs

Open a GitHub issue with:

  • DocGuard version (docguard --version)
  • Node.js version (node --version)
  • OS and version
  • Steps to reproduce
  • Expected vs actual behavior

Research & Academic Credits

DocGuard's architecture is informed by peer-reviewed research in AI-driven documentation generation and multi-agent quality evaluation. We gratefully acknowledge the following contributions:

Key Research Contributors

  • Martin Manuel Lopez · ORCID 0009-0002-7652-2385
    • Ph.D. Candidate, Dept. of Electrical and Computer Engineering, University of Arizona
    • Lead author on AITPG and TRACE — the two papers that informed DocGuard's quality evaluation, multi-perspective analysis, and standards-grounded generation patterns

Cited Papers

The following papers directly influenced DocGuard's design:

[1] M. M. Lopez, M. W. U. Rahman, C. Farthing, J. Battle, K. Buckley, G. Altintarla, and S. Hariri, "AITPG: Agentic AI-Driven Test Plan Generator using Multi-Agent Debate and Retrieval-Augmented Generation," IEEE Transactions on Software Engineering, 2026. — Introduced the three-stage pipeline (generate → debate → calibrated evaluation), RAG-grounded standards alignment, and multi-agent role specialization (Positive/Negative/Edge + Critic) for documentation generation.

[2] M. M. Lopez, M. W. U. Rahman, C. Farthing, J. Battle, K. Buckley, G. Altintarla, and S. Hariri, "TRACE: Telecommunications Root Cause Analysis through Calibrated Explainability via Multi-Agent Debate," IEEE Transactions on Machine Learning in Communications and Networking, 2026. — Introduced Calibrated Judge Evaluation (CJE) with weighted multi-signal composite scoring, HIGH/MEDIUM/LOW quality labels, the "equalizer effect" for agent-aware prompt scaling, and adversarial debate (Advocate/Challenger/Mediator/Explainer) for robust quality assessment.

Concepts Adopted in DocGuard

DocGuard Feature Research Origin Paper
Quality labels (HIGH/MED/LOW) in guard output — label vocabulary only; the labels are deterministic strata, not calibrated probabilities (see PHILOSOPHY.md) CJE quality stratification TRACE [2]
Standards citations in generated docs RAG-grounded standards alignment AITPG [1]
Multi-signal composite scoring in score 5-signal weighted composite (Eq. 1) TRACE [2]
Traceability matrix (trace command) Requirements traceability AITPG [1]
Multi-perspective diagnose --debate prompts Multi-agent role specialization AITPG [1], TRACE [2]
Agent-aware prompt complexity CJE equalizer effect TRACE [2]

License

By contributing, you agree that your contributions will be licensed under the MIT License.

Explain every finding change before a release

A release is compared with the previous one, finding by finding (tools/release-ab.mjs, spec 087). The release fails unless the CHANGELOG section being released names each finding code whose findings changed. When your change adds, removes or rewords findings, name the code in your [Unreleased] entry and say why it changed. If it changes an exit code, add a line starting Exit codes: that says which runs now exit differently. For a release that grows a budget on purpose, add Budget-Exempt: <metric> — <reason> as a plain line in that section. For a benchmark fixture you corrected on purpose, add Benchmark-Rebaselined: <case id> — <reason> the same way. To preview the gate locally against the last release:

git archive "$(git describe --tags --abbrev=0)" | tar -x -C ../prev
ln -s "$PWD/node_modules" ../prev/
node tools/release-ab.mjs --base ../prev --head .

Fuzz the document parsers

npm test fuzzes every text parser with a fixed seed (tests/validator-fuzz.test.mjs). Each parser must return without throwing, give the same answer twice, and finish within a budget linear in its input. When you add or change a parser that reads text people or agents write, add it to TARGETS in tools/fuzz.mjs, and add a structured block to the grammar if the generator doesn't reach it yet. The suite asserts every parser is reached.

For a deep run, use npm run fuzz -- --cases 20000 --seed <n>. Add --save to shrink each failing input and keep it in tests/fixtures/fuzz/, where the suite replays it. Fix the parser; never exempt the input (specs/088-validator-fuzz).

Turn a finding into a regression test

A report or a test-only contribution is useful even when you cannot fix the detector. Start with docguard feedback --code <CODE> --preview for a disputed finding, or copy templates/feedback-fixture.json for a false negative or unsupported syntax case. Validate without writes using docguard feedback --fixture-manifest <path> --reduce --preview --format json.

  1. Open the generated search link and check existing issues and pull requests, including closed work. Add evidence to a matching discussion instead of opening a duplicate. A previously fixed issue needs a current reproduction before it is treated as a regression.
  2. State expected behavior and actual behavior. Distinguish a false positive, a missed defect, unsupported syntax, and a policy preference. A suppression by itself does not demonstrate a bug.
  3. Create the smallest synthetic project that preserves the behavior. Use invented names and values. Include the tool version, runtime version, minimal configuration, exact command, and expected finding code. Avoid attaching the full local feedback record: it can contain private project details.
  4. Set provenance.synthetic and redactionAttested only after reviewing every inline file. Record the parser tier, exact expected identity, explicit predicate, same-path opposite control, supported scope, and candidate benchmark delta. The command refuses incomplete contribution evidence.
  5. Generate a direct test with --contribution tests/<name>.test.mjs. A false-positive case should stay clean while its neighboring real defect still emits; a missed detection should emit while its clean control remains clean. Ambiguous and policy cases require adjudication before test generation.
  6. Run the focused test and node benchmarks/run.mjs --external --baseline benchmarks/baseline.json against the affected version. Describe the result honestly. A test that never exercised the intended parser path is not evidence of a fix.
  7. If the baseline must change, regenerate it with --write-baseline benchmarks/baseline.json --replace-baseline and set review.status, reviewedAt, and reviewer by hand after adjudicating every case. Never edit review.caveat or core.metrics: both are derived from core.cases and the loader rejects an envelope where they disagree. When quoting a benchmark number anywhere, quote it with its n, its Wilson bound, and the envelope's review.caveat — a precision figure without them reads as a calibrated probability, which DocGuard does not measure.
  8. Submit the reviewed example or a test-only PR. Include the duplicate identity so maintainers can find related open and closed work. Follow the repository's changelog and canonical-documentation rules when changing behavior.

Maintainers should acknowledge reproducible reports, preserve contributor attribution with consent, and record the causal defect family in the regression test. Reproduction success and triage time matter more than issue volume. Do not execute arbitrary submitted scripts with repository credentials. Security-sensitive reports belong in the private channel described by SECURITY.md.

The public issue draft contains metadata only. Opening a link is optional, and DocGuard never submits a report automatically.