Skip to content

Comments

Merge latest head#1

Closed
rsnodgrass wants to merge 544 commits intorsnodgrass:mainfrom
steveyegge:main
Closed

Merge latest head#1
rsnodgrass wants to merge 544 commits intorsnodgrass:mainfrom
steveyegge:main

Conversation

@rsnodgrass
Copy link
Owner

No description provided.

Steve Yegge and others added 30 commits November 2, 2025 17:57
Implement a new `bd status` command that provides a quick snapshot of the
issue database state, similar to how `git status` shows working tree state.

Features:
- Summary counts by state (open, in-progress, blocked, closed)
- Ready to work count
- Recent activity stats (last 7 days): created, closed, updated issues
- Support for --assigned flag to filter by current user
- JSON output format with --json flag
- Comprehensive test coverage

Usage examples:
  bd status                # Show summary
  bd status --json         # JSON output
  bd status --assigned     # Filter to assigned issues
  bd status --no-daemon    # Direct mode with recent activity

Note: Recent activity currently only works in direct mode (--no-daemon).
Daemon mode support marked with TODO for future enhancement.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
CRITICAL FIX: The daemon could enter a corrupt state when auto-import
was triggered while uncommitted changes existed in .beads/ files. This
caused mysterious EOF errors requiring daemon restarts.

Root Causes Fixed:
1. No git dirty check before auto-import
2. Missing timeout protection (could hang indefinitely)
3. Race condition in importInProgress flag management
4. Improper git status parsing (false positives)
5. Context leak in export goroutine (accessing closed DB)
6. Data loss in triggerExport (missing deps/labels/comments)

Changes:
- Add hasUncommittedBeadsFiles() to check git status before import
  - Properly parses git porcelain format ("XY filename")
  - Ignores untracked files, only checks tracked .jsonl changes
  - 5-second timeout on git command to prevent hangs

- Add 30-second timeout to import operations
  - Prevents daemon from hanging on stuck imports
  - Returns clear error message on timeout

- Fix race condition in importInProgress flag
  - Explicitly release before early returns
  - Prevents other requests seeing incorrect "import in progress"

- Fix context leak in onChanged export goroutine
  - Check daemon shutdown state before export
  - Suppress expected "database closed" errors during shutdown
  - Pass storage/path as parameters to avoid closure issues

- Fix data loss in triggerExport()
  - Populate dependencies, labels, comments (mirrors handleExport)
  - Use atomic file write (temp + rename)
  - Sort issues for consistent output

Impact: Prevents daemon corruption in collaborative workflows where
users frequently pull updates while having local uncommitted changes.

Testing: All auto-import tests passing
- TestDaemonAutoImportAfterGitPull ✓
- TestDaemonAutoImportDataCorruption ✓
- internal/autoimport tests ✓

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
CRITICAL BUG: The previous fix had a race condition where the
importInProgress flag could be released twice, allowing two goroutines
to think they both hold the lock.

Bug scenario:
1. Goroutine A: acquires lock (CAS true)
2. Goroutine A: manually releases at line 208 for git dirty skip
3. Goroutine B: CAS succeeds, acquires lock
4. Goroutine A: defer runs, releases flag AGAIN (clears B lock)
5. Goroutine C: CAS succeeds - now TWO goroutines have lock

Root cause: Using both manual Store(false) AND defer Store(false)
created a window where the flag could be cleared twice.

Fix: Use a shouldDeferRelease flag to disable the deferred release
when we manually release early. This ensures exactly one release
per acquisition.

Testing: All auto-import tests still passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Added 4 new integration tests to ensure sync-branch workflow is robust:

1. TestSyncBranchConfigChange - Validates changing sync.branch config
   after worktrees exist. Tests smooth transition between branches.

2. TestSyncBranchMultipleConcurrentClones - Tests 3-way clone sync
   workflow. Simulates real multi-agent collaboration scenario.

3. TestSyncBranchPerformance - Validates commit overhead < 150ms.
   Current performance: avg 77ms (well within target).

4. TestSyncBranchNetworkFailure - Tests graceful handling of network
   errors during push. Ensures local commits succeed even when remote
   is unreachable.

All tests pass:
- cmd/bd (sync branch): PASS (5.361s)
- internal/git: PASS (1.071s)
- internal/syncbranch: PASS (0.312s)

Test coverage now includes: config change handling, multiple concurrent
clones, network failure recovery, performance validation, fresh setup,
issue operations, error handling, sparse checkout.

Closes bd-0e74

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Add BEADS_DIR as a replacement for BEADS_DB to point to the .beads
directory instead of the database file directly.

Rationale:
- With --no-db mode, there's no .db file to point to
- The .beads directory is the logical unit (contains config.yaml, db
  files, jsonl files)
- More intuitive: point to the beads directory not the database file

Implementation:
- Add BEADS_DIR environment variable support to FindDatabasePath()
- Priority order: BEADS_DIR > BEADS_DB > auto-discovery
- Maintain backward compatibility with BEADS_DB (now deprecated)
- Update --no-db mode to respect BEADS_DIR
- Update MCP integration (config.py, bd_client.py)
- Update documentation to show BEADS_DIR as preferred method

Testing:
- Backward compatibility: BEADS_DB still works
- BEADS_DIR works with regular database mode
- BEADS_DIR works with --no-db mode
- Priority: BEADS_DIR takes precedence over BEADS_DB

Follow-up issues for refactoring:
- bd-efe8: Refactor path canonicalization into helper function
- bd-c362: Extract database search logic into helper function

Closes bd-e16b

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Created complete documentation suite for using beads with protected branches:

1. **docs/PROTECTED_BRANCHES.md** - Comprehensive 600+ line guide covering:
   - Quick start and setup
   - How git worktrees work
   - Daily workflow for agents and humans
   - Merging strategies (PR and direct)
   - Troubleshooting common issues
   - Platform-specific notes (GitHub, GitLab, Bitbucket)
   - Advanced topics (CI/CD, multi-clone sync, etc.)

2. **AGENTS.md** - Added "Protected Branch Workflow" section:
   - Quick reference for agents
   - No changes needed to agent workflows
   - Commands for setup and merging
   - Link to detailed docs

3. **README.md** - Updated with:
   - Protected branch support feature flag
   - Quick start instructions with --branch flag
   - Link to comprehensive guide

4. **examples/protected-branch/** - Working example with:
   - Step-by-step demo
   - Multi-clone sync workflow
   - GitHub Actions integration example
   - Directory structure explanation
   - Troubleshooting tips

All commands verified:
- bd init --branch <name>
- bd config get/set sync.branch
- bd sync --status
- bd sync --merge

Documentation is platform-agnostic and works with GitHub, GitLab,
Bitbucket, or any git platform with branch protection.

Closes bd-5ce8

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Fixed TestHashIDs_IdenticalContentDedup test failure by removing duplicate
--json flag definitions that were shadowing the global persistent flag.

Root cause: Commands had both a persistent --json flag (main.go) and local
--json flags (in individual command files). The local flags shadowed the
persistent flag, preventing jsonOutput variable from being set correctly.

Changes:
- Removed 31 duplicate --json flag definitions from 15 command files
- All commands now use the single persistent --json flag from main.go
- Commands now correctly output JSON when --json flag is specified

Test results:
- TestHashIDs_IdenticalContentDedup: Now passes (was failing)
- TestHashIDs_MultiCloneConverge: Passes without JSON parsing warnings
- All other tests: Pass with no regressions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Extracts duplicated path canonicalization logic (filepath.Abs + EvalSymlinks)
into a reusable helper function utils.CanonicalizePath() in internal/utils/path.go.

Changes:
- Add internal/utils/path.go with CanonicalizePath() function
- Add comprehensive tests in internal/utils/path_test.go
- Replace inline canonicalization in beads.go:131-140
- Replace inline canonicalization in cmd/bd/main.go:446-454
- Replace inline canonicalization in cmd/bd/nodb.go:25-33

The new helper maintains identical behavior:
1. Converts path to absolute form via filepath.Abs
2. Resolves symlinks via filepath.EvalSymlinks
3. Falls back gracefully on errors (returns absPath if EvalSymlinks fails,
   returns original path if Abs fails)

Fixes bd-efe8

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add #nosec directives with explanations for all gosec warnings in worktree operations
- Tighten directory permissions from 0755 to 0750 for better security
- Fix misspellings: archaeological -> archeological, cancelled -> canceled
- Remove unused jsonlPath parameter from syncBranchCommitAndPush
- Change branchExists to return bool instead of (bool, error) - error was never used

All changes maintain backward compatibility and improve code quality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add #nosec comments for remaining G204 subprocess warnings in syncBranchPull
- Update .golangci.yml to exclude G306 and G204 warnings for worktree files
- Simplified exclusion pattern from "G306.*0644" to "G306" to match actual error text

All linter checks now pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Updated tests to match the new branchExists() signature that returns
bool instead of (bool, error).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Updated all component versions:
- bd CLI: 0.21.3 → 0.21.4
- Plugin: 0.21.3 → 0.21.4
- MCP server: 0.21.3 → 0.21.4
- Documentation: 0.21.3 → 0.21.4

Generated by scripts/bump-version.sh
# Conflicts:
#	.beads/beads.jsonl
Steve Yegge and others added 25 commits November 5, 2025 14:18
- Add exclusive lock protocol compatibility (bd-u8j)
- Add library consumer migration guide (bd-824)
- Add self-hosting project guidance (bd-x47)
- Add performance benchmarks and targets (bd-wta)
- Clarify JSONL size bounds: per-repo <25k (bd-4ry)

Closes bd-u8j, bd-824, bd-x47, bd-wta, bd-4ry
Filed feature request based on real-world silent export failure in VC
project where SQL DELETE + export appeared to succeed but JSONL was
never updated, causing 240 deleted issues to 'come back'.

Simple verification after export would catch this immediately:
- Count lines in written JSONL file
- Compare to number of exported issues
- Error if mismatch detected

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
* feat: enhance bd doctor sync detection with count and prefix mismatch checks

Improves bd doctor to detect actual database-JSONL sync issues instead of relying only on file modification times:

Key improvements:
1. Count detection: Reports when database issue count differs from JSONL (e.g., "Count mismatch: database has 0 issues, JSONL has 61")
2. Prefix detection: Identifies prefix mismatches when majority of JSONL issues use different prefix than database config
3. Error handling: Returns errors from helper functions instead of silent failures, distinguishing "can't open DB" from "counts differ"
4. Query optimization: Single database connection for all checks (reduced from 3 opens to 1)
5. Better error reporting: Shows actual error details when database or JSONL can't be read

This addresses the core issue where bd doctor would incorrectly report "Database and JSONL are in sync" when the database was empty but JSONL contained issues (as happened in privacy2 project).

Tests:
- Added TestCountJSONLIssuesWithMalformedLines to verify malformed JSON handling
- Existing doctor tests still pass
- countJSONLIssues now returns error to indicate parsing issues

🤖 Generated with Claude Code

Co-Authored-By: Claude <[email protected]>

* fix: correct git hooks installation instructions in bd doctor

The original message referenced './examples/git-hooks/install.sh' which doesn't exist in user projects. This fix changes the message to point to the actual location in the beads GitHub repository:

Before: "Run './examples/git-hooks/install.sh' to install recommended git hooks"
After: "See https://github.com/steveyegge/beads/tree/main/examples/git-hooks for installation instructions"

This works for any project using bd, not just the beads repository itself.

🤖 Generated with Claude Code

Co-Authored-By: Claude <[email protected]>

* feat: add recovery suggestions when database fails but JSONL has issues

When bd doctor detects that the database cannot be opened/queried but the JSONL file contains issues, it now suggests the recovery command:

  Fix: Run 'bd import -i issues.jsonl --rename-on-import' to recover issues from JSONL

This addresses the case where:
- Database is corrupted or inaccessible
- JSONL has all the issues backed up
- User needs a clear path to recover

The check now:
1. Reads JSONL first (doesn't depend on database)
2. If database fails but JSONL has issues, suggests recovery command
3. If database can be queried, continues with sync checks as before

Tested on privacy2 project which has 61 issues in JSONL but inaccessible database.

🤖 Generated with Claude Code

Co-Authored-By: Claude <[email protected]>

* fix: support hash-based issue IDs in import rename

The import --rename-on-import flag was rejecting valid issue IDs with
hash-based suffixes (e.g., privacy-09ea) because the validation only
accepted numeric suffixes. Beads now generates and accepts base36-encoded
hash IDs, so update the validation to match.

Changes:
- Update isNumeric() to accept base36 characters (0-9, a-z)
- Update tests to reflect hash-based ID support
- Add gosec nolint comment for safe file path construction

Fixes the error: "cannot rename issue privacy-09ea: non-numeric suffix '09ea'"

---------

Co-authored-by: Claude <[email protected]>
Enable CGO_ENABLED=1 in goreleaser config to ensure SQLite driver
is properly compiled and linked in all builds.

Fixes #230

Amp-Thread-ID: https://ampcode.com/threads/T-06ef71cb-d3e8-42bf-8876-b09814c2c09c
Co-authored-by: Amp <[email protected]>
After atomic rename during export, verify that the JSONL file contains
exactly the same number of lines as issues exported. This catches silent
export failures where the operation appears to succeed but doesn't
actually write all issues.

Real-world scenario that motivated this:
- SQL DELETE removed 240 issues
- 'bd export' appeared to succeed
- But JSONL was never updated
- Later session found all 240 deleted issues 'came back'

Changes:
- Add verification after os.Rename in exportCmd
- Reuse existing countIssuesInJSONL() helper
- Exit with clear error if mismatch detected
- Add test case that verifies detection works

Error message shown on mismatch:
  Error: Export verification failed
    Expected: 276 issues
    JSONL file: 516 lines
    Mismatch indicates export failed to write all issues

Tests:
✓ All existing export tests pass
✓ New test verifies line counting works correctly
✓ Test simulates corruption by truncating file

Performance: Verification is fast (just counts lines), minimal overhead
- bd edit now bypasses daemon to prevent broken pipe errors
- Editing in /usr/local/bin/emacsclient can take minutes, causing daemon timeout
- Added check in PersistentPreRun to set noDaemon=true for edit command
- Human-only interactive command doesn't benefit from daemon anyway

Fixes #227
- Create .devcontainer/devcontainer.json with Go 1.23 environment
- Add setup.sh to build bd from source and install git hooks
- Add devcontainer README with documentation
- Update main README to mention devcontainer support
- Resolves bd-ry1u and GitHub issue #229
…ng hygiene

- AGENTS.md: Updated 'landing the plane' section to encourage creative,
  methodical conflict resolution instead of rigid step-by-step commands
- README.md: Added 'Configuring Your Own AGENTS.md' section recommending
  that project maintainers add session-ending protocols to their own
  AGENTS.md files to ensure proper database hygiene
@mesa-dot-dev
Copy link

mesa-dot-dev bot commented Nov 6, 2025

Mesa Description

TL;DR

Merge of the latest head from the main branch.

What changed?

File changes are not available in the provided summaries.

Description generated by Mesa. Update settings

@rsnodgrass rsnodgrass closed this Nov 6, 2025
rsnodgrass added a commit that referenced this pull request Jan 14, 2026
Security fixes from SECURITY_AUDIT.md:

1. JSONL file locking (#1 & steveyegge#3): Add flock-based shared/exclusive locking
   for JSONL reads (auto-import) and writes (export/flush) to prevent
   corruption when sync and daemon flush race.

2. Transaction tracking (steveyegge#2): Add atomic counter to track active
   transactions, ensuring reconnect waits for in-flight transactions.

3. Daemon crash lock cleanup (steveyegge#5): Use flock semantics instead of PID
   checking for stale lock detection. Add cleanup in panic handler.

4. Sync lock scope (steveyegge#6): Extend lock to cover entire sync operation
   including git pull/push, not just DB operations.

5. FlushManager shutdown timeout (steveyegge#7): Increase timeout from 30s to 120s
   for large databases. Add context-aware flush with progress logging.

6. WAL checkpoint error handling (steveyegge#8): Add 5-retry exponential backoff
   for checkpoint failures. Show user warning on persistent failure.

Also includes comprehensive race condition test suite covering:
- Concurrent export/import operations
- Daemon lifecycle and lock handling
- Sync/flush race scenarios
- SQLite store concurrency

Co-Authored-By: SageOx <[email protected]>
rsnodgrass added a commit that referenced this pull request Jan 23, 2026
Add flock-based shared/exclusive locking for JSONL reads (auto-import)
and writes (export/flush) to prevent corruption when sync and daemon
flush race.

Security fixes from SECURITY_AUDIT.md:
- Issue #1: Sync export and daemon auto-flush race causing data loss
- Issue steveyegge#3: Auto-import reading partially-written JSONL during export

Changes:
- Add jsonl_lock.go with JSONLLock type using gofrs/flock (existing dep)
- Update exportToJSONLDeferred to acquire exclusive lock during export
- Update autoImportIfNewer to acquire shared lock during import
- Add comprehensive race condition tests

The JSONL lock ensures that:
- Export operations have exclusive access (no concurrent reads/writes)
- Import operations share access with other readers, block during writes
- Lock is held for entire export+commit+finalize sequence

Co-Authored-By: SageOx <[email protected]>
rsnodgrass added a commit that referenced this pull request Jan 23, 2026
Add flock-based shared/exclusive locking for JSONL reads (auto-import)
and writes (export/flush) to prevent corruption when sync and daemon
flush race.

**No new dependencies** - uses existing gofrs/flock v0.13.0 (already in go.mod)

Security fixes from SECURITY_AUDIT.md:
- Issue #1: Sync export and daemon auto-flush race causing data loss
- Issue steveyegge#3: Auto-import reading partially-written JSONL during export

Changes:
- Add jsonl_lock.go with JSONLLock type using gofrs/flock
- Update exportToJSONLDeferred to acquire exclusive lock during export
- Update autoImportIfNewer to acquire shared lock during import
- Add comprehensive race condition tests
- Add generateUniqueTestID helper for test isolation

The JSONL lock ensures that:
- Export operations have exclusive access (no concurrent reads/writes)
- Import operations share access with other readers, block during writes
- Lock is held for entire export+commit+finalize sequence

Co-Authored-By: SageOx <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants