fix(chunker): stop emitting a redundant chunk at the end of every file - #333

Description

Chunker.Chunk (indexer/chunker.go) steps back by the overlap after every chunk, including the chunk that already reaches the end of the content. The loop then runs once more and emits a final chunk holding only the last overlap × 4 characters of the chunk before it. Nothing in that chunk is new.

With the default settings (size: 512, overlap: 50, so 2048/200 characters), a 1,000-character file of ten 100-character lines is indexed as two chunks: lines 1-10, then lines 9-10 again. Every file longer than 200 characters gets this extra chunk, which means one more embedding request, one more vector in the store, and a second search hit for the same lines. With search.dedup on (the default), the short tail can take the file's slot instead of the full chunk.

On this repository's own .go files, 236 of 243 files get the extra chunk: 1,425 chunks where 1,189 cover the same content, so about 17% of embedding requests are redundant. I measured this with a line-by-line Python port of Chunk, not by running grepai.

ReChunk has a copy of the same loop and the same problem.

The fix: after emitting a chunk, stop if it reaches the end of the content. Earlier chunks keep the same boundaries, IDs and hashes, so existing indexes stay valid. A file's leftover tail chunk is removed the next time that file is re-indexed, because re-indexing deletes the file's chunks first.

Related Issue

No open issue covers this. I found it while reading the chunker.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test update

How Has This Been Tested?

• Unit tests (added, not run locally; see below)
• Integration tests
• Manual testing

TestChunker_NoRedundantTrailingChunk is table-driven and covers files that fit in one, two and three chunks. TestChunker_ReChunkNoRedundantTrailingChunk covers ReChunk. Both check the exact chunk count, that the last chunk ends on the file's last line, and that every chunk ends after the one before it. I took the expected values from the Python port. On main the same inputs give 2, 3 and 4 chunks and 4 sub-chunks, and in each case the last chunk only repeats lines the previous chunk already covers.

What I did not run: I had no Go toolchain, so I did not run go test, go vet, gofmt or golangci-lint on this branch. I formatted the code by hand to gofmt style and checked both files with a tree-sitter Go parser, which checks syntax only. The existing chunker and indexer tests only assert lower bounds on chunk counts, so they should still pass, but CI will be the first real run.

Test Configuration:

• OS: macOS
• Go version: none (tests not run locally)
• Embedding provider: n/a

Checklist

• My code follows the project's code style
• I have run golangci-lint run and fixed any issues (not run, no Go toolchain)
• I have added tests that prove my fix/feature works (not run locally)
• I have updated the documentation if needed (no docs change needed)
• I have added an entry to CHANGELOG.md (if applicable)
• My changes generate no new warnings (not checked locally)
• All new and existing tests pass (not run locally)

Additional Notes

The CHANGELOG entry has no PR number yet.

🤖 Generated with Claude Code