Update packfile design doc to match merged writer and reader implementations - #737

Summary

The packfile design doc has been out of sync with the implementation. Most of the divergence came in with the writer (#702) and was never reflected in the doc; the reader (#712) added a few more deltas on top. This PR brings the doc end-to-end into alignment with the merged code.

No code changes — doc-only.

Major changes

From #702 (writer) — doc was never updated

• Format model. WriterOptions.Format is now a caller-assigned Format uint32 identifier, not the old RecordFormat enum (Compressed / Uncompressed / Raw). The built-in formats are gone — codec is plug-in via NewRecordEncoder.
• New WriterOptions fields. NewRecordEncoder func() RecordEncoder (per-worker factory), ContentHashExtract for per-item transformation before hashing.
• Trailer is 76 bytes (was 64). New format uint32 field at offset 8; only flagContentHash remains (flagNoCompression / flagNoCRC are gone). Magic constant corrected to 0x48434C53. Worked-example file-size math updated.
• Records. Caller-encoded payload + library-managed item-size-index CRC. Per-record payload integrity is encoder-supplied (zstd's xxHash64, or caller's choice, or none with passthrough).
• Writer single-goroutine constraint documented (parallel to the Reader's concurrency disclosure).
• AppendItem semantics: no-args is a no-op, []byte{} records an empty item, parts are copied, oversized items error.
• Sync nit: doc said fdatasync; the code calls os.File.Sync() which is fsync.
• Concurrency sizing advice: values above runtime.NumCPU() tend to hurt throughput.
• ErrWriterClosed added to the Errors section.

From #712 (reader)

• ReaderOptions is a struct (was variadic WithConcurrency). Default Concurrency is 1 (serial), not 8.
• RecordDecoder is a single concurrent-safe instance, caller-owned, not closed by Reader.Close.
• Error semantics. ReadItems / ReadRange return errors instead of panicking. New ErrPositionsUnsorted sentinel.
• ReadRange lifetimes. Yielded slice is invalidated after the for-range loop exits (break or natural completion); iterator itself is not safe for concurrent iteration; count == 0 is valid.
• ReadItems cancellation / atomicity. Context is checked at batch boundaries, not between items inside a batch; ReadItems is not atomic on error (fn may have run for some positions and not others). Empty positions slice is valid. fn's error is returned verbatim.
• Close blocks on the background open if it is still in flight.
• Implementation notes. ReadItems uses errgroup.WithContext (no more atomic-flag cancellation); single concurrent-safe decoder + package-level workspace pool replaces the old per-Reader decoder pool.

New section

• Codec Contract. Documents Format uint32, the RecordEncoder / RecordDecoder interfaces, and the in-tree zstd codec (*zstd.Compressor / *zstd.Decompressor).

Polish

• Disambiguated "format version" (the on-disk version byte vs the caller-assigned Format field) at the index-encoding section.
• Used KiB/MiB binary prefixes where referring to specific code constants (1 MiB pooled buffer, 256 KiB speculative read).
• Parallel-pipeline gate description corrected (triggered by encoder/hash, not by Concurrency).

Verification

Doc was fact-checked end-to-end against the current implementation in cmd/stellar-rpc/internal/packfile/ via five review passes: three parallel agents on API surface, on-disk format, and behavior/implementation; followed by deep audits dedicated to the writer side and the reader side. All trailer offsets, struct fields, method signatures, error names, default values, lifetime contracts, and implementation claims verified against code.

Test plan

• No code changes; no tests to run
• Doc renders cleanly in GitHub preview
• All anchors / cross-references resolve

🤖 Generated with Claude Code