events,eventstore: concurrent index; migrate off membitmaps - #756

Summary

Part of the series extracting reviewable components from the rpc-hack branch into feature/full-history.

This reworks the in-memory events index (internal/events) and the full-history eventstore as one unit — they're tightly coupled: the eventstore consumes the events-index types, and removing the old membitmaps implementation forces the eventstore migration in lockstep. The query engine (Query + postfilter) and the view-based ingest extractor are deliberately left out (see below) and follow in separate PRs.

internal/events

• Add ConcurrentBitmaps and ConcurrentLedgerOffsets for lock-free concurrent reads during ingest; remove the old membitmaps implementation.
• ConcurrentLedgerOffsets.Append is a single positional primitive (no ledger argument, no error) — the structure is purely positional, so ledger N lives at slot N−startLedger. Sequence / capacity / cumulative-overflow validation lives at each caller's trust boundary: ingest validates up front, and warmupOffsets validates the untrusted on-disk rows. (The cold sibling LedgerOffsets.Append keeps its validating two-arg form because its caller decodes untrusted bytes.)
• Payload carries the contract event purely as raw XDR in ContractEventBytes — the decoded xdr.ContractEvent struct field is gone. Marshal writes the bytes; Unmarshal aliases them zero-copy (no XDR decode).
• Ingest (LCMToPayloads) marshals each event into ContractEventBytes; term keys are derived straight from the raw XDR via xdr.ContractEventView (events.TermsForBytes), without a full UnmarshalBinary.
• index / ledgeroffsets / bitmaps reworked accordingly.

The view-based ingest extractor (ingest_view.go / LCMToPayloadsFromRaw, walking xdr.LedgerCloseMetaView to emit payloads zero-copy) is not in this PR — it is tracked in #764. The Payload term-precompute plumbing that only that extractor populated is removed here too.

eventstore

• Migrate the cold store (cold_format / cold_index / cold_reader / cold_writer) and the hot store onto the new events-index API.
• reader.go interface updates.
• The read path always decodes via XDR views (no struct decode). The previous per-Reader useXDRViews toggle (HotStore.WithXDRViews / ColdReaderOptions.UseXDRViews) is removed.
• Buffer ownership. FetchEvents returns owned Payloads (safe to retain). FetchRange / All yield borrowed Payloads — ContractEventBytes aliases the reader's per-step iteration buffer and is valid only until the next step, so a consumer that retains one must clone. See events.Payload.Unmarshal for the alias contract.
• IngestLedgerEvents marshals each payload into one reused scratch buffer inside the RocksDB batch (BatchWriter.Put copies the value synchronously), avoiding a per-event allocation. It is idempotent on retry — re-ingesting an already-committed ledger is a no-op, while a gap or out-of-range ledger errors.
• Warmup integrity. On open, warmup cross-checks the three per-chunk CFs (verifyChunkConsistency): the index may not reference an event beyond the committed count, and the data tail must align with it (event total-1 present, nothing at id ≥ total). A corrupt or tampered chunk fails to open loudly instead of silently serving an inconsistent in-memory cache. This is a cheap open-time tripwire on denormalized state, not load-bearing correctness — the atomic write batch guarantees these invariants for the writer, and an interior data hole (which the tripwire does not detect) is caught lazily by FetchRange's short-scan check on first read.

Dependencies

• RoaringBitmap/roaring/v2 v2.18.0 → v2.18.2 (upstream FastOr/runContainer16 fix, replacing the previous fork).
• go-stellar-sdk bumped to match the latest main (v0.5.1-0.20260604220920-ff1e140adca5).
• streamhash is a direct dependency on github.com/stellar/streamhash (moved off github.com/tamirms/streamhash).

go mod tidy pruned the upload-cold/bench-only deps, so go.mod carries only what this PR uses.

Why the simple/concurrent bitmap split (rationale)

The original memBitmaps was a single RWMutex-protected type used for both the live query path and the single-threaded cold build. It was split into a lock-free ConcurrentBitmaps (hot) and a plain, unsynchronized Bitmaps map (cold build). Motivation, from the implementing session and the (squashed) PR #751 commits — see 060993502 "split bitmaps + offsets into simple vs concurrent" and 00aa03944 "ConcurrentBitmaps termState + COW":

1. Read-path clone cost. memBitmaps.Get cloned the bitmap on every lookup so concurrent ingest couldn't mutate it mid-read. Query is read-only, so those clones are pure overhead — projected at ~15–30% of a core at 1000 QPS. (This is an analytical projection, not yet a benchmarked figure — the harness's --query-concurrency sweep is what confirms it at load.)
2. Borrow safety. The only way to drop the clone under the old design was to return a borrowed pointer, which races with ingest mutation. ConcurrentBitmaps' immutable COW snapshots (atomic.Pointer[termState]) give safe, zero-clone reads with no lock spanning the borrow.
3. Cold-path simplicity. The cold build path is single-threaded build-then-freeze and never queries, so it carries no synchronization at all — concurrency is opt-in, not the default.

The roaring index, flat-file cold storage, and hot-chunk-index structure are unchanged; only the hot in-memory index went lock-free. Note 00aa03944 records a ~+7% hot-ingest wall cost (clawed back from +42% via COW), so this trades a small write-side cost for the read-side clone elimination.

Ingest robustness (rationale)

IngestLedgerEvents writes data + index + offsets to the per-chunk DB in one atomic, WAL-synced batch, then updates the in-memory cache (mirror + offsets). The crash/retry contract rests on three properties, made explicit in this PR:

• Disk is the source of truth; the in-memory cache is rebuilt by warmup. A crash anywhere is recovered by warmup replaying the atomic batch's CFs on next open.
• The cache update is infallible by construction. The watermark advances last and ConcurrentLedgerOffsets.Append is positional, so the post-batch phase has no reachable error path; a crash is the only non-completion, and warmup repairs it.
• Validation lives at trust boundaries. Ingest validates the ledger sequence up front; warmupOffsets re-validates untrusted on-disk rows (gap / out-of-order / excess / cumulative overflow); verifyChunkConsistency cross-checks the CFs on open. Internal already-validated values are not re-checked.

Reviewer notes

• The eventstore.Query concurrency test (TestHotStore_QueryUnderConcurrentIngest) is intentionally absent here — it moves to the query-engine PR alongside query.go.
• The view-based ingest extractor and the useXDRViews read toggle were pared out of this PR (extractor → XDR view extractors: events, tx-hashes, tx-details, tx-pages from LedgerCloseMetaView #764; the read path is now unconditionally view-mode) to keep this change focused on the concurrent index + eventstore migration.

Test plan

• go build + go vet + go test ./cmd/stellar-rpc/internal/events/
• go build + go vet + go test ./cmd/stellar-rpc/internal/fullhistory/pkg/stores/eventstore/
• Warmup integrity + corruption-rejection tests (TestWarmup_Rejects*: data orphan, offsets gap, offsets overflow, missing tail, index-beyond-committed, orphan-in-empty-chunk)
• Offsets lock-free path under -race (go test -race ./cmd/stellar-rpc/internal/events/ -run ConcurrentLedgerOffsets)
• golangci-lint run clean on both packages
• go-stellar-sdk bump verified: all Go packages compile (go build ./...; the cmd/stellar-rpc binary's native preflight/xdr2json link is unchanged and environment-dependent)

🤖 Generated with Claude Code