txhash: cold-store streamhash index — build + read (#728) by tamirms · Pull Request #780 · stellar/stellar-rpc

tamirms · 2026-06-08T22:24:38Z

Part of #728 — implements the buildable slice: the cold (immutable-file) half of the txhash store.

Base: stacked on events-eventstore-core (#756) for github.com/stellar/streamhash. Retarget to feature/full-history once #756 lands.

What this adds

A per-index streamhash MPHF over (txhash, ledgerSeq) and a ColdReader that resolves a tx hash to the ledger it was committed in.

cold_format.go — per-index on-disk format: 3-byte ledger-seq payload (offset from MinLedger), 1-byte fingerprint, and an 8-byte [MinLedger, MaxLedger] coverage anchor in the index metadata. ColdReader.Get(hash) -> ledgerSeq, ErrNotFound on miss, idempotent mmap Close; MinLedger()/MaxLedger() report the index's ledger span so a reader can pick an index without probing it. Rejects a payload width ≠ ColdPayloadSize at open.
cold_index.go + cold_merge.go — BuildColdIndex(min, max) merges per-chunk sorted .bin files into one streamhash index via a parallel O_DIRECT fan-in merge (ported from streamhash cmd/bench), fed single-pass into the sorted-mode builder so I/O, merge CPU, and the MPHF build overlap. Validates that the coverage span fits the payload and every entry falls in [min, max] — so the advertised coverage is a real invariant, not just metadata. Header/size guards, ctx cancellation, first-error-wins pipeline.
odirect_{linux,other}.go — O_DIRECT page-cache bypass on Linux (cached-open fallback on tmpfs/EINVAL; no-op elsewhere).

Tuning (measured on warm macOS + cold Linux NVMe over 382M real keys)

streamhash block-build workers default to NumCPU/2 (the end-to-end gate; ~2.7× over serial).
merge leaves capped at NumCPU/2: NumCPU/2 leaves + NumCPU/2 builders fill the cores without oversubscription — ~+18% e2e on cold NVMe vs NumCPU, neutral on warm.
k1 merge tiebreak kept for byte-reproducible output regardless of input order (k0 alone already satisfies streamhash's block routing).
Sorted k-way merge kept over NewUnsortedBuilder (measured 1.7–6× faster for the pre-sorted inputs).

Peak build memory is bounded by file count, not entry count — ~145 MiB for a full 1000-chunk index, ~constant from 5M to 50M+ keys (the index file is pwrite-streamed, not held in RAM).

Out of scope (blocked on unbuilt deps)

Production build wiring to the cold txhash ingester (Ingestion: all data types for both cold and hot stores #765) — the .bin input format is the documented seam for it.
The getTransaction read assembly over the tx-details-by-hash view (XDR view extractors: events, tx-hashes, tx-details, tx-pages from LedgerCloseMetaView #764) + cold ledger reader (Ledger query path over packfiles (immutable-file reader) #725). ColdReader stops at ledgerSeq.

Tests

Build/query round-trip, miss, concurrent reads, fan-in tree (sorted + complete), large-file refill, ledger coverage (metadata + per-entry bounds), and error/format guards. go test -race + golangci-lint clean. No benchmarks ship — the tuning sweeps that produced the figures above were removed once the decisions were settled.

🤖 Generated with Claude Code

…tmaps Rework the in-memory events index and the full-history eventstore as one unit — they are tightly coupled, since the eventstore consumes the events index types and the membitmaps removal forces the eventstore migration in lockstep. The query engine (Query + postfilter) is deliberately left out and follows in a separate stacked PR. events: - Add ConcurrentBitmaps and ConcurrentLedgerOffsets for lock-free concurrent reads during ingest; remove the old membitmaps implementation. - Add the ingest_view path: build the index directly from xdr LedgerCloseMetaView / TransactionMetaView zero-copy views. - payload / index / ledgeroffsets / bitmaps reworked accordingly. eventstore: - Migrate the cold store (format / index / reader / writer) and the hot store onto the new events index API. - reader.go interface updates. deps: roaring v2.18.0 -> v2.18.2 (upstream FastOr/runContainer16 fix), go-stellar-sdk bump (XDR View types used by ingest_view), and tamirms/streamhash promoted to a direct dependency. The eventstore.Query concurrency test (TestHotStore_QueryUnderConcurrentIngest) moves to the query-engine PR alongside query.go. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The view-based ingest extractor (ingest_view.go / LCMToPayloadsFromRaw) and its Payload term-precompute plumbing move to the separate #764 work, so remove them here along with the now-dead TermKeys() skip-branch in IngestLedgerEvents and the V3-SorobanMeta test fixture only the extractor's test used. Payload now carries the event purely as raw XDR (ContractEventBytes) — the decoded xdr.ContractEvent field is gone. Ingest (LCMToPayloads) marshals each event into the bytes, and terms are derived straight from the raw XDR via xdr.ContractEventView (events.TermsForBytes), no full UnmarshalBinary. Remove the per-Reader useXDRViews toggle from HotStore and ColdReader; the read path always decodes via views. Payload.Unmarshal is the sole consumer decoder (struct decoder removed; former UnmarshalView renamed to Unmarshal). FetchEvents returns owned Payloads; FetchRange/All yield borrowed Payloads (ContractEventBytes aliases the iterator's step buffer — clone to retain). IngestLedgerEvents marshals each payload into one reused scratch buffer (BatchWriter.Put copies the value synchronously), and is idempotent on retry: re-ingesting an already-committed ledger is a no-op (a gap or out-of-range ledger still errors). Warmup now cross-checks the per-chunk CFs on open (verifyChunkConsistency): the index may not reference an event beyond the committed count, and the data tail must align with it (event total-1 present, nothing at id >= total) — a corrupt or tampered chunk fails to open loudly instead of serving an inconsistent cache. ConcurrentLedgerOffsets.Append is now a single positional primitive (no ledger arg, no error); the sequence, capacity, and cumulative-overflow checks live at the warmup trust boundary in warmupOffsets, where on-disk rows are untrusted. Deps: - go-stellar-sdk -> latest main (v0.5.1-0.20260604220920-ff1e140adca5) - streamhash -> github.com/stellar/streamhash (was tamirms/streamhash) - roaring/v2 unchanged at v2.18.2 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 909541d7a4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Copilot

Pull request overview

Implements the “cold” (immutable-file) half of the txhash store by introducing an on-disk streamhash MPHF index format, a parallel k-way merge + build pipeline for producing those indexes from per-chunk .bin inputs, and an mmap-backed reader for resolving txhash -> ledgerSeq.

Changes:

Define the cold txhash index format (payload/fingerprint sizing + [MinLedger, MaxLedger] metadata) and helpers for index grouping/naming.
Add BuildColdIndex (validated, parallel fan-in merge feeding streamhash.NewSortedBuilder) and supporting merge implementation with optional O_DIRECT reads on Linux.
Add ColdReader to open/query a cold index with payload-width + metadata validation, plus comprehensive unit tests.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_format.go	Defines cold index format constants, metadata encoding/decoding, and index naming/grouping helpers.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_format_test.go	Validates metadata parsing/encoding, grouping math, and reader guardrails (metadata/payload size) + fingerprint behavior.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_index.go	Adds `BuildColdIndex` with coverage-span validation, input header/size validation, and streamhash builder configuration.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_index_test.go	End-to-end build/query fixtures and extensive failure/guard coverage for the cold index build path.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_merge.go	Implements the parallel, pooled-batch k-way merge fan-in tree feeding the sorted builder; includes O_DIRECT-capable file reading path.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_merge_test.go	Exercises multi-level fan-in merge tree ordering/completeness independent of CPU count.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_reader.go	Adds mmap-backed reader for point lookups with payload-width + metadata validation and consistent error mapping.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_reader_test.go	Tests reader open errors, hit/miss, idempotent close, covered range, closed behavior, and concurrent lookups.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/odirect_linux.go	Linux-specific O_DIRECT open flag plumbing.
cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/odirect_other.go	Non-Linux fallback for direct-open flag (no page-cache bypass).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e1f03c0c6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chowbao · 2026-06-09T16:46:33Z

+
+// CoveredRange returns the inclusive [minLedger, maxLedger] the index covers.
+// Immutable; safe after Close.
+func (r *ColdReader) CoveredRange() (uint32, uint32) {


nit: Hmm in other code we normally have a min and max function right? For completeness it might be useful to make func for minLedger() and maxLedger()

View-based ingestion at internal/fullhistory/ingest: - HotIngester / ColdIngester over xdr.LedgerCloseMetaView; 6 ingesters (ledger/events/txhash x hot/cold) using the #764 view extractors; hot stores injected (daemon-owned), cold writers per-chunk. - HotService (parallel per-LCM fan-out) / ColdService (sequential + Finalize), config-driven; MetricSink + Prometheus impl; extensible ChunkSource (pack/GCS/S3/datastore). - drain validates per-ledger LedgerSequence; cold tier finalizes per-chunk; failure-path + metric coverage. The cold txhash store (build + read) is PR #780 (#728); this PR does not bundle it. The cold txhash ingester writes the per-chunk sorted .bin files (8-byte LE count + 20-byte [16B key + LE uint32 seq] entries, sorted by big-endian key prefix, named <chunkID:08d>.bin) that #780's BuildColdIndex consumes — the documented seam. Uses the base txhash hot store API (NewHotStore/Get). Closes #765.

View-based ingestion at internal/fullhistory/ingest: HotIngester / ColdIngester over xdr.LedgerCloseMetaView; 6 ingesters (ledger/events/ txhash x hot/cold) using the #764 view extractors; hot stores injected, cold writers per-chunk; HotService (parallel per-LCM fan-out) / ColdService (sequential + Finalize); MetricSink + Prometheus impl; extensible ChunkSource (pack/GCS/S3/datastore); per-ledger seq guard. The cold txhash store (build + read) is PR #780 (#728); this PR does not bundle it — the cold txhash ingester writes the per-chunk sorted .bin files (8-byte LE count + 20-byte [16B key + LE uint32 seq] entries, big-endian key order, <chunkID:08d>.bin) that #780's BuildColdIndex consumes. Uses the base txhash hot store API. Closes #765.

View-based ingestion at internal/fullhistory/ingest: HotIngester / ColdIngester over xdr.LedgerCloseMetaView; 6 ingesters using the #764 view extractors; hot stores injected, cold writers per-chunk; HotService (parallel per-LCM fan-out) / ColdService (sequential + Finalize); MetricSink + Prometheus impl; extensible ChunkSource (pack/GCS/S3). eventless chunks: eventsCold.Finalize treats ErrEmptyBuildSet as success (keeps the valid empty events.pack, skips the index) — a real WriteColdIndex failure still removes the orphan pack. drain validates per-ledger LedgerSequence; per-ledger seq guard; failure-path + metric coverage. Uses base txhash hot API; cold txhash store is PR #780, fed by the per-chunk .bin this writes. NOTE (cross-package follow-up): eventstore.OpenColdReader must learn to serve an eventless chunk (pack present, index absent) as zero events. Closes #765.

Implements the buildable slice of #728: the cold (immutable-file) half of the txhash store — a per-index streamhash MPHF over (txhash, ledgerSeq) and a ColdReader that resolves a tx hash to the ledger it was committed in. - cold_format.go: per-index format — 3-byte ledger-seq payload (offset from the index's MinLedger), 1-byte fingerprint, and an 8-byte [MinLedger, MaxLedger] coverage anchor in the index metadata. ColdReader.Get(hash) -> ledgerSeq, ErrNotFound on miss, idempotent mmap Close; MinLedger() / MaxLedger() report the index's ledger span so a reader can pick an index without probing it. Rejects a payload width other than ColdPayloadSize at open. - cold_index.go + cold_merge.go: BuildColdIndex(min, max) merges per-chunk sorted .bin files into one streamhash index via a parallel O_DIRECT fan-in merge (ported from streamhash cmd/bench), fed single-pass into the sorted-mode builder so I/O, merge CPU, and the MPHF build overlap. Validates the coverage span fits the payload and every entry falls in [min, max], so the advertised coverage is a real invariant; header/size guards; ctx cancellation; first-error-wins pipeline. - odirect_{linux,other}.go: O_DIRECT page-cache bypass on Linux, cached-open fallback on tmpfs/EINVAL, no-op elsewhere. Measured tuning (warm macOS + cold Linux NVMe over 382M real keys): - streamhash block-build workers default to NumCPU/2 (the e2e gate; ~2.7x over serial, saturates at NumCPU/2). - merge leaves capped at NumCPU/2: NumCPU/2 leaves + NumCPU/2 builders fill the cores without oversubscription; ~+18% e2e cold vs NumCPU, neutral warm. - k1 merge tiebreak kept for byte-reproducible output regardless of input order (k0 alone already satisfies streamhash's block routing). - Sorted k-way merge kept over NewUnsortedBuilder (measured 1.7-6x faster for the pre-sorted inputs). Peak build memory is bounded by file count, not entry count (~145 MiB for a full 1000-chunk index, ~constant from 5M to 50M+ keys; the index file is pwrite-streamed, not held in RAM). Out of scope (blocked on unbuilt deps): production build wiring to the cold txhash ingester (#765) and the getTransaction read assembly over the tx-details-by-hash view (#764) + cold ledger reader (#725). The .bin input format is the documented seam for #765. Tests cover build/query round-trip, miss, concurrent reads, fan-in tree, large-file refill, ledger coverage (metadata + per-entry bounds), and error/format guards. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

View-based ingestion at internal/fullhistory/ingest: HotIngester / ColdIngester over xdr.LedgerCloseMetaView; 6 ingesters using the #764 view extractors; hot stores injected, cold writers per-chunk; HotService (parallel per-LCM fan-out) / ColdService (sequential + Finalize); MetricSink + Prometheus impl; extensible ChunkSource (pack/GCS/S3). eventless chunks keep the empty events.pack (skip index, no failure); drain validates per-ledger LedgerSequence. Uses base txhash hot API; the cold txhash store is PR #780, fed by the per-chunk .bin this writes. NOTE (cross-package follow-up): eventstore.OpenColdReader must serve an eventless chunk (pack present, index absent) as zero events. Closes #765.

View-based ingestion at internal/fullhistory/ingest: HotIngester / ColdIngester over xdr.LedgerCloseMetaView; 6 ingesters using the #764 view extractors; HotService (parallel per-LCM fan-out) / ColdService (sequential + Finalize); MetricSink + Prometheus impl; extensible ChunkSource (pack/GCS/S3). eventless chunks keep the empty events.pack (skip index, no failure); drain validates per-ledger LedgerSequence. Uses base txhash hot API; cold txhash store is PR #780, fed by the per-chunk .bin this writes. NOTE (cross-package follow-up): eventstore.OpenColdReader must serve an eventless chunk (pack present, index absent) as zero events. Closes #765.

…ectations Pin the tx-hash cold-index format the streaming rebuild produces to the merged #728/#780 cold path, and record the design's Part-4 perf figures. perf_test.go: - TestStreamingRebuild_ByteIdenticalToColdPath builds the SAME coverage via the streaming buildTxhashIndex and a direct txhash.BuildColdIndex over the same .bin inputs, asserting the two .idx files are byte-identical -- the precondition that lets the bench-fullhistory figures transfer. - TestStreamingBin_MatchesSpecFormat / TestStreamingIdx_MatchesSpecFormat pin the on-disk formats to gettransaction sec 6.1/6.2 (16-byte key, 3-byte payload offset from MinLedger, 1-byte fingerprint, uint64-LE count header, [MinLedger,MaxLedger] metadata). - TestColdIndexSizing_ConsistentWithPart4 asserts a B/tx sanity band around the design's ~4.2 B/tx and the inviolable 4 B/tx payload+fingerprint floor. PERF.md records the expected figures (~1-min dense-window rebuild, ~4.2 B/tx index, ~60 GB .bin floor) and points at bench-fullhistory on rpc-hack as the measurement source -- transferred because the formats are byte-identical, not re-measured here.

tamirms and others added 2 commits May 28, 2026 14:04

tamirms force-pushed the txhash-cold-store-728 branch 3 times, most recently from d639670 to 909541d Compare June 9, 2026 10:58

tamirms marked this pull request as ready for review June 9, 2026 11:13

tamirms requested a review from Copilot June 9, 2026 11:13

Copilot started reviewing on behalf of tamirms June 9, 2026 11:13 View session

chatgpt-codex-connector Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_index.go

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_merge.go

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_merge.go Outdated

tamirms added rpc-sdk-scrum rpc-full-history labels Jun 9, 2026

tamirms added this to Platform Scrum Jun 9, 2026

github-project-automation Bot moved this to To Do in Platform Scrum Jun 9, 2026

tamirms added this to the platform sprint 72 milestone Jun 9, 2026

tamirms moved this from To Do to Needs Review in Platform Scrum Jun 9, 2026

tamirms force-pushed the txhash-cold-store-728 branch from 909541d to 1e1f03c Compare June 9, 2026 11:27

chatgpt-codex-connector Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/odirect_linux.go

tamirms requested a review from a team June 9, 2026 11:39

chowbao reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_index.go

chowbao reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_index.go

chowbao reviewed Jun 9, 2026

View reviewed changes

chowbao approved these changes Jun 9, 2026

View reviewed changes

chowbao reviewed Jun 9, 2026

View reviewed changes

Comment thread cmd/stellar-rpc/internal/fullhistory/pkg/stores/txhash/cold_reader.go Outdated

chowbao mentioned this pull request Jun 9, 2026

ingest: full-history ingestion for cold + hot stores (#765) #779

Merged

1 task

tamirms force-pushed the txhash-cold-store-728 branch from 1e1f03c to c0577a8 Compare June 9, 2026 19:34

Base automatically changed from events-eventstore-core to feature/full-history June 16, 2026 13:05

tamirms merged commit 2208876 into feature/full-history Jun 17, 2026
15 checks passed

tamirms deleted the txhash-cold-store-728 branch June 17, 2026 10:44

github-project-automation Bot moved this from Needs Review to Done in Platform Scrum Jun 17, 2026

Navigation Menu

Uh oh!

txhash: cold-store streamhash index — build + read (#728) - #780

txhash: cold-store streamhash index — build + read (#728)#780
tamirms merged 3 commits into
feature/full-historyfrom
txhash-cold-store-728

tamirms commented Jun 8, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chowbao Jun 9, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tamirms commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Tuning (measured on warm macOS + cold Linux NVMe over 382M real keys)

Out of scope (blocked on unbuilt deps)

Tests

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chowbao Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tamirms commented Jun 8, 2026 •

edited

Loading