Add getEvents full-history design doc by urvisavla · Pull Request #635 · stellar/stellar-rpc

urvisavla · 2026-03-18T18:08:46Z

What

This PR contains the design doc for the full-history getEvents storage and indexing layer.

The design covers:

Parts 1-3: Problem scope, architecture (hot/cold segments, bitmap index), and implementation reference (event addressing, write/read paths, freeze, startup, backfill)
Part 4: Capacity, performance, and scaling
- Memory profile (measured bitmap index sizing)
- Ingestion performance (backfill throughput, live ingestion latency, freeze time)
- Query performance (event fetch and index lookup benchmarks on NVMe)
- Scaling projections (disk and memory at 2x/5x/10x event density)
- Tiered storage (NVMe/EBS approach)

Tracking issue: #665

Why

#637

Test plan

Review doc for technical accuracy
Verify diagrams render correctly (mermaid flowchart in Section 11.1)

Add the RPC getEvents full-history design document with architecture diagrams. The doc covers the segment-based storage model, bitmap indexing, hot/cold segment lifecycle, and backfill process. Hot segment storage is presented as storage-backend agnostic (flat files vs embedded DB), while keeping in-memory bitmaps as a design decision.

Copilot

Pull request overview

Adds a design document for a proposed getEvents full-history storage/indexing backend, plus extracted diagram assets to support the doc.

Changes:

Added getEvents full-history design doc describing segmentation, hot/cold storage, bitmap indexing, freeze/recovery, and capacity estimates.
Added query-routing flowchart diagram as a standalone PNG asset.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 9 comments.

File	Description
`docs/getevents-full-history-design.md`	New full-history `getEvents` backend design doc (architecture, indexing, storage, query paths, capacity placeholders).
`docs/query-routing-flowchart.png`	Diagram asset referenced by the design doc for query routing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Copilot

Pull request overview

Adds a design doc describing the proposed storage + indexing architecture for serving full-history getEvents queries, along with an accompanying query-routing flowchart diagram.

Changes:

Added a full-history getEvents backend design doc (segments, bitmap indexing, hot/cold storage, freeze/backfill, query path).
Added a query-routing flowchart diagram referenced by the design doc.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 4 comments.

File	Description
docs/getevents-full-history-design.md	New full-history `getEvents` design document with architecture and implementation reference sections.
docs/query-routing-flowchart.png	New diagram referenced by the design doc’s query-routing section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Adds a design doc describing a proposed storage/indexing architecture to support full-history getEvents queries, including a query-routing flowchart diagram referenced by the doc.

Changes:

Add full-history getEvents backend design doc (segments, bitmap index, freeze/backfill flows, capacity placeholders).
Add query routing flowchart image used by the design doc.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 4 comments.

File	Description
design-docs/getevents-full-history-design.md	New design doc covering architecture + implementation reference for full-history `getEvents`.
design-docs/query-routing-flowchart.png	New diagram referenced by the doc for query routing flow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Copilot

Pull request overview

Adds a design document describing a full-history storage + indexing backend for the getEvents RPC, including segment architecture (hot/cold), roaring-bitmap indexing, freeze/backfill workflows, and query routing, with supporting diagrams.

Changes:

Add a full-history getEvents backend design doc covering data model, segment/index layout, query path, and operational workflows.
Add a query-routing flowchart diagram referenced by the doc.

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 4 comments.

File	Description
design-docs/getevents-full-history-design.md	New design doc defining storage/indexing architecture, query execution, freeze/backfill, and capacity placeholders.
design-docs/query-routing-flowchart.png	New diagram referenced by the doc for query routing flow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

karthikiyer56 · 2026-03-30T18:27:44Z

@urvisavla
i tried to trace through the cold index path end to end and couldn't follow it without filling in gaps myself.
would be really helpful to have a worked example(s) covering both build and query.

1/ build:
take a few terms from events with topics like ["transfer", "C-address-sender", "C-address-receiver", "asset-name"]. show how the term keys are computed, how the MPHF assigns slots, what gets physically written to index.pack and index.hash.

2/ query - hit:
query for contract X which exists in this chunk.
show: compute term key → feed into MPHF → get slot → read from index.pack → fingerprint check passes (first 4 bytes of term key match stored fingerprint (this is the partial match that can yeild false positives, right?) → deserialize bitmap → fetch events from events.pack → post-filter verifies fetched events actually match query terms → return results.

3/ query - miss:
query for contract Y which does NOT exist in this chunk.
show: compute term key → feed into MPHF → MPHF still returns a slot (because it maps any input) → read from index.pack → fingerprint check fails (first 4 bytes don't match) → term not present, return empty.

4/ query — multi-term:
query for contract X AND topic0 = "transfer".
show: look up both bitmaps, AND them together, iterate the intersection.

…30B/22B clarification - Rename "blocks" to "records" throughout to match packfile terminology - Add paragraph explaining queries spanning cold + hot segments - Add Notes column to Network Parameters table clarifying 30B vs 22B event counts Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

urvisavla · 2026-03-30T23:03:27Z

@urvisavla
i tried to trace through the cold index path end to end and couldn't follow it without filling in gaps myself.
would be really helpful to have a worked example(s) covering both build and query.

1/ build:
take a few terms from events with topics like ["transfer", "C-address-sender", "C-address-receiver", "asset-name"]. show how the term keys are computed, how the MPHF assigns slots, what gets physically written to index.pack and index.hash.

2/ query - hit:
query for contract X which exists in this chunk.
show: compute term key → feed into MPHF → get slot → read from index.pack → fingerprint check passes (first 4 bytes of term key match stored fingerprint (this is the partial match that can yeild false positives, right?) → deserialize bitmap → fetch events from events.pack → post-filter verifies fetched events actually match query terms → return results.

3/ query - miss:
query for contract Y which does NOT exist in this chunk.
show: compute term key → feed into MPHF → MPHF still returns a slot (because it maps any input) → read from index.pack → fingerprint check fails (first 4 bytes don't match) → term not present, return empty.

4/ query — multi-term:
query for contract X AND topic0 = "transfer".
show: look up both bitmaps, AND them together, iterate the intersection.

Good suggestion. A worked example would help with traceability, but the design doc is meant to describe the system's structure and behavior at a higher level. A concrete worked example with specific byte values, slot numbers, and step-by-step traces fits better as implementation-level documentation alongside the code, where it can stay in sync with the actual data formats. We'll add it there.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

tamirms · 2026-03-31T12:28:59Z

+### 11.2 Hot Segment Read Path
+
+```
+1. Look up bitmaps for all query terms from the in-memory concurrent map. The map ensures readers always get a consistent snapshot without blocking writes.


nit: based on what you wrote earlier ("The map is protected by a read-write lock so that concurrent readers do not block each other and only contend briefly with the single writer" )I think they do block writes but it shouldn't be a big deal because the read write lock should prevent starvation

…w fixes - Add events cold segment as third process_chunk output (PR #635) - Switch LFS from .data+.index to .pack format (PR #633) - Add chunk:{C}:events meta store key, atomic 3-flag WriteBatch - Add events_base to config Optional Sections table - Add events/ to directory structure - Add DAG setup pseudocode with explicit BUILD_READY handling - Replace ASCII dependency diagram with Mermaid flowchart - Expand LFS, BSB, MPHF acronyms on first use - Explain 10,000 multiplier in validation rules - Remove "Future: getEvents" section (events now first-class) - Remove dead pseudocode branch, hedging language

… design doc - Section 15: Memory Profile — bitmap index memory breakdown from measured data - Section 16: Ingestion Performance — backfill throughput, live ingestion latency, freeze time - Section 17: Query Performance — event fetch and index lookup benchmarks on NVMe - Section 18: Scaling Projections — disk and memory projections at 2x/5x/10x event density - Section 19: Tiered Storage — NVMe/EBS approach and EBS query latency - Remove query routing flowchart PNG (replaced by mermaid in prior commit) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Stacked on slice 1 (the ledgers skeleton); this commit's diff is only the events additions on top of it. Adds the EVENTS data type to the streaming daemon: - events column families in the per-chunk hot RocksDB (hotchunk), so one atomic synced WriteBatch per ledger now carries ledgers + events; - the events cold-segment writer in processChunk; - the chunk:{c}:events catalog key + its sweeps; - events coverage in the audit (INV-3 disk<->catalog) and in the crash-injection/convergence suite and lifecycle E2E. Events is a per-chunk artifact, like ledgers — no window/index subsystem (that is tx-hash, deferred to slice 3). Composes the events store (pkg/stores/eventstore, #740/#756) and the events design (getevents-full-history-design.md, #635), already on feature/full-history. Built against RocksDB 10.9.1 (grocksdb 1.10.7); fullhistory tree green on the non-short suite incl. the lifecycle E2E.

Copilot AI review requested due to automatic review settings March 18, 2026 18:08

Copilot started reviewing on behalf of urvisavla March 18, 2026 18:09 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Potential fix for pull request finding

116c0d5

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

urvisavla requested a review from Copilot March 18, 2026 18:15

Copilot started reviewing on behalf of urvisavla March 18, 2026 18:16 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Move getEvents design doc to design-docs directory

4f80187

urvisavla requested a review from Copilot March 18, 2026 18:22

Copilot started reviewing on behalf of urvisavla March 18, 2026 18:23 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

urvisavla and others added 2 commits March 18, 2026 11:44

Potential fix for pull request finding

89fadce

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Address review comments: fix wording and capitalization

5d5dacf

urvisavla requested a review from Copilot March 18, 2026 19:35

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Comment thread design-docs/getevents-full-history-design.md

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

Comment thread design-docs/getevents-full-history-design.md Outdated

urvisavla and others added 3 commits March 18, 2026 12:54

Potential fix for pull request finding

953549c

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Fix inconsistent event count: 28B -> 30B

f04af28

Remove unnecessary markdown escape characters

e664916

urvisavla marked this pull request as draft March 18, 2026 20:09

Fix markdown rendering: escape bracket in range notation

244cf83

tamirms changed the base branch from main to feature/full-history March 19, 2026 13:13

karthikiyer56 mentioned this pull request Mar 21, 2026

Checking in full-history design docs into stellar-rpc for review #617

Merged

tamirms reviewed Mar 21, 2026

View reviewed changes