Skip to content

xdr: Add zero-copy XDR view types and code generator - #5937

Merged
tamirms merged 3 commits into
stellar:mainfrom
tamirms:xdr-views-pr1
Apr 29, 2026
Merged

xdr: Add zero-copy XDR view types and code generator#5937
tamirms merged 3 commits into
stellar:mainfrom
tamirms:xdr-views-pr1

Conversation

@tamirms

@tamirms tamirms commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Closes #5926

Today, reading any field from a LedgerCloseMeta requires decoding the
entire message into Go structs. A typical pubnet ledger is ~1.5MB of XDR;
decoding it allocates ~8.5MB across ~107,000 objects, even if you only
need one transaction hash.

This PR adds a zero-copy view API. Every XDR type has a corresponding
View type that is a typed []byte alias. Accessors navigate into
sub-slices without allocating, unmarshaling, or copying.

Benchmarks

Across 1,000 sampled pubnet ledgers (avg ~1.5 MB):

Operation Full Decode View Speedup
Find tx by hash (early match) 6.5ms 48µs 137x
Find tx by hash (mid match) 7.2ms 125µs 58x
Find tx by hash (late match) 8.5ms 699µs 12x
Extract events by tx hash 8.4ms 723µs 12x
Extract all tx hashes 8.0ms 542µs 15x
Extract all events 6.8ms 388µs 18x
Extract all transactions 10.9ms 978µs 11x

Full decode allocates ~8.5MB / ~107K objects per ledger. Views: zero
heap allocation for navigation.

What's in this PR

  • xdrgen/ — Go code generator that reads a JSON IR emitted by the rust generator-definitions-json and produces xdr/xdr_views_generated.go.
  • xdr/xdr_views_generated.go — generated view types and runtime support (primitive views, Try/TryVoid, error types).
  • xdrgen/views_api.md — API design document.
  • Makefilexdr/xdr_views_generated.go target that regenerates views from xdr/*.x via a two-stage docker pipeline (rust → JSON IR → Go).
  • Tests:
    • xdr/xdr_view_test.go — hand-built LedgerCloseMeta → view → navigate round-trip
    • xdr/view_support_test.go — runtime-helper unit tests
    • xdr/view_randxdr_test.go — property tests over randomly-generated ledgers (round-trip and accessor correctness)
    • xdr/view_fuzz_test.goFuzzLedgerCloseMetaView with seed corpus; asserts no panic on validation or navigation
    • xdr/*_bench_test.go — view vs full-decode benchmarks on a committed ledger fixture

Copilot AI review requested due to automatic review settings April 23, 2026 00:00
@tamirms tamirms changed the title Add zero-copy XDR view types and code generator xdr: Add zero-copy XDR view types and code generator Apr 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a zero-copy “view” API for Stellar XDR that allows navigating into large XDR blobs (e.g., LedgerCloseMeta) without fully decoding into Go structs, backed by a new Go code generator that emits []byte-based view types and runtime helpers.

Changes:

  • Added xdrgen/ code generator that consumes Rust-emitted JSON IR and generates view types/runtime support.
  • Added view runtime support (scalar/opaque/array/optional helpers, Try/TryVoid, ViewError) and extensive tests/benchmarks/fuzzing around LedgerCloseMetaView.
  • Updated build tooling (Makefile, .gitignore) and added generator fixtures/golden tests.

Reviewed changes

Copilot reviewed 28 out of 31 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
xdrgen/view_union.go Union view code emission (discriminant/arm accessors, size/valid switch).
xdrgen/view_types.go Generator-internal type model for resolved view types.
xdrgen/view_support.go.txt Embedded runtime support template emitted into generated views.
xdrgen/view_struct.go Struct view code emission (field accessors, size/valid traversal).
xdrgen/view_plan_test.go Unit tests for view planning (struct/union/inline types).
xdrgen/view_plan.go Planning phase: resolves IR into an emission-ready plan (incl. inline types).
xdrgen/view_emitters.go Shared emit helpers + concrete emitters (arrays/opaque/optional/enums/typedefs).
xdrgen/type_resolver_test.go Unit tests for TypeResolver’s ViewType mapping and typedef handling.
xdrgen/type_resolver.go TypeRef → ViewType resolution (scalars, opaque, arrays, optionals, refs).
xdrgen/testdata/mini_views.golden Golden output for generator regression testing.
xdrgen/testdata/mini.x Minimal XDR schema fixture covering generator patterns.
xdrgen/testdata/mini.json Rust JSON IR fixture derived from mini.x.
xdrgen/main.go CLI entrypoint for running the generator over a JSON IR file.
xdrgen/ir.go JSON IR schema/types and custom unmarshalling for definition wrappers.
xdrgen/golden_test.go Golden test that compares generated output to committed fixture.
xdrgen/gen_views.go Generator orchestration (plan → emit; embeds runtime template).
xdrgen/codegen_test.go Tests for naming/casing utilities used by the generator.
xdrgen/codegen.go Generated file/printer utilities and Go naming conversion helpers.
xdr/xdr_view_test.go Hand-built round-trip test navigating LedgerCloseMetaView.
xdr/views_api.md Design and usage documentation for the view API (incl. safety model).
xdr/view_support_test.go Unit tests for runtime helpers (ViewError, Try, scalar/opaque helpers).
xdr/view_randxdr_test.go Property-style tests using randxdr to validate Raw()/accessor correctness.
xdr/view_fuzz_test.go Fuzz test ensuring no panics on validation/navigation for arbitrary bytes.
xdr/try_bench_test.go Benchmarks comparing direct error handling vs Must+Try navigation patterns.
xdr/selective_decode_bench_test.go Benchmark comparing full decode vs view-based selective extraction.
xdr/extract_tx_bench_test.go Benchmarks extracting tx summaries/hashes via decode vs views.
xdr/event_extraction_bench_test.go Benchmarks extracting Soroban events via decode vs views.
Makefile Adds regeneration target for xdr_views_generated.go via Rust→JSON IR→Go pipeline.
.gitignore Ignores generator binary and intermediate IR file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread xdrgen/view_support.go.txt
Comment thread xdr/try_bench_test.go Outdated
Comment thread xdr/try_bench_test.go Outdated
Comment thread xdrgen/view_support.go.txt
Comment thread Makefile
Comment thread xdrgen/main.go Outdated
Comment thread xdr/try_bench_test.go Outdated
Comment thread xdr/try_bench_test.go Outdated
Comment thread xdr/try_bench_test.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 28 out of 31 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread xdrgen/ir.go
@tamirms
tamirms force-pushed the xdr-views-pr1 branch 2 times, most recently from f63ef77 to 383891d Compare April 24, 2026 13:37
Introduces a zero-copy view API over XDR bytes: every XDR type has a
corresponding View type that is a typed []byte alias. Views expose
accessor methods that navigate into sub-slices without allocating,
unmarshaling, or copying — enabling consumers to read a few fields out
of a large encoded value (e.g., LedgerCloseMeta) at a fraction of the
cost of a full decode.

Structure:
- xdrgen/ — Go-based code generator. Reads a JSON IR emitted by the
  rust generator-definitions-json tool, produces xdr/xdr_views_generated.go.
  The generator itself uses an inlineable array-traversal helper to
  ensure generic dispatch doesn't sneak into hot paths.
- xdr/xdr_views_generated.go — Generated view types and runtime support
  (primitive views, error types, Try/TryVoid panic-catching helpers).
- xdr/views_api.md — API design document covering the view model,
  method shapes, and consumer patterns.
- Makefile — `xdr/xdr_views_generated.go` target runs the two-stage
  pipeline (rust IR emitter in docker, Go xdrgen) to regenerate views
  from .x files.

Tests and benchmarks:
- xdr/xdr_view_test.go — round-trip integration test on a hand-built
  LedgerCloseMeta.
- xdr/view_support_test.go — focused tests for runtime helpers:
  safeUint32ToInt overflow, BoolView invalid discriminants, VarOpaqueView
  edge cases (empty, truncated, non-zero padding), arrayViewCount bounds,
  Try/TryVoid panic recovery, ViewError formatting.
- xdr/view_randxdr_test.go — property-style tests on randxdr-generated
  LedgerCloseMeta. RawRoundTrip proves the top-level slice matches input;
  AccessorCorrectness navigates to a struct field and a variable-array
  element and compares sub-view Raw() against value-side MarshalBinary().
- xdr/view_fuzz_test.go — FuzzLedgerCloseMetaView with seed corpus from
  a real ledger fixture plus randxdr-generated ledgers. Asserts no panic
  on ValidateFull, Raw, or field navigation for arbitrary bytes.
- xdr/event_extraction_bench_test.go, xdr/extract_tx_bench_test.go,
  xdr/selective_decode_bench_test.go, xdr/try_bench_test.go —
  benchmarks comparing view-based access vs full-decode on a committed
  real ledger fixture (xdr/testdata/ledger_58752000.bin).

This PR covers only the view implementation and tests. Consumer-side
integration (wiring views into BufferedStorageBackend for real ingest
paths) is a separate follow-up PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 32 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tamirms
tamirms requested a review from a team April 24, 2026 14:14

@Shaptic Shaptic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing! A few security comments below but otherwise LGTM (as best as I can grok it).

Comment thread xdrgen/type_resolver.go
Comment thread xdrgen/type_resolver.go Outdated
Two defensive fixes for malformed IR input (codegen-time, trusted source
in practice but cheap to harden):

- resolveTypeRef now returns (*TypeRef, error) and uses a visited-set to
  detect cyclic typedef chains (e.g., typedef A B; typedef B A;) instead
  of looping forever. caseValueExpr propagates this error through
  planUnion.

- sizeVal returns (uint32, error) and rejects values exceeding
  math.MaxUint32 instead of silently truncating. Callers in
  BuildViewType wrap with field-context messages.

Addresses reviewer feedback on PR stellar#5937.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@tamirms
tamirms merged commit 3e86de9 into stellar:main Apr 29, 2026
11 checks passed
@tamirms
tamirms deleted the xdr-views-pr1 branch April 29, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zero-copy XDR view types

3 participants