xdr: Add zero-copy XDR view types and code generator - #5937
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a zero-copy “view” API for Stellar XDR that allows navigating into large XDR blobs (e.g., LedgerCloseMeta) without fully decoding into Go structs, backed by a new Go code generator that emits []byte-based view types and runtime helpers.
Changes:
- Added
xdrgen/code generator that consumes Rust-emitted JSON IR and generates view types/runtime support. - Added view runtime support (scalar/opaque/array/optional helpers,
Try/TryVoid,ViewError) and extensive tests/benchmarks/fuzzing aroundLedgerCloseMetaView. - Updated build tooling (
Makefile,.gitignore) and added generator fixtures/golden tests.
Reviewed changes
Copilot reviewed 28 out of 31 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| xdrgen/view_union.go | Union view code emission (discriminant/arm accessors, size/valid switch). |
| xdrgen/view_types.go | Generator-internal type model for resolved view types. |
| xdrgen/view_support.go.txt | Embedded runtime support template emitted into generated views. |
| xdrgen/view_struct.go | Struct view code emission (field accessors, size/valid traversal). |
| xdrgen/view_plan_test.go | Unit tests for view planning (struct/union/inline types). |
| xdrgen/view_plan.go | Planning phase: resolves IR into an emission-ready plan (incl. inline types). |
| xdrgen/view_emitters.go | Shared emit helpers + concrete emitters (arrays/opaque/optional/enums/typedefs). |
| xdrgen/type_resolver_test.go | Unit tests for TypeResolver’s ViewType mapping and typedef handling. |
| xdrgen/type_resolver.go | TypeRef → ViewType resolution (scalars, opaque, arrays, optionals, refs). |
| xdrgen/testdata/mini_views.golden | Golden output for generator regression testing. |
| xdrgen/testdata/mini.x | Minimal XDR schema fixture covering generator patterns. |
| xdrgen/testdata/mini.json | Rust JSON IR fixture derived from mini.x. |
| xdrgen/main.go | CLI entrypoint for running the generator over a JSON IR file. |
| xdrgen/ir.go | JSON IR schema/types and custom unmarshalling for definition wrappers. |
| xdrgen/golden_test.go | Golden test that compares generated output to committed fixture. |
| xdrgen/gen_views.go | Generator orchestration (plan → emit; embeds runtime template). |
| xdrgen/codegen_test.go | Tests for naming/casing utilities used by the generator. |
| xdrgen/codegen.go | Generated file/printer utilities and Go naming conversion helpers. |
| xdr/xdr_view_test.go | Hand-built round-trip test navigating LedgerCloseMetaView. |
| xdr/views_api.md | Design and usage documentation for the view API (incl. safety model). |
| xdr/view_support_test.go | Unit tests for runtime helpers (ViewError, Try, scalar/opaque helpers). |
| xdr/view_randxdr_test.go | Property-style tests using randxdr to validate Raw()/accessor correctness. |
| xdr/view_fuzz_test.go | Fuzz test ensuring no panics on validation/navigation for arbitrary bytes. |
| xdr/try_bench_test.go | Benchmarks comparing direct error handling vs Must+Try navigation patterns. |
| xdr/selective_decode_bench_test.go | Benchmark comparing full decode vs view-based selective extraction. |
| xdr/extract_tx_bench_test.go | Benchmarks extracting tx summaries/hashes via decode vs views. |
| xdr/event_extraction_bench_test.go | Benchmarks extracting Soroban events via decode vs views. |
| Makefile | Adds regeneration target for xdr_views_generated.go via Rust→JSON IR→Go pipeline. |
| .gitignore | Ignores generator binary and intermediate IR file. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 31 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f63ef77 to
383891d
Compare
Introduces a zero-copy view API over XDR bytes: every XDR type has a corresponding View type that is a typed []byte alias. Views expose accessor methods that navigate into sub-slices without allocating, unmarshaling, or copying — enabling consumers to read a few fields out of a large encoded value (e.g., LedgerCloseMeta) at a fraction of the cost of a full decode. Structure: - xdrgen/ — Go-based code generator. Reads a JSON IR emitted by the rust generator-definitions-json tool, produces xdr/xdr_views_generated.go. The generator itself uses an inlineable array-traversal helper to ensure generic dispatch doesn't sneak into hot paths. - xdr/xdr_views_generated.go — Generated view types and runtime support (primitive views, error types, Try/TryVoid panic-catching helpers). - xdr/views_api.md — API design document covering the view model, method shapes, and consumer patterns. - Makefile — `xdr/xdr_views_generated.go` target runs the two-stage pipeline (rust IR emitter in docker, Go xdrgen) to regenerate views from .x files. Tests and benchmarks: - xdr/xdr_view_test.go — round-trip integration test on a hand-built LedgerCloseMeta. - xdr/view_support_test.go — focused tests for runtime helpers: safeUint32ToInt overflow, BoolView invalid discriminants, VarOpaqueView edge cases (empty, truncated, non-zero padding), arrayViewCount bounds, Try/TryVoid panic recovery, ViewError formatting. - xdr/view_randxdr_test.go — property-style tests on randxdr-generated LedgerCloseMeta. RawRoundTrip proves the top-level slice matches input; AccessorCorrectness navigates to a struct field and a variable-array element and compares sub-view Raw() against value-side MarshalBinary(). - xdr/view_fuzz_test.go — FuzzLedgerCloseMetaView with seed corpus from a real ledger fixture plus randxdr-generated ledgers. Asserts no panic on ValidateFull, Raw, or field navigation for arbitrary bytes. - xdr/event_extraction_bench_test.go, xdr/extract_tx_bench_test.go, xdr/selective_decode_bench_test.go, xdr/try_bench_test.go — benchmarks comparing view-based access vs full-decode on a committed real ledger fixture (xdr/testdata/ledger_58752000.bin). This PR covers only the view implementation and tests. Consumer-side integration (wiring views into BufferedStorageBackend for real ingest paths) is a separate follow-up PR. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 32 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Shaptic
left a comment
There was a problem hiding this comment.
Amazing! A few security comments below but otherwise LGTM (as best as I can grok it).
Two defensive fixes for malformed IR input (codegen-time, trusted source in practice but cheap to harden): - resolveTypeRef now returns (*TypeRef, error) and uses a visited-set to detect cyclic typedef chains (e.g., typedef A B; typedef B A;) instead of looping forever. caseValueExpr propagates this error through planUnion. - sizeVal returns (uint32, error) and rejects values exceeding math.MaxUint32 instead of silently truncating. Callers in BuildViewType wrap with field-context messages. Addresses reviewer feedback on PR stellar#5937. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes #5926
Today, reading any field from a
LedgerCloseMetarequires decoding theentire message into Go structs. A typical pubnet ledger is ~1.5MB of XDR;
decoding it allocates ~8.5MB across ~107,000 objects, even if you only
need one transaction hash.
This PR adds a zero-copy view API. Every XDR type has a corresponding
Viewtype that is a typed[]bytealias. Accessors navigate intosub-slices without allocating, unmarshaling, or copying.
Benchmarks
Across 1,000 sampled pubnet ledgers (avg ~1.5 MB):
Full decode allocates ~8.5MB / ~107K objects per ledger. Views: zero
heap allocation for navigation.
What's in this PR
xdrgen/— Go code generator that reads a JSON IR emitted by the rustgenerator-definitions-jsonand producesxdr/xdr_views_generated.go.xdr/xdr_views_generated.go— generated view types and runtime support (primitive views,Try/TryVoid, error types).xdrgen/views_api.md— API design document.Makefile—xdr/xdr_views_generated.gotarget that regenerates views fromxdr/*.xvia a two-stage docker pipeline (rust → JSON IR → Go).xdr/xdr_view_test.go— hand-builtLedgerCloseMeta→ view → navigate round-tripxdr/view_support_test.go— runtime-helper unit testsxdr/view_randxdr_test.go— property tests over randomly-generated ledgers (round-trip and accessor correctness)xdr/view_fuzz_test.go—FuzzLedgerCloseMetaViewwith seed corpus; asserts no panic on validation or navigationxdr/*_bench_test.go— view vs full-decode benchmarks on a committed ledger fixture