ingest/loadtest: add stellar-core apply-load tooling - #5940
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new ingest/loadtest helper for generating synthetic ingestion fixtures from stellar-core apply-load, so other repos can reuse the same load-test data generation flow from the SDK.
Changes:
- Introduces an
ApplyLoadworkflow that runsstellar-core new-db,new-hist, andapply-load. - Parses apply-load config/output to locate generated metadata and the pre-benchmark checkpoint boundary.
- Adds logic to export benchmark ledgers and checkpoint-derived fixture entries as zstd-compressed XDR files.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@cjonas9 most of these issues are pre-existing from the horizon code but I think they're worth fixing now that we plan to include them into the go sdk |
completely agreed -- I'd noticed some of these as I'd gone along, but reckoned there were already discussions had about them. even so, awesome review! a ton of super valuable finds I missed in here |
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
CHANGELOG.mdwithin the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
This PR adds tooling for synthetic ledger generation using stellar-core's
apply-loadcommand for ingestion load testing. Given anapply-loadconfig, this enables one to generate an output that can be used for follow-up RPC or Horizon ingestion load tests through the pattern ofNewApplyLoad(...)->ApplyLoad.RunApplyLoadAndWrite()(+ApplyLoad.Cleanup()).Originally, this lived in
stellar-horizonas an integration test. The core codeflow and design is the same here as it was there, except functions here no longer take a*testing.Tand are oriented around a*ApplyLoadstruct. It's migrated to the SDK to ensure that it may be common to ingestion load testing tools in bothstellar-horizonandstellar-rpc.Why
See stellar-rpc/711. This PR focuses on making the utility for ingestion load-tests common so that it can be used in any repo that requires it.
Known limitations
N/A.