Skip to content

feat(test-fill,test-specs): add t8n call caching to test specs#2084

Merged
marioevz merged 30 commits intoethereum:forks/amsterdamfrom
danceratopz:t8n-call-cache-specs
Feb 20, 2026
Merged

feat(test-fill,test-specs): add t8n call caching to test specs#2084
marioevz merged 30 commits intoethereum:forks/amsterdamfrom
danceratopz:t8n-call-cache-specs

Conversation

@danceratopz
Copy link
Member

@danceratopz danceratopz commented Jan 27, 2026

🗒️ Description

Add an in-memory output cache for transition tool results during fixture generation. When multiple fixture formats share the same t8n inputs (e.g., blockchain_test and blockchain_test_engine), the cache eliminates redundant t8n calls.

Changes

  • Add OutputCache class that stores t8n results per test, keyed by call counter.
  • Add strip_fixture_format_from_node() to derive consistent cache keys across fixture formats.
  • Integrate cache lookup/store in TransitionTool.evaluate(), used by BlockchainTest.generate_block_data().
  • Add xdist_group markers to keep related formats on the same worker.
  • Sort test items during collection: slow tests first (LPT scheduling), cacheable formats grouped together, deterministic order.
  • Display cache hit/miss statistics in the pytest session summary.
    =  T8n cache: 100% hit rate (29454/29454 tests expected), 32822 t8n calls saved =
    

Performance

The t8n-call-cache-specs branch reduces the py3 fixture fill step duration by 31% compared to other recent PRs (mean 18:58 vs 27:35, n=3 vs n=15).

Methodology

The analysis compares the timing of the "Run py3 tests" step from the py3 job from this PR with other PRs.

  • Metric: Duration of the "Run py3 tests" step within the py3 CI job (GitHub Actions test.yaml workflow).
  • Baseline: 15 successful runs from other PRs, filtered to runs created on or after 2026-02-12. This date cutoff ensures all baseline runs include the xdist worker count change from #2120 (merged Feb 12), which changed CI xdist parallelism - making the comparison fair.

Notes on the Achieved vs Expected Gain

The numbers below regarding % of t8n overhead shouldn't be taken too seriously, but it was fun to try and derive these 🙂

The py3 CI job generates 2 fixture formats for BlockchainTest specs and up to 3 for StateTest specs (blockchain_test_engine_x is not yet part of py3). The cache stores t8n output from blockchain_test and replays it for blockchain_test_engine, which shares the same transition_tool_cache_key.

Test item counts (--collect-only, py3 tox environment)

Format Count Cached
state_test 23,422 no
blockchain_test (native) ~11,900 no (generates cache)
blockchain_test_engine (native) ~11,900 yes
blockchain_test_from_state_test 23,414 no (generates cache)
blockchain_test_engine_from_state_test 18,765 yes
Total 89,401

Native blockchain counts are estimated as an equal split of the 23,800 native blockchain items (65,979 total blockchain minus 23,414 and 18,765 derived-from-state-test items).

Not all StateTest specs generate blockchain_test_engine: 4,649 state tests produce blockchain_test but not the engine variant (due to tests for forks that predate the engine format).

Expected vs observed savings

Value
Cached items ~30,665 (11,900 + 18,765)
Total items 89,401
t8n call reduction 30,665 / 89,401 = 34.3%
Observed fill time reduction 31.2%
Implied t8n share of fill time 31.2 / 34.3 = ~91%

The remaining ~9% of fill time is non-t8n overhead: fixture serialization, pre-allocation, and pytest machinery.

🔗 Related Issues or PRs

N/A.

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

Cute Animal Picture

image

@danceratopz danceratopz added C-feat Category: an improvement or new feature A-test-fill Area: execution_testing.cli.pytest_commands.plugins.filler A-test-specs Area: execution_testing.specs labels Jan 27, 2026
@marioevz marioevz self-requested a review January 27, 2026 14:15
@codecov
Copy link

codecov bot commented Jan 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.11%. Comparing base (4e9ec28) to head (910c76d).

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #2084   +/-   ##
================================================
  Coverage            86.11%   86.11%           
================================================
  Files                  599      599           
  Lines                39472    39472           
  Branches              3780     3780           
================================================
  Hits                 33992    33992           
  Misses                4852     4852           
  Partials               628      628           
Flag Coverage Δ
unittests 86.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running the latest version I see that we are adding @t8n-cache-<md5-hash> to the names of all tests for some reason. This seems incorrect to me, so I'd like to give this a proper re-review.

@danceratopz
Copy link
Member Author

Running the latest version I see that we are adding @t8n-cache-<md5-hash> to the names of all tests for some reason. This seems incorrect to me, so I'd like to give this a proper re-review.

This is to ensure that all parametrized test formats (state_test, blockchain_test, blockchain_test_engine) for a single test case get distributed to the same xdist worker to ensure they use the same per-worker cache; the cache is not global across workers.

@danceratopz danceratopz force-pushed the t8n-call-cache-specs branch 2 times, most recently from a294da7 to 0b598ec Compare February 2, 2026 13:52
@danceratopz danceratopz marked this pull request as draft February 2, 2026 13:56
@SamWilsn
Copy link
Contributor

SamWilsn commented Feb 3, 2026

Just a general comment (haven't looked at the code yet), but are the caches per-thread/per-process or global? If they aren't global, you might need to load group them to get the biggest benefit.

@danceratopz danceratopz force-pushed the t8n-call-cache-specs branch 2 times, most recently from f57d1d5 to bebe55a Compare February 16, 2026 14:22
@danceratopz
Copy link
Member Author

Just a general comment (haven't looked at the code yet), but are the caches per-thread/per-process or global? If they aren't global, you might need to load group them to get the biggest benefit.

The caches are per xdist process, not global. So yes, to make this work, fill marks tests with xdist_group markers and uses --dist=loadgroup to ensure that these group share the cache by running them on the same xdist worker.

danceratopz and others added 11 commits February 17, 2026 09:32
- Enable pytest-xdist loadgroup distribution mode by default.
- Required for xdist_group markers to control worker assignment.
- Add strip_fixture_format_from_nodeid() to extract base nodeid.
- Add get_all_fixture_format_names() for format name lookup.
- Used to ensure related fixture formats share cache keys.
- Add T8nOutputCache LRU cache class for storing t8n outputs.
- Add t8n_output_cache field to FillingSession.
- Add xdist_group markers during collection for --dist=loadgroup.
- Use t8n-cache-{hash} prefix to distinguish from user-defined groups.
- Strip cache-specific @t8n-cache-* suffix from nodeids in TestInfo.
- Add cache key helpers to BaseTest (_get_base_nodeid, _get_t8n_cache_key).
- Add _get_filling_session() to access cache from test instances.
- Cache t8n outputs in _generate_block_data() for reuse across formats.
- Skip caching for engine_x and engine_sync variants (different execution).
- Test T8nOutputCache LRU behavior, eviction, and hit/miss tracking.
- Test strip_fixture_format_from_nodeid for various nodeid patterns.
- Test get_all_fixture_format_names ordering and contents.
- Test cache key consistency across fixture format variants.
- Test _strip_xdist_group_suffix preserves non-cache group markers.
Add tests to verify that test items are sorted during collection
to ensure deterministic cache hits. The tests demonstrate:

- Sorting groups related fixture formats by base nodeid.
- Without xdist, items are correctly sorted.
- With xdist, items are NOT sorted (BUG causing high variance).
- Expected vs actual behavior comparison.

The xfail test `test_xdist_sorting_required_for_cache_hits` asserts
the correct behavior (sorting with xdist) and fails until the fix
is applied.
marioevz and others added 17 commits February 17, 2026 09:32
- Add helper methods to TransitionToolCacheStats for serialization.
- Initialize aggregated stats on xdist controller in pytest_configure.
- Send worker stats via workeroutput in fixture teardown.
- Add pytest_testnodedown hook to aggregate stats from workers.
- Update pytest_terminal_summary to display aggregated stats.
- Clear `_cache` in `remove_cache()` to prevent stale data leakage.
- Tests without `transition_tool_cache_key` (e.g., state_test) could
  previously retrieve cached results from prior tests via matching
  `call_counter` subkeys.
Sort test items by (is_slow, base_nodeid, nodeid) to optimize execution:
- Slow tests first (LPT scheduling for xdist load balance).
- Related fixture formats grouped together (cache locality).
- Deterministic order within groups.

If ANY fixture format variant of a test is marked slow, ALL variants
are treated as slow to keep them grouped together for cache hits.

Reuses the base_nodeid cache for xdist marker generation to avoid
redundant strip_fixture_format_from_node calls.
BlockchainEngineXFixture and BlockchainEngineSyncFixture had
can_use_cache=False which was dead code (never checked anywhere).
Replace with transition_tool_cache_key="" which is the actual mechanism
that controls caching — empty string means no caching.
For StateTest specs with --generate-all-formats, the _from_state_test
label suffixes cause alphabetical sort to interleave cacheable and
non-cacheable formats: blockchain_test_engine_from_state_test (cacheable)
→ blockchain_test_engine_x_from_state_test (non-cacheable, clears cache)
→ blockchain_test_from_state_test (cacheable, but cache is gone).

Add has_cache_key to the sort key so cacheable formats cluster together
within each base nodeid group, ensuring the second cacheable format hits
the warm cache before any non-cacheable format clears it.
node_id_for_entropy strips fixture format and fork names from the node
ID before hashing it for deterministic address generation. However, it
did not strip the xdist @group_name suffix (e.g., @t8n-cache-abc12345),
causing different addresses when running with vs without xdist workers.

Strip the suffix so addresses are deterministic regardless of whether
xdist is active.
Replace the raw hit/miss counts with an efficiency metric where 100%
means all tests that could have hit the cache did hit it. Track unique
cache keys to compute expected hits (total cacheable - unique keys).

Also filter subkey stats to only count cacheable tests, eliminating
phantom misses from non-cacheable tests that still interact with the
OutputCache after remove_cache().

Before: T8n cache: key_hits=6, key_misses=6 (50.0%), subkey_hits=6, subkey_misses=18 (25.0%)
After:  T8n cache: 100% hit rate (6/6 expected), 6 t8n calls saved
Pydantic's ModelMetaclass caches __init__ wrappers for dynamically
created classes. When pytester runs multiple fill sessions in-process,
cached wrappers from an earlier session re-invoke __init__ re-entrantly
in later sessions, causing generate() to run twice per test and
doubling the opcode count.

- Switch fill tests to runpytest_subprocess() for process isolation.
- Normal `fill` runs are unaffected (each invocation is a fresh process).
The rebase introduced a second pytest_testnodedown definition for cache
stats aggregation, shadowing the existing timing logs hook. Extract the
cache stats logic into _aggregate_cache_stats() and call it from the
single hook.
Remove dead code that was never called: _get_base_nodeid(),
_get_t8n_cache_key(), _get_filling_session() from BaseTest and
remove_opcode_count() from TransitionTool. Also remove the
now-unused strip_fixture_format_from_node import, TYPE_CHECKING
import, and guarded FillingSession import block from base.py.
Add a clear() method to OutputCache to encapsulate clearing the
internal _cache dict and resetting the key, instead of having
TransitionTool.remove_cache() reach into private members directly.

Also fix the set_cache docstring ("LRU behavior" → "single-key
eviction"), fix the cached_result truthiness check to use
`is not None` for defensive correctness, and fix the node()
docstring to say "pytest node" instead of "node ID".
The @t8n-cache-* xdist group suffix was leaking into the test_id and
group_salt stored in pre-alloc groups, causing fixture output to differ
from runs without xdist. Use _strip_xdist_group_suffix() (already used
by node_to_test_info) on both the group_salt fallback and the test_id
passed to add_test_pre.
Test set_key eviction, get/set round-trip, hit/miss counter
accuracy, clear() behavior, and state across key changes.
Uses sentinel objects as lightweight stand-ins for
TransitionToolOutput.
@danceratopz danceratopz marked this pull request as ready for review February 17, 2026 14:00
@danceratopz
Copy link
Member Author

danceratopz commented Feb 17, 2026

I ran hasher compare on the output of the py3 env fixtures locally with (~30 mins) and without (45 mins) this PR and got an exact match. I also tested static tests with --generate-all-formats for EngineX.

@danceratopz
Copy link
Member Author

There's a further possible optimization for the state_test format with Paris and Shanghai forks. I didn't want to complicate this PR any more so made a follow-up issue for this to evaluate whether it's worth the additional (fork-dependent) complexity:

@marioevz marioevz self-assigned this Feb 18, 2026
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks for implementing this!

@marioevz
Copy link
Member

I'm fixing the merge conflict in a bit and then merging 👍

@marioevz marioevz merged commit b3d9be1 into ethereum:forks/amsterdam Feb 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-fill Area: execution_testing.cli.pytest_commands.plugins.filler A-test-specs Area: execution_testing.specs C-feat Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants