Skip to content

[WIP] Mock Engine API req/resp in tests#2637

Closed
hwwhww wants to merge 4 commits intodevfrom
engine-api-tests
Closed

[WIP] Mock Engine API req/resp in tests#2637
hwwhww wants to merge 4 commits intodevfrom
engine-api-tests

Conversation

@hwwhww
Copy link
Contributor

@hwwhww hwwhww commented Sep 30, 2021 •
edited
Loading

Based on @zilm13's new merge fork-choice tests in #2598/#2630, I tried to generate the expected Engine API request & mock the response.

It would add engine_api event in steps.yaml.
A sample steps.yaml that provides engine_preparePayload call:

EDIT: added engine_executePayload and reworked the format. Still incomplete.

- {tick: 0}
- block_production:
    prepare_execution_payload:
      engine_api:
        request:
          method: engine_preparePayload
          params: {parentHash: '0xb1bb023570c3ec3156ec647755d0320a634c6fb714163da9a06c223718eaa779',
            timestamp: 0, random: '0xdadadadadadadadadadadadadadadadadadadadadadadadadadadadadadadada',
            feeRecipient: '0x1212121212121212121212121212121212121212'}
        response:
          error: false
          result: {payloadId: 1}
- {tick: 6}
- block:
    block: block_0x27700996d90a742359d291acf1602dfe4270746dae1e5e548e609878aa9bb8bb
    process_execution_payload:
      engine_api:
        request:
          method: engine_executePayload
          params:
            parentHash: '0x38c294cbcedc2df7e1f0a4a731b8beba598454f9a2b1fe9e277f000fc65df983'
            coinbase: '0x0000000000000000000000000000000000000000'
            stateRoot: '0x0000000000000000000000000000000000000000000000000000000000000000'
            receiptRoot: '0x6e6f207265636569707473206865726500000000000000000000000000000000'
            logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
            random: '0xbe3f3033d3a3532aed109ea96eef8995d46d80701baaf8bdd2c8e876eac98cc2'
            blockNumber: 2
            gasLimit: 0
            gasUsed: 0
            timestamp: 6
            extraData: 0x
            baseFeePerGas: 0
            blockHash: '0x24c88dd3356939401a03bd8e2364eb6b892f78415c47124965fbc3ea33a81f31'
            transactions: []
        response:
          error: false
          result: {status: VALID}
    pow_block: pow_block_0xb1bb023570c3ec3156ec647755d0320a634c6fb714163da9a06c223718eaa779
- checks:
    time: 6
    head: {slot: 1, root: '0xb9f485520ccb7281b6f60f502a5030265b627725096ff9a81e5b7fc1e4ec874c'}
    justified_checkpoint: {epoch: 0, root: '0x8b3e1c2c1405dda39b7744d8ec4ba9fd1ece6caf8e3053211269a1b4605c927e'}
    finalized_checkpoint: {epoch: 0, root: '0x8b3e1c2c1405dda39b7744d8ec4ba9fd1ece6caf8e3053211269a1b4605c927e'}
    best_justified_checkpoint: {epoch: 0, root: '0x8b3e1c2c1405dda39b7744d8ec4ba9fd1ece6caf8e3053211269a1b4605c927e'}

My idea is that the consensus client teams can:

  1. Compare if the expected API requests match
  2. Use the given mocking response to continue the next step
  3. Test against the edge cases in CI

Question for client teams: Would it be helpful and worth developing?
Feedback would be appreciated. 🙏

Rezii8282 referenced this pull request Sep 30, 2021
@terencechain @hwwhww
Update specs/merge/beacon-chain.md
2ea262f 
Co-authored-by: Hsiao-Wei Wang <[email protected]>
@mkalinin
Copy link
Contributor

mkalinin commented Sep 30, 2021

I am a little bit hesitant on mocking execution engine this way because it ties consensus specs to the Engine API implementation of EE. We could have a similar effect by mocking the ExecutionEngine abstraction. Thoughts?

@hwwhww
Copy link
Contributor Author

hwwhww commented Sep 30, 2021

Thanks for the feedback @mkalinin

it ties consensus specs to the Engine API implementation of EE.

It does tie to the interface of Engine APIs, but we can set arbitrary responses from the consensus side!

We could have a similar effect by mocking the ExecutionEngine abstraction

Do you mean like disposable TestEngine in test_process_execution_payload tests? Or a more reusable and general ExecutionEngine cross consensus pyspec tests?

This PR also mock disposable ExecutionEngine, the difference is it makes mocking work with higher level state_transition function instead of process_execution_payload function by doing spec.EXECUTION_ENGINE = my_mock_engine.

I chose fork choice test format because it would cover most Engine APIs. Since the INVALID cases would be covered by granular process_execution_payload operation tests, I think for most engine_executePayload calls in the new tests, we can just return VALID.

@hwwhww hwwhww added the testing CI, actions, tests, testing infra label Oct 2, 2021
@hwwhww hwwhww mentioned this pull request Oct 2, 2021
Closed
@hwwhww
Copy link
Contributor Author

hwwhww commented Oct 18, 2021

replaced with #2639

@hwwhww hwwhww closed this Oct 18, 2021
@jtraglia jtraglia deleted the engine-api-tests branch January 22, 2025 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

testing CI, actions, tests, testing infra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hwwhww @mkalinin