Skip to content

Conversation

@msooseth
Copy link
Collaborator

@msooseth msooseth commented Aug 11, 2025

Description

We now validate Cex-es produced by hevm test. I moved the tracing interpreter to Stepper from Tracing, so it can be reused. The tracing interpreter actually also gives me back a vm which is useful because it allows me to read out the vm state, which for Dapps is actually necessary. Dapps read out the failed bit from the cheat contract.

Checking 1 function(s) in contract src/contract-keccak-correct.sol:MyContract
[RUNNING] prove_keccak_correct(address,uint256)
   [FAIL] prove_keccak_correct
   Counterexample:   [reproducible]
     calldata: prove_keccak_correct(0x0000000000000faaAaaFfaFAfafAFAaaAA472134,115792089237316195423570985008687907853269984665640564039457584007913129639935)
     result:   Revert: 0x4e487b710000000000000000000000000000000000000000000000000000000000000001

Checking 1 function(s) in contract src/contract-keccak-wrong.sol:MyContract
[RUNNING] prove_keccak_wrong(uint256)
   [FAIL] prove_keccak_wrong
   Counterexample:   [not reproducible]
     calldata: prove_keccak_wrong(0)
     result:   Revert: 0x4e487b710000000000000000000000000000000000000000000000000000000000000001

Note that this does NOT validate symbolic and equivalence Cex-es. They may not be too difficult to validate, given a similar logic, though. I decided not to do it, though, so as to keep this PR small.

Tests with proveFail_ also validate now:

Checking 1 function(s) in contract src/contract-shouldfail.sol:MyContract
[RUNNING] proveFail_single_fail(uint256)
   [FAIL] proveFail_single_fail
   Counterexample:   [validated]
     calldata: proveFail_single_fail(100)
     result:   Successful execution

Checklist

  • tested locally
  • added automated tests
  • updated the docs
  • updated the changelog

@msooseth msooseth force-pushed the reproduce-cexes branch 3 times, most recently from c80e67a to 0e9c2d2 Compare August 13, 2025 09:31
@msooseth msooseth changed the title [DRAFT] initial concrete check -- currently wrong [DRAFT] initial concrete check Aug 13, 2025
@msooseth msooseth force-pushed the reproduce-cexes branch 3 times, most recently from a98ac5e to 2096a65 Compare August 13, 2025 12:50
@msooseth msooseth changed the title [DRAFT] initial concrete check [DRAFT] Check that CEX-es can be reproduced in test mode Aug 13, 2025
@msooseth msooseth force-pushed the reproduce-cexes branch 2 times, most recently from 44ecf9e to 42c536b Compare August 18, 2025 09:56
@msooseth msooseth marked this pull request as ready for review August 18, 2025 10:12
@msooseth msooseth force-pushed the reproduce-cexes branch 2 times, most recently from 9a42df2 to fb27c70 Compare August 18, 2025 10:27
Thanks to @blishko for the idea :)

Update naming
@msooseth msooseth changed the title [DRAFT] Check that CEX-es can be reproduced in test mode Check that CEX-es can be reproduced in test mode Aug 18, 2025

calldataFromCex :: App m => SMTCex -> Expr Buf -> Text -> [AbiType] -> m (Err ByteString)
calldataFromCex cex buf funName types = do
let fullSig = (funName <> "(" <> T.intercalate "," (map abiTypeSolidity types) <> ")")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use the function name and types only to construct this signature, Wouldn't it be better to take signature directly?
I was looking at the users of this function, and basically the only user is symRun, which constructs the signature anyway.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not 100% sure I understand what you are saying, I'm sorry.

We actually don't really get a calldata from the SMT solver, because we try to minimize each part of the calldata. So we get chunks of the calldata (i.e. the values for the types), and that's why I am using this function to re-assemble it. Also, we may want to, in the future, rewrite the Cex in other ways, not just minimize it, so I thought it'd best to re-assemble it from the Cex, rather than try to pass it from the SMT solver directly. I find this somewhat cleaner, plus it helps to essentially fuzz our de- and re-construction of the calldata from the types + values.

Is this what you were referring to? Or is there some place where symRun has the calldata that I don't see? I think it's only reconstructing a pretty calldata here:

        txtFails <- symFailure opts testName (fst cd) types toPrintData

which calls:

      mkMsg conf (leaf, cex, repro) = intercalate "\n" $
        ["Counterexample:   " <> reproToText repro
        ,"  calldata: " <> let ?context = dappContext (traceContext leaf)
                           in prettyCalldata cex cd testName types
        ,"  result:   " <> showRes leaf
        ] <> verbText conf leaf

But I think that doesn't actually give us a raw calldata? Or am I missing something? Sorry, I am a bit confused.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comment related to this code:

It will be great to have two functions:

  • A function to transform a CEX into a high-level solidity call (e.g. f(1,2,3)).
  • A function to transform a high-level solidity call into calldata to run the transaction and verify the result.

So the user will be able to see the values in the call, and then debug an invalid CEX if necessary. Echidna already have the code for these two can that can be easily adapted or even re-used here if necessary.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

The A function to transform a CEX into a high-level solidity call (e.g. f(1,2,3)). is already available, it's called prettyCallData, it's called in symFailure like this: prettyCalldata cex buf funName types. It returns a text that does what you want, I think!

The A function to transform a high-level solidity call into calldata to run the transaction and verify the result. that's what I wrote, I think, i.e. calldataFromCex cex buf funName types.

At least as far as I can understand, that's what they do. However, it may be that I am misunderstanding what you mean, or that you need something more specific/etc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a second look and instead of taking Text and [AbyType], I think this method should be taking Sig.
And we should have a method to convert this to the text representation of a signature. Not sure if Show does that for Sig or not.
We should not have to repeat code like (funName <> "(" <> T.intercalate "," (map abiTypeSolidity types) <> ")")

One take 4 is enough. Thanks @blishko

Don't list calldataFromCex twice. Thanks @blishko

Fixing extraneous directive. Thanks to @blishko

Cleaner code

calldataFromCex :: App m => SMTCex -> Expr Buf -> Text -> [AbiType] -> m (Err ByteString)
calldataFromCex cex buf funName types = do
let fullSig = (funName <> "(" <> T.intercalate "," (map abiTypeSolidity types) <> ")")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a second look and instead of taking Text and [AbyType], I think this method should be taking Sig.
And we should have a method to convert this to the text representation of a signature. Not sure if Show does that for Sig or not.
We should not have to repeat code like (funName <> "(" <> T.intercalate "," (map abiTypeSolidity types) <> ")")

@msooseth msooseth mentioned this pull request Sep 8, 2025
4 tasks
@msooseth
Copy link
Collaborator Author

msooseth commented Sep 8, 2025

@blishko Thanks for the review! I fixed everything, and I am now using Sig. I have also created a new helper function, callSig in ABI.hs that takes a Sig and returns a Text. This now makes thing cleaner. I hope this is nicer now :)

Copy link
Collaborator

@blishko blishko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one final question, then I think we can go ahead with this one!

Copy link
Collaborator Author

@msooseth msooseth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to change constant :)

@msooseth msooseth merged commit 1d6d03a into main Sep 9, 2025
7 checks passed
@msooseth msooseth deleted the reproduce-cexes branch September 9, 2025 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants