Skip to content

Conversation

@glozow
Copy link
Member

@glozow glozow commented Feb 2, 2021

This is the first 4 commits of #20833, and does refactoring only. It should be relatively simple to review, and offers a few nice things:

  • It makes accessing values that don't make sense (e.g. fee) when the tx is invalid an error.
  • Returning MempoolAcceptResult from ATMP makes the interface cleaner. The caller can get a const instead of passing in a mutable "out" param.
  • We don't have to be iterating through a bunch of lists for package validation, we can just return a std::vector<MempoolAcceptResult>.
  • We don't have to refactor all ATMP call sites again if/when we want to return more stuff from it.

@DrahtBot
Copy link
Contributor

DrahtBot commented Feb 2, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link

@ariard ariard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting this from #20833!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

abc6ff1

What do you think about making bypass_limits part of the new MempoolAcceptResult ?

Actually we don't have a mempool acceptance evaluation. The set of rules verified is already configurable by passing bypass_limits=true to ATMP. This flag will latch feerate and size checks (L729 and L1018 in src/validation.cpp). A consumer of this cleaner interface might be interested with the effective set of rules enforced. And consumer might not be ATMP caller who initially picked up the options.

It would be judicious if we introduce future configurable options in the future like bypass_timelocks.

Copy link
Member Author

@glozow glozow Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of changing m_accepted to an enum to encompass states beyond valid/invalid? Something like

enum class ResultType : uint8_t {
        UNSET, //!> Not fully validated, quit early for whatever reason.
        INVALID, //!> Invalid.
        VALID, //!> Valid.
        VALID_BYPASSED, //!> some kind of limits bypassed
}

This would leave room for bypassing timelocks as well. We probably don't need to include more details than "valid albeit some rules were bypassed," because the caller should already know what args they called ATMP with.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other comment but I would keep a binary state for validity. I don't think TxValidationResult/BlockValidationResult are great examples, it makes it harder to reason on once you multiply states.

Let's keep this suggestion in mind for now, it's not a must for this PR. We'll see if we need to introduce something like this if we have situation where we have one aware caller and multiple blind consumers.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

abc6ff1

I don't think that's a good idea to encumber code path like UpdateMempoolForReorg with std::optional. If this nullopt will throw an exception. And we do have the failure/unfinished constructor allowing m_accepted to be initialized to nullopt, even if AFAICT such constructor is never called with finished=badfor now ?

I know there is a discussion about std::optional usage here but could we restrain its usage to only m_base_fee/m_replaced_transaction ? They would be nullopt if m_accepted=false.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check now? I think it's better with enum

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need an enum and can't we rely only on a boolean ? Maybe you can point me to a branch how you're using UNFINISHED because we don't use it with this PR ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't published the branch yet but the idea is to return a std::vector<MempoolAcceptResult> from ProcessNewPackage, quit early when a tx fails, and set the not-fully-validated txns in the package to UNFINISHED. See comment?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think UNFINISHED/not_fully_validated to mark package partial failure doesn't bring further value compared to unvalid. Do you plan to consume this UNFINISHED in a special way ? Otherwise we can just extend TxValidationResult with a PACKAGE_PARENT_FAILED instead of yet-another-state variable.

Copy link
Member Author

@glozow glozow Feb 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the future with Package Relay, we may want to punish nodes differently for invalidity in packages vs invalidity in transactions, and maybe cache failed transactions differently. Obviously this isn't set in stone, but I think it's better to not put package-specific validation info in TxValidationState - what do you think?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a straightforward reason to punish faulty pacakge-relay peers from regular tx-relay ones, at least at the mempool level. If we don't have a motivation for UNFINISHED, let's remove it for now and defer its introduction when we actually hit the case ? Would be easier to argue at that point.

src/validation.h Outdated
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

abc6ff1

I think we should commit to a clearer terminology. Validity is always wholesome, a transaction or package is either valid or not. But validity is function of a mempool acceptance evaluation and this is configurable (bypass_limit), stateful (e.g mempool min feerate), depends if the transaction is part of a package, etc.

If we follow this line, maybe we should rename m_accepted to m_valid and have only binary state (true, false). The interface could be extended in the future to indicate it's part of a package, and we may have a TxValidationState for package children to inherit the invalidity (PACKAGE_INVALID_PARENT).

Cleans up reundant code and reduces the diff of the next commit.
@glozow glozow force-pushed the 2021-02-refactor-validation branch from 847ad68 to 1c4ec9a Compare February 2, 2021 15:28
Copy link
Member

@darosior darosior left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK -will review soon

@glozow
Copy link
Member Author

glozow commented Feb 2, 2021

Teeny rebase for the compiler warnings and changed from std::optional<bool> m_accepted to an enum ResultType m_result_type so there's no risk of throwing for bad optional access.

@glozow glozow force-pushed the 2021-02-refactor-validation branch from 1c4ec9a to bc992b7 Compare February 2, 2021 16:32
@maflcko maflcko changed the title pure refactor: return MempoolAcceptResult from ATMP refactor: return MempoolAcceptResult from ATMP Feb 4, 2021
Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK bc992b7a394629137929647998149f18fea5ab29

A few non-essential style suggestions inline. You're going to be touching this code again in the next PRs, so feel free to defer these suggestions.

@glozow glozow force-pushed the 2021-02-refactor-validation branch from bc992b7 to a9ff9c1 Compare February 8, 2021 15:18
@glozow
Copy link
Member Author

glozow commented Feb 8, 2021

Addressed @jnewbery comments and @ariard #21062 (comment) and removed the ResultType::UNFINISHED for now since it's unused. Leaving it as an enum because I think we agree that there's value in having more than 2 states, and the struct is the same size

@jnewbery
Copy link
Contributor

jnewbery commented Feb 8, 2021

Code review ACK a9ff9c1ca0530e448341e9d24ecd5f8bc6f2ee42

Cirrus failure looks spurious, but I don't know how to restart it.

maflcko
maflcko previously requested changes Feb 9, 2021
Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The third commit doesn't compile

e2713614089f77a01c98bb6783f2b1a22aa6ab13 🎪

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

e2713614089f77a01c98bb6783f2b1a22aa6ab13 🎪
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUidXwwAiGoRXoyy71GeKNoMewKKtoqWSYvjSb9iQMeQIIsS9u0CZA6zPQxjkE9J
6uynw8usu+HcHAKQ852r4hmrmgJZiuZalgGPVAfJbODgp89rJzhbVElwiwW77xHh
EYUET22yB00viK1zkIzIMtcPsNhcyTfxTYZIxI0DP/bqFBkpwAmTP3axwqmaoLAN
iHtInc+D4ajOTGWwzimQShQBvjPjeFvtf/ErqaS7Nj1IqGCXLu9nadt14Gf0OcgS
MTceMpLpPZVGGHJedlWtXrLFICSGFTNWsZQ7GcDQEPX8axNligCDtIZZ/PHhi5j9
Z52tD04emkuxI+mma80ejcnkKh4oI88dG9/GL4C8Qhua7MCJHzyASY7kmHZ7hqJB
Og8AYIXJVFHXw0wsgiIdhDQ3A4bwX31T8xc5lhpvSaZUJgnbih4QdA6oohHr4vRf
Z+Sx0DgksuscH8LIVgcJwcCLukX2Wb5IRvBiR3cSclQM20WEL66zt8ce15nMVECa
zO++FfcZ
=jSjm
-----END PGP SIGNATURE-----

Timestamp of file with hash 1a5436e3da8426ac64d2e23be49dd4c3bbb869d0d35ecba05ef4c37aaa88e2a9 -

This creates a cleaner interface with ATMP, allows us to make results const,
and makes accessing values that don't make sense (e.g. fee when tx is
invalid) an error.
ATMPArgs should contain const arguments for validation.
The Workspace should contain state that may change
throughout validation.
@glozow glozow force-pushed the 2021-02-refactor-validation branch from a9ff9c1 to 53e716e Compare February 9, 2021 15:04
@glozow
Copy link
Member Author

glozow commented Feb 9, 2021

The third commit doesn't compile

Should be fixed now 🤦 thanks @MarcoFalke! Fixed the constructors a bit as well to take out the unnecessary TxValidationState in the success case and Assume not valid in the failure case.

@maflcko
Copy link
Member

maflcko commented Feb 10, 2021

Changes since previous review:

  • No longer pass redundant default-constructed state when valid
  • Remove uint8_t from enum class
  • Add Assume(!state.IsValid());
  • Make it compile

ACK 53e716e 💿

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 53e716ea119658c28935fee24eb50090907c500e 💿
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUh02Qv/VKTbpbQkoToYPmzKcSX5aJ+JAaQThI2/aGnpdgFnl/GdIVDyMcWHyygP
iduWhmF6BeEerKuctDAbfLB+EjeQdIr0xMa6lg5nkSsZTi0W+wn+5okOnngsjAzI
JazlKJJGQj/WjUJof04cTtIfcT9ecLS9yN8TJ/W8EWnm+RpSCOMBgUvRfBgdCBXX
GKs2xoxN6BT1ML5+y7qQQB2VyZE/DAEykbR6NhgaAIkrA5UZM6tP4t5/YTNtjoZq
hBgKkesPqym/WjB/6LTEwannfiW/ACvpM3Czg5pPS1svMyMxT4LQh4rMDiF4U21x
apPhPIMwvGce1pdX6UwMuDhRdA0VE6y23rxjSjhZaA0ZEBOVEW2sYBCa9bVcROrY
60WzJ2j6dYY2xIHJsYo5vG1LxjSrz42My50LLIKTBsihTg1LNst52KXZfFjozng6
2gewYrzP0lJe/fyUf9BxNK1w9+mbd5kCJDR1VHxsu9Yc5B+4gVKmva4uFodLd+Yo
Hyw2/p/q
=G4nn
-----END PGP SIGNATURE-----

Timestamp of file with hash 5da03708a5e913024d3ccd093bac9c7a16f54f296f91a24b4c60d85316f124f4 -

Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some non-critical style nits

VALID, //!> Fully validated, valid.
INVALID, //!> Invalid.
};
ResultType m_result_type;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would be nice if this was const, so that compilation fails instead of getting a valgrind error on runtime if this is unset.

INVALID, //!> Invalid.
};
ResultType m_result_type;
TxValidationState m_state;
Copy link
Member

@maflcko maflcko Feb 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for all other members?


/** Constructor for success case */
explicit MempoolAcceptResult(std::list<CTransactionRef>&& replaced_txns, CAmount fees)
: m_result_type(ResultType::VALID), m_state(TxValidationState{}),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tried, but I think this can be written shorter

Suggested change
: m_result_type(ResultType::VALID), m_state(TxValidationState{}),
: m_result_type(ResultType::VALID), m_state{},

Copy link

@ariard ariard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 53e716e, I did tweak a bit the touched paths to see if we had good test coverage. Didn't find holes.

Nit: if you have to re-touch the code, you forget this one #21062 (comment)

@jnewbery
Copy link
Contributor

Code review ACK 53e716e

@glozow
Copy link
Member Author

glozow commented Feb 10, 2021

Opened followup #21146 to address style comments from @MarcoFalke and @ariard (oopsie for accidentally pushing here, please ignore that).

@maflcko maflcko merged commit 8e1913a into bitcoin:master Feb 11, 2021
@glozow glozow deleted the 2021-02-refactor-validation branch February 11, 2021 14:02
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 11, 2021
maflcko pushed a commit to bitcoin-core/gui that referenced this pull request Apr 28, 2021
…rs const

363df75 doc/style followups in MempoolAcceptResult (glozow)

Pull request description:

  Follow up to #21062. Was going to be a part of #20833 but I'm trying to break it down as much as possible.

  - Make members const (bitcoin/bitcoin#21062 (comment))
  - List fee units (bitcoin/bitcoin#21062 (comment))
  - Use default value for `TxValidationState` in the success case (bitcoin/bitcoin#21062 (comment)).

ACKs for top commit:
  jnewbery:
    ACK 363df75
  practicalswift:
    cr ACK 363df75: patch looks correct and `const` is better than non-`const` (where possible :))
  ariard:
    Code Review ACK 363df75

Tree-SHA512: 0ff1a0279e08e03204e48d0f4c92428d7f39c32f52c1d20fe6a0283d605839898297344be82ca69640ba9f878ca4ebd5da2d717e26d719a183b211d709334082
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 28, 2021
363df75 doc/style followups in MempoolAcceptResult (glozow)

Pull request description:

  Follow up to bitcoin#21062. Was going to be a part of bitcoin#20833 but I'm trying to break it down as much as possible.

  - Make members const (bitcoin#21062 (comment))
  - List fee units (bitcoin#21062 (comment))
  - Use default value for `TxValidationState` in the success case (bitcoin#21062 (comment)).

ACKs for top commit:
  jnewbery:
    ACK 363df75
  practicalswift:
    cr ACK 363df75: patch looks correct and `const` is better than non-`const` (where possible :))
  ariard:
    Code Review ACK 363df75

Tree-SHA512: 0ff1a0279e08e03204e48d0f4c92428d7f39c32f52c1d20fe6a0283d605839898297344be82ca69640ba9f878ca4ebd5da2d717e26d719a183b211d709334082
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 22, 2022
Summary:
Cleans up reundant code and reduces the diff of the next commit.

This is a backport of [[bitcoin/bitcoin#21062 | core#21062]] [1/4]
bitcoin/bitcoin@9db10a5

Test Plan: `ninja all check-all`

Reviewers: #bitcoin_abc, sdulfari

Reviewed By: #bitcoin_abc, sdulfari

Differential Revision: https://reviews.bitcoinabc.org/D11778
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 22, 2022
Summary:
This creates a cleaner interface with ATMP, allows us to make results const,
and makes accessing values that don't make sense (e.g. fee when tx is
invalid) an error.

This is a backport of [[bitcoin/bitcoin#21062 | core#21062]] [2/4]
bitcoin/bitcoin@f82baf0

Note: changes related to RBF are not applicable

Depends on D11778

Test Plan: `ninja all check-all bitcoin-bench`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D11779
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 22, 2022
Summary:
ATMPArgs should contain const arguments for validation.
The Workspace should contain state that may change
throughout validation.

This is a backport of [[bitcoin/bitcoin#21062 | core#21062]] [3/4]
bitcoin/bitcoin@174cb53

Note: changes related to RBF are not applicable
Depends on D11779

Test Plan: `ninja all check-all`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D11780
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jul 22, 2022
Summary:
This concludes backport of [[bitcoin/bitcoin#21062 | core#21062]] [4/4]
bitcoin/bitcoin@53e716e

Test Plan: `ninja all check-all`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D11781
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

Development

Successfully merging this pull request may close these issues.

6 participants