-
Notifications
You must be signed in to change notification settings - Fork 38.7k
tests: Use test utils each time that it is possible #25732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests: Use test utils each time that it is possible #25732
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
test/functional/interface_zmq.py:367: error: unexpected indent [syntax]
Found 1 error in 1 file (errors prevented further checking)
/tmp/cirrus-ci-build/test/functional/interface_zmq.py:367: unexpected indent at "assert_raises_rpc_error(-8, "Verbose results cannot contain mempool sequence values.", self.nodes[0].getrawmempool, True, True)"
Python dead code detection found some issues |
78ef523 to
97ff626
Compare
|
Thanks! Cherry-pick problem with an older version of the |
97ff626 to
ad9dcab
Compare
stickies-v
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approach ACK ad9dcabd5d9cef3f16accbc9dc1572cd82a99081
test/functional/feature_taproot.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
== and is are not entirely equivalent in Python, with both differences in evaluation speed and outcome. Even though I think it's fine to use the equality operator here, I think the proper way to do it would be to add assert_is and/or assert_is_none functions (e.g. unittest also has these: https://docs.python.org/3/library/unittest.html#assert-methods)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I'm getting confused here :)
We need to converge on some code style here otherwise we cause only confusion #23127 (comment)
cc @MarcoFalke
* feature_taproot: Introduce another assert called assert_true. * interface_zmq.py: Remove assert in favor of test utils functions * feature_taproot: Fixed imports * mining_getblocktemplate_longpoll.py: Replace assert with assert_equal * p2p_addrfetch.py: Replace assert with assert_equal * p2p_blockfilters.py: Replace assert with test utils function * test framework: Adding possibility to specify messages in the assert functions * feature_taproot.py: Use the err_msg introduced in the assert framework. * feature_taproot.py: Use the err_msg introduced in the assert framework. * feature_taproot.py: Use the err_msg introduced in the assert framework.
|
Thanks @stickies-v, you remember me that we have a pending decision on the API that we are using right now #23127 (comment) maybe we can discuss it here! |
ad9dcab to
640edd9
Compare
stickies-v
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't want to bikeshed over this too much, it's not a big deal. But my view (and it has changed a bit after reading the previous issues/PRs) is that:
- we check values for being (not)
Nonequite often. Even though==comparison would work fine in probably all cases, it's not the Python way of doing things and given how often this is used I feel adding aassert_is_none()andassert_is_not_none()function is warranted. Since generally nothing except forNoneshould be compared throughis, I'd say we can leave outassert_is()andassert_is_none()(which I suggested earlier) until we actually need it (probably never). - even though boolean
assert_true()andassert_false()functions can have their merit, upon re-reading I don't see any in the current PR because we currently only ever pass it booleans. In all the cases whereassert_true()is used, a simpleassert <expression>, <err_msg>is functionally equivalent, shorter, and without the need to defineassert_true(). In all places whereassert_true()is used, the logs would just showFalse is not True, which does not help. I believe this is what @MarcoFalke was referring to, and I agree with him for this scenario (but not for theassert_equal()on which the comment thread was based). Having a boolean assertion function is useful as soon as you need to pass non-boolean types to it. If we do decide to implement anassert_true(expr, err_msg)function, it should be implemented to evaluatebool(expr) is not Trueinstead of justexpr is not True(as commented), in which case it offers benefit overassert_equal(bool(expr), True)because we can now log the value ofexprinstead ofbool(expr). But again, it looks like we currently don't have a need for this. - From my reading of previous issues, it seems like there is no reason to necessarily phase out all usage of plain
assertstatements here - the only reason to add bespokeassert_...()functions was to make it easier to include error messages. As such, I think it's fine to keep plainassertstatements where it's easier until we have a clear reason (please tell me if I've missed that) to do so.
|
|
||
|
|
||
| def assert_true(thing, err_msg=None): | ||
| if thing is not True: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These functions are generally (e.g. https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTrue) used to do a boolean comparison, which makes it functionally different than using assert_equal(expr, True)
It wouldn't make a functional difference in the places where it's currently used (where a simple assert would suffice), but otherwise, this is how I think it should be done:
| if thing is not True: | |
| if bool(thing) is not True: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with this suggestion, you will create an error a runtime and no static checker can find it. We can use the only good thing of python language to write something like that
def assert_true(thing: bool, err_msg=None):
if thing is not True:
In this way, the editor is able to catch the type mismatch! the cast of a type at runtime should be an antipattern in all modern language
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only way bool(thing) throws an error is if thing is of a custom type that explicitly overrides __bool__() to return something other than True or False, which is... really bad code. I don't think we need to handle that, and we can easily observe if it happens from the stacktrace.
Even though I'm a fan of type hints, I don't think they change anything here - afaik we don't use tooling like mypy that would enforce them? It also wouldn't do the boolean comparison which was kinda my point: you don't want assert_true() to just verify if something is equal to True/False (assert_equal can do that), it's helpful if you need to check if the boolean of an object would evaluate to True/False. For example:
result = some_func() # some_func returns 0 if it failed, an int between 1-5 if successful
assert_equal(result, True) # this would not work, because e.g. `3 == True` would fail
assert_equal(bool(result), True) # this would work, but you can't inspect `result` when the assertion fails
assert_true(result) # this would work if `assert_true` evaluates bool(result), because `bool(3) is True` would pass
if(result):
do_something_else()There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But to reiterate, I'm advocating against implementing assert_true() at this moment, so I'll leave it here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right! The thing is worst than the exception because happens something like that
➜ ~ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> bool(12)
True
>>> bool("Alibaba")
True
>>>
This can not hide bugs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think to make all happy here is to use the assert_equal everywhere it is possible and move on.
This is code that I wrote one year ago, so I do not remember why I insert assert_true, but if I was not drunk I think because somewhere there is some comparison with custom error messages that check a boolean condition.
Anyway, I think if you agree we can move all on assert_equal and move on.
you don't want assert_true() to just verify if something is equal to True/False
To sustain your thesis, I'm not able to see any real use case other than toy example of how this can be used
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is that worse? You just need to use the right tool for the job. If you need to compare equality, you use assert_equal. If you need to do a boolean comparison, you use assert_true. If you don't know your input type, you should probably validate/convert that first. If you don't need to do a boolean comparison (i.e. if you wouldn't want "Alibaba" to evaluate to True), then you just don't use the boolean assertion?
The point of these functions is to check how something would be evaluated to a boolean, as in the example I gave where the program flow is based on if(result). We don't seem to have that need in our test suite atm, so hence we shouldn't have assert_true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mh! I do not argue this on the PR because it is the wrong place but I disagree with this statement
How is that worse? You just need to use the right tool for the job.
Of course, bug is a bug, and it is there because there is a mistake in the code, so if a comparison between bool, I want to compare boolean value and not a string that always evaluates to true except for empty.
Is worse because if we were able to catch this bug easily we were still coding in C without struggling with rust compiler, if you do this case you should define some rules like: bool("False") -> False or True?
Imagine to have assert_true(json['foo']) and make an API change that move the type of a filed from string to int
you fall in case of
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> bool(0)
False
>>> bool("0")
True
>>> bool(0.0)
False
>>> bool("0.0")
True
It is not the right tool anyway at least without a good reason to allow this, and I do not see any till now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A better example can be
take the from our assert_true(hash_str is None) with assert_true(hash_str)
➜ ~ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> bool(None)
False
>>> bool("")
False
This can be a bug not catch by the CI, and if it is an API change you can break the client that use a strongly typed language
|
In this PR I have no strong opinion in whatever decision we take, and I just do not care too much about what type of assert we will use, as I see this will not bring too much value inside the code base. I'm fine with more function we add, more code to maintain we have, and if there is no strong agreement I will preferer to remove all the |
|
Yeah okay let's do the common denominator approach now and potentially add more functions later on, I don't want to push on the |
|
Dup from IRC: this PR has been quiet for 2 months. Is it ok if someone else completes the last recommendations? Secondly, the latest ask is essentially the opposite of the the ask in the issue this PR is meant to address. Can we get some confirmation on what the approach should be? cc: @theStack |
Feel free to cherry-pick my commits and continue in a new PR it is totally fine, and also it useful that someone else will continue an unfinished work |
@vincenzopalazzo does that mean you're no-longer working on / maintaining this PR? If so, would you like to close it? |
|
@fanquake Yes I think that there are people that can give a better love to this PR |
This is one of my attempts to make less painful the #23127 and try to bring this simple refactoring in.
Issue related #23119
PR 1/N I think there are a couple of equal size later