Skip to content

Conversation

@theStack
Copy link
Contributor

@theStack theStack commented Aug 6, 2020

This tiny PR substitutes Python loops in the form of for x in range(N): ... by for _ in range(N): ... where applicable. The idea is indicating to the reader that a block (or statement, in list comprehensions) is just repeated N times, and that the loop counter is not used in the body, hence using the throwaway variable. This is already done quite often in the current tests (see e.g. $ git grep "for _ in range("). Another alternative would be using itertools.repeat (according to Python core developer Raymond Hettinger it's even faster), but that doesn't seem to be widespread in use and I'm not sure about a readability increase.

The only drawback I see is that whenever one wants to debug loop iterations, one would need to introduce a loop variable again. Reviewing this is basically a no-brainer, since tests would fail immediately if a a substitution has taken place on a loop where the variable is used.

Instances to replace were found by $ git grep "for.*in range(" and manually checked.

substitutes "for x in range(N):" by "for _ in range(N):"
indicates to the reader that a block is just repeated N times, and
that the loop counter is not used in the body
@DrahtBot
Copy link
Contributor

DrahtBot commented Aug 6, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@practicalswift
Copy link
Contributor

practicalswift commented Aug 6, 2020

Concept ACK: the idiomatic/Pythonic _ communicates clearly to the reader that the variable in question is a throwaway variable

While you're at it consider using vulture too to find more unused variables :)

for F in $(git ls-files -- "*.py"); do vulture "$F" | grep "unused variable"; done

Note that vulture might give you false positives, so make sure to check all candidates manually (probably goes without saying! :)).

@troygiorshev
Copy link
Contributor

Concept ACK. I'm really hesitant to review this without us trying to prevent it in the future.

Playing with vulture for a couple minutes seemed to give far too many false positives.

Pylint, on the other hand, surely had many false negatives, but seemed to do a good job otherwise.

pylint test/functional/**/*.py --disable all --enable W0612

Possible extension that I don't want to forget: replace range(len(foo)) with enumerate(foo).

@practicalswift
Copy link
Contributor

practicalswift commented Aug 6, 2020

pylint test/functional/**/*.py --disable all --enable W0612

I think pylint $(git ls-files -- "*.py") --disable all --enable W0612 will catch more cases :)

The reason we disabled vulture in Travis was due to false positives. Is pylint's W0612 check free from false positives in our code base?

(False negatives are not a big deal, but false positives are :))

@troygiorshev
Copy link
Contributor

troygiorshev commented Aug 6, 2020

The reason we disabled vulture in Travis was due to false positives. Is pylint's W0612 check free from false positives in our code base?

Just manually checked on this PR. Surprisingly, yes! All instances can be replaced, by either _ or del foo in the case of an unused parameter (2.1.4 of https://google.github.io/styleguide/pyguide.html).

@laanwj
Copy link
Member

laanwj commented Aug 7, 2020

-0 on this, I agree this looks somewhat neater, but also, I'm not sure it's worth making this change and being pedantic about it in the linter.

@theStack
Copy link
Contributor Author

theStack commented Aug 7, 2020

Thanks for the conceptual reviews and linting tipps so far!

Concept ACK. I'm really hesitant to review this without us trying to prevent it in the future.

Yes, I was thinking the same. My hope is that with the pattern being more widespread in the code, awareness for this raises and hence it is on the long term easier catched both by PR creators and by reviewers. Not sure if that's enough -- adding it as rule to the linter seems to be too strong though, agree with laanwj here.

For anyone wanting to try out the linters: note that (at least on my system) pylint is for linting Python 2 code, hence we should ensure to use pylint3 here, which interestingly enough also detects a bit more. (Some systems may have a symlink from pylint to pylint3, but better check via pylint --version.)

@practicalswift
Copy link
Contributor

practicalswift commented Aug 7, 2020

ACK dac7a11 -- the updated code is easier to reason about since the throwaway nature of a variable is expressed explicitly (using the Pythonic _ idiom) instead of implicitly. Explicit is better than implicit was we all know by now :)

@laanwj
Copy link
Member

laanwj commented Aug 7, 2020

Explicit is better than implicit was we all know by now :)

Yes, but also some people contributing to the test framework have been complaining about the Python linter framework that it does 'silly' checks that don't really help catch errors, but just feel overly pedantic. I think this is one such one.

Edit: To be clear:
ACK on this change as it is now, NACK on adding a linter for this

Copy link
Member

@darosior darosior left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK dac7a11

@instagibbs
Copy link
Member

manual inspection ACK dac7a11

though I think this kind of thing is a very low risk of testing bugs.

@fanquake fanquake merged commit cb1ee15 into bitcoin:master Aug 11, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Aug 11, 2020
…sed loop counters

dac7a11 refactor: test: use _ variable for unused loop counters (Sebastian Falbesoner)

Pull request description:

  This tiny PR substitutes Python loops in the form of `for x in range(N): ...` by `for _ in range(N): ...` where applicable. The idea is indicating to the reader that a block (or statement, in list comprehensions) is just repeated N times, and that the loop counter is not used in the body, hence using the throwaway variable. This is already done quite often in the current tests (see e.g. `$ git grep "for _ in range("`). Another alternative would be using `itertools.repeat` (according to Python core developer Raymond Hettinger it's [even faster](https://twitter.com/raymondh/status/1144527183341375488)), but that doesn't seem to be widespread in use and I'm not sure about a readability increase.

  The only drawback I see is that whenever one wants to debug loop iterations, one would need to introduce a loop variable again. Reviewing this is basically a no-brainer, since tests would fail immediately if a a substitution has taken place on a loop where the variable is used.

  Instances to replace were found by `$ git grep "for.*in range("` and manually checked.

ACKs for top commit:
  darosior:
    ACK dac7a11
  instagibbs:
    manual inspection ACK bitcoin@dac7a11
  practicalswift:
    ACK dac7a11 -- the updated code is easier to reason about since the throwaway nature of a variable is expressed explicitly (using the Pythonic `_` idiom) instead of implicitly. Explicit is better than implicit was we all know by now :)

Tree-SHA512: 5f43ded9ce14e5e00b3876ec445b90acda1842f813149ae7bafa93f3ac3d510bb778e2c701187fd2c73585e6b87797bb2d2987139bd1a9ba7d58775a59392406
@theStack theStack deleted the 20200804-refactor-test-use-underscore-variable branch December 1, 2020 09:58
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Sep 8, 2021
Summary:
substitutes "for x in range(N):" by "for _ in range(N):"
indicates to the reader that a block is just repeated N times, and
that the loop counter is not used in the body

Backport notes:
 - this PR also replaces a few occurences of `range(0, x)` with `range(x)`
 - test/functional/p2p_tx_download.py has been included in test/functional/p2p_inv_download.py, and was already up to date

This is a backport of [[bitcoin/bitcoin#19674 | core#19674]]

Test Plan: `ninja check-functional-extended`

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D10068
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants