Skip to content

Conversation

@theStack
Copy link
Contributor

@theStack theStack commented Jan 26, 2023

In the functional test wallet_groups.py we whitelist peers on all nodes ([email protected]) to enable immediate tx relay for fast mempool synchronization. However, considering that this setting only applies to inbound peers and the default test topology looks like this:

    node0 <--- node1 <---- node2 <--- ... <-- nodeN

txs propagate fast only from lower- to higher-numbered nodes (i.e. "left to right" in the above diagram) and take long from higher- to lower-numbered nodes ("right to left") since in the latter direction we only have outbound peers, where the trickle relay is still active. As a consequence, if a tx is submitted from any node other than node0, the mempool synchronization can take quite long.

This PR fixes this by simply adding another connection from node0 to the last node, leading to a ~2-3x speedup (5 runs measured via time ./test/functional/wallet_groups.py are shown):

master:
    0m53.31s real     0m08.22s user     0m05.60s system
    0m32.85s real     0m07.44s user     0m04.08s system
    0m46.40s real     0m09.18s user     0m04.23s system
    0m46.96s real     0m11.10s user     0m05.74s system
    0m57.23s real     0m10.53s user     0m05.59s system

PR:
    0m19.64s real     0m09.58s user     0m05.50s system
    0m18.05s real     0m07.77s user     0m04.03s system
    0m18.99s real     0m07.90s user     0m04.25s system
    0m17.49s real     0m07.56s user     0m03.92s system
    0m18.11s real     0m07.74s user     0m03.88s system

Note that in most tests this is not a problem since txs very often originate from node0.

@DrahtBot
Copy link
Contributor

DrahtBot commented Jan 26, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK brunoerg

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

@DrahtBot DrahtBot added the Tests label Jan 26, 2023
@furszy
Copy link
Member

furszy commented Jan 26, 2023

Good catch.

I think that would be preferable to enable whitelisting for outbound peers rather than tackle each specific test case with workarounds like this one (the trickle mechanism is irrelevant for the large majority of tests).

Without knowing the background story, it seems odd to only apply the whitelist flag to connections that weren't originated by the node. Supposedly, as the node picked it, outbound connections should be "more" trustworthy than inbound (so, disable the trickle mechanism if the address was whitelisted shouldn't be a problem).

@brunoerg
Copy link
Contributor

Got these results from my macOS 13 machine:

master:
./test/functional/wallet_groups.py 2.69s user 0.81s system 9% cpu 35.928 total
./test/functional/wallet_groups.py 2.65s user 0.84s system 9% cpu 36.626 total
./test/functional/wallet_groups.py 2.80s user 0.88s system 9% cpu 40.715 total
./test/functional/wallet_groups.py 2.72s user 0.90s system 10% cpu 34.687 total
./test/functional/wallet_groups.py 2.54s user 0.78s system 13% cpu 25.363 total

pr branch:
./test/functional/wallet_groups.py 2.29s user 0.63s system 24% cpu 11.881 total
./test/functional/wallet_groups.py 2.39s user 0.65s system 25% cpu 12.164 total
./test/functional/wallet_groups.py 2.30s user 0.71s system 22% cpu 13.155 total
./test/functional/wallet_groups.py 2.29s user 0.72s system 28% cpu 10.510 total
./test/functional/wallet_groups.py 2.37s user 0.74s system 25% cpu 12.234 total

Copy link
Contributor

@brunoerg brunoerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK ab4efad

@furszy
Copy link
Member

furszy commented Feb 12, 2023

Seems that what I wrote above was suggested years ago #9923. Might be a good opportunity to revive the topic.

@brunoerg
Copy link
Contributor

Seems that what I wrote above was suggested years ago #9923. Might be a good opportunity to revive the topic.

See #26441

@maflcko maflcko merged commit a631659 into bitcoin:master Feb 13, 2023
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 13, 2023
ab4efad test: fix immediate tx relay in wallet_groups.py (Sebastian Falbesoner)

Pull request description:

  In the functional test wallet_groups.py we whitelist peers on all nodes (`[email protected]`) to enable immediate tx relay for fast mempool synchronization. However, considering that this setting only applies to inbound peers and the default test topology looks like this:
  ```
      node0 <--- node1 <---- node2 <--- ... <-- nodeN
  ```

  txs propagate fast only from lower- to higher-numbered nodes (i.e. "left to right" in the above diagram) and take long from higher- to lower-numbered nodes ("right to left") since in the latter direction we only have outbound peers, where the trickle relay is still active. As a consequence, if a tx is submitted from any node other than node0, the mempool synchronization can take quite long.

  This PR fixes this by simply adding another connection from node0 to the last node, leading to a ~2-3x speedup (5 runs measured via `time ./test/functional/wallet_groups.py` are shown):

  ```
  master:
      0m53.31s real     0m08.22s user     0m05.60s system
      0m32.85s real     0m07.44s user     0m04.08s system
      0m46.40s real     0m09.18s user     0m04.23s system
      0m46.96s real     0m11.10s user     0m05.74s system
      0m57.23s real     0m10.53s user     0m05.59s system

  PR:
      0m19.64s real     0m09.58s user     0m05.50s system
      0m18.05s real     0m07.77s user     0m04.03s system
      0m18.99s real     0m07.90s user     0m04.25s system
      0m17.49s real     0m07.56s user     0m03.92s system
      0m18.11s real     0m07.74s user     0m03.88s system
  ```
  Note that in most tests this is not a problem since txs very often originate from node0.

ACKs for top commit:
  brunoerg:
    utACK ab4efad

Tree-SHA512: 12675357e6eb5a18383f2bfe719a184c0790863b37a98749d8e757dd5dc3a36212e16a81f0a192340c11b793eda00db359c7011f46f7c27e3a093af4f5b62147
@theStack theStack deleted the 202301-test-fix_wallet_groups_immediate_tx_relay branch February 13, 2023 16:17
@bitcoin bitcoin locked and limited conversation to collaborators Feb 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants