Skip to content

Commit 9335a10

Browse files
committed
Update test/functional/README.md
Added more details regarding the purpose, writing, and use of the regression test suite.
1 parent 20f2513 commit 9335a10

File tree

1 file changed

+114
-45
lines changed

1 file changed

+114
-45
lines changed

test/functional/README.md

Lines changed: 114 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,67 +1,136 @@
1-
Regression tests
2-
================
1+
# Functional tests
2+
3+
### Writing Functional Tests
4+
5+
#### Example test
6+
7+
The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both
8+
the RPC and P2P interfaces. If you are writing your first test, copy that file
9+
and modify to fit your needs.
10+
11+
#### Coverage
12+
13+
Running `test_runner.py` with the `--coverage` argument tracks which RPCs are
14+
called by the tests and prints a report of uncovered RPCs in the summary. This
15+
can be used (along with the `--extended` argument) to find out which RPCs we
16+
don't have test cases for.
17+
18+
#### Style guidelines
19+
20+
- Where possible, try to adhere to [PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/)
21+
- Use a python linter like flake8 before submitting PRs to catch common style
22+
nits (eg trailing whitespace, unused imports, etc)
23+
- The oldest supported Python version is specified in [doc/dependencies.md](/doc/dependencies.md).
24+
Consider using [pyenv](https://github.com/pyenv/pyenv), which checks [.python-version](/.python-version),
25+
to prevent accidentally introducing modern syntax from an unsupported Python version.
26+
The Travis linter also checks this, but [possibly not in all cases](https://github.com/bitcoin/bitcoin/pull/14884#discussion_r239585126).
27+
- See [the python lint script](/test/lint/lint-python.sh) that checks for violations that
28+
could lead to bugs and issues in the test code.
29+
- Avoid wildcard imports
30+
- Use a module-level docstring to describe what the test is testing, and how it
31+
is testing it.
32+
- When subclassing the BitcoinTestFramwork, place overrides for the
33+
`set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of
34+
the subclass, then locally-defined helper methods, then the `run_test()` method.
35+
- Use `'{}'.format(x)` for string formatting, not `'%s' % x`.
36+
37+
#### Naming guidelines
38+
39+
- Name the test `<area>_test.py`, where area can be one of the following:
40+
- `feature` for tests for full features that aren't wallet/mining/mempool, eg `feature_rbf.py`
41+
- `interface` for tests for other interfaces (REST, ZMQ, etc), eg `interface_rest.py`
42+
- `mempool` for tests for mempool behaviour, eg `mempool_reorg.py`
43+
- `mining` for tests for mining features, eg `mining_prioritisetransaction.py`
44+
- `p2p` for tests that explicitly test the p2p interface, eg `p2p_disconnect_ban.py`
45+
- `rpc` for tests for individual RPC methods or features, eg `rpc_listtransactions.py`
46+
- `tool` for tests for tools, eg `tool_wallet.py`
47+
- `wallet` for tests for wallet features, eg `wallet_keypool.py`
48+
- use an underscore to separate words
49+
- exception: for tests for specific RPCs or command line options which don't include underscores, name the test after the exact RPC or argument name, eg `rpc_decodescript.py`, not `rpc_decode_script.py`
50+
- Don't use the redundant word `test` in the name, eg `interface_zmq.py`, not `interface_zmq_test.py`
51+
52+
#### General test-writing advice
53+
54+
- Set `self.num_nodes` to the minimum number of nodes necessary for the test.
55+
Having additional unrequired nodes adds to the execution time of the test as
56+
well as memory/CPU/disk requirements (which is important when running tests in
57+
parallel or on Travis).
58+
- Avoid stop-starting the nodes multiple times during the test if possible. A
59+
stop-start takes several seconds, so doing it several times blows up the
60+
runtime of the test.
61+
- Set the `self.setup_clean_chain` variable in `set_test_params()` to control whether
62+
or not to use the cached data directories. The cached data directories
63+
contain a 200-block pre-mined blockchain and wallets for four nodes. Each node
64+
has 25 mature blocks (25x50=1250 BTC) in its wallet.
65+
- When calling RPCs with lots of arguments, consider using named keyword
66+
arguments instead of positional arguments to make the intent of the call
67+
clear to readers.
68+
- Many of the core test framework classes such as `CBlock` and `CTransaction`
69+
don't allow new attributes to be added to their objects at runtime like
70+
typical Python objects allow. This helps prevent unpredictable side effects
71+
from typographical errors or usage of the objects outside of their intended
72+
purpose.
73+
74+
#### RPC and P2P definitions
75+
76+
Test writers may find it helpful to refer to the definitions for the RPC and
77+
P2P messages. These can be found in the following source files:
78+
79+
- `/src/rpc/*` for RPCs
80+
- `/src/wallet/rpc*` for wallet RPCs
81+
- `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages
82+
83+
#### Using the P2P interface
84+
85+
- `messages.py` contains all the definitions for objects that pass
86+
over the network (`CBlock`, `CTransaction`, etc, along with the network-level
87+
wrappers for them, `msg_block`, `msg_tx`, etc).
88+
89+
- P2P tests have two threads. One thread handles all network communication
90+
with the pivxd(s) being tested in a callback-based event loop; the other
91+
implements the test logic.
92+
93+
- `P2PConnection` is the class used to connect to a pivxd. `P2PInterface`
94+
contains the higher level logic for processing P2P payloads and connecting to
95+
the Bitcoin Core node application logic. For custom behaviour, subclass the
96+
P2PInterface object and override the callback methods.
97+
98+
- Can be used to write tests where specific P2P protocol behavior is tested.
99+
Examples tests are `p2p_unrequested_blocks.py`, `p2p_compactblocks.py`.
100+
101+
### test-framework modules
3102

4-
### [test_framework/authproxy.py](test_framework/authproxy.py)
103+
#### [test_framework/authproxy.py](test_framework/authproxy.py)
5104
Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc).
6105

7-
### [test_framework/test_framework.py](test_framework/test_framework.py)
8-
Base class for new regression tests.
106+
#### [test_framework/test_framework.py](test_framework/test_framework.py)
107+
Base class for functional tests.
9108

10-
### [test_framework/util.py](test_framework/util.py)
109+
#### [test_framework/util.py](test_framework/util.py)
11110
Generally useful functions.
12111

13-
### [test_framework/mininode.py](test_framework/mininode.py)
14-
Basic code to support p2p connectivity to a pivxd.
112+
#### [test_framework/mininode.py](test_framework/mininode.py)
113+
Basic code to support P2P connectivity to a pivxd.
15114

16-
### [test_framework/comptool.py](test_framework/comptool.py)
115+
#### [test_framework/comptool.py](test_framework/comptool.py)
17116
Framework for comparison-tool style, p2p tests.
18117

19-
### [test_framework/script.py](test_framework/script.py)
118+
#### [test_framework/script.py](test_framework/script.py)
20119
Utilities for manipulating transaction scripts (originally from python-bitcoinlib)
21120

22-
### [test_framework/blockstore.py](test_framework/blockstore.py)
121+
#### [test_framework/blockstore.py](test_framework/blockstore.py)
23122
Implements disk-backed block and tx storage.
24123

25-
### [test_framework/key.py](test_framework/key.py)
124+
#### [test_framework/key.py](test_framework/key.py)
26125
Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib)
27126

28-
### [test_framework/bignum.py](test_framework/bignum.py)
127+
#### [test_framework/bignum.py](test_framework/bignum.py)
29128
Helpers for script.py
30129

31-
### [test_framework/blocktools.py](test_framework/blocktools.py)
130+
#### [test_framework/blocktools.py](test_framework/blocktools.py)
32131
Helper functions for creating blocks and transactions.
33132

34-
P2P test design notes
35-
---------------------
36-
37-
## Mininode
38-
39-
* ```mininode.py``` contains all the definitions for objects that pass
40-
over the network (```CBlock```, ```CTransaction```, etc, along with the network-level
41-
wrappers for them, ```msg_block```, ```msg_tx```, etc).
42-
43-
* P2P tests have two threads. One thread handles all network communication
44-
with the pivxd(s) being tested (using python's asyncore package); the other
45-
implements the test logic.
46-
47-
* ```NodeConn``` is the class used to connect to a pivxd. If you implement
48-
a callback class that derives from ```NodeConnCB``` and pass that to the
49-
```NodeConn``` object, your code will receive the appropriate callbacks when
50-
events of interest arrive.
51-
52-
* You can pass the same handler to multiple ```NodeConn```'s if you like, or pass
53-
different ones to each -- whatever makes the most sense for your test.
54-
55-
* Call ```NetworkThread.start()``` after all ```NodeConn``` objects are created to
56-
start the networking thread. (Continue with the test logic in your existing
57-
thread.)
58-
59-
* RPC calls are available in p2p tests.
60-
61-
* Can be used to write free-form tests, where specific p2p-protocol behavior
62-
is tested. Examples: ```p2p-accept-block.py```, ```maxblocksinflight.py```.
63-
64-
## Comptool
133+
### Comptool
65134

66135
* Testing framework for writing tests that compare the block/tx acceptance
67136
behavior of a pivxd against 1 or more other pivxd instances, or against

0 commit comments

Comments
 (0)