|
1 | | -Regression tests |
2 | | -================ |
| 1 | +# Functional tests |
| 2 | + |
| 3 | +### Writing Functional Tests |
| 4 | + |
| 5 | +#### Example test |
| 6 | + |
| 7 | +The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both |
| 8 | +the RPC and P2P interfaces. If you are writing your first test, copy that file |
| 9 | +and modify to fit your needs. |
| 10 | + |
| 11 | +#### Coverage |
| 12 | + |
| 13 | +Running `test_runner.py` with the `--coverage` argument tracks which RPCs are |
| 14 | +called by the tests and prints a report of uncovered RPCs in the summary. This |
| 15 | +can be used (along with the `--extended` argument) to find out which RPCs we |
| 16 | +don't have test cases for. |
| 17 | + |
| 18 | +#### Style guidelines |
| 19 | + |
| 20 | +- Where possible, try to adhere to [PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/) |
| 21 | +- Use a python linter like flake8 before submitting PRs to catch common style |
| 22 | + nits (eg trailing whitespace, unused imports, etc) |
| 23 | +- The oldest supported Python version is specified in [doc/dependencies.md](/doc/dependencies.md). |
| 24 | + Consider using [pyenv](https://github.com/pyenv/pyenv), which checks [.python-version](/.python-version), |
| 25 | + to prevent accidentally introducing modern syntax from an unsupported Python version. |
| 26 | + The Travis linter also checks this, but [possibly not in all cases](https://github.com/bitcoin/bitcoin/pull/14884#discussion_r239585126). |
| 27 | +- See [the python lint script](/test/lint/lint-python.sh) that checks for violations that |
| 28 | + could lead to bugs and issues in the test code. |
| 29 | +- Avoid wildcard imports |
| 30 | +- Use a module-level docstring to describe what the test is testing, and how it |
| 31 | + is testing it. |
| 32 | +- When subclassing the BitcoinTestFramwork, place overrides for the |
| 33 | + `set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of |
| 34 | + the subclass, then locally-defined helper methods, then the `run_test()` method. |
| 35 | +- Use `'{}'.format(x)` for string formatting, not `'%s' % x`. |
| 36 | + |
| 37 | +#### Naming guidelines |
| 38 | + |
| 39 | +- Name the test `<area>_test.py`, where area can be one of the following: |
| 40 | + - `feature` for tests for full features that aren't wallet/mining/mempool, eg `feature_rbf.py` |
| 41 | + - `interface` for tests for other interfaces (REST, ZMQ, etc), eg `interface_rest.py` |
| 42 | + - `mempool` for tests for mempool behaviour, eg `mempool_reorg.py` |
| 43 | + - `mining` for tests for mining features, eg `mining_prioritisetransaction.py` |
| 44 | + - `p2p` for tests that explicitly test the p2p interface, eg `p2p_disconnect_ban.py` |
| 45 | + - `rpc` for tests for individual RPC methods or features, eg `rpc_listtransactions.py` |
| 46 | + - `tool` for tests for tools, eg `tool_wallet.py` |
| 47 | + - `wallet` for tests for wallet features, eg `wallet_keypool.py` |
| 48 | +- use an underscore to separate words |
| 49 | + - exception: for tests for specific RPCs or command line options which don't include underscores, name the test after the exact RPC or argument name, eg `rpc_decodescript.py`, not `rpc_decode_script.py` |
| 50 | +- Don't use the redundant word `test` in the name, eg `interface_zmq.py`, not `interface_zmq_test.py` |
| 51 | + |
| 52 | +#### General test-writing advice |
| 53 | + |
| 54 | +- Set `self.num_nodes` to the minimum number of nodes necessary for the test. |
| 55 | + Having additional unrequired nodes adds to the execution time of the test as |
| 56 | + well as memory/CPU/disk requirements (which is important when running tests in |
| 57 | + parallel or on Travis). |
| 58 | +- Avoid stop-starting the nodes multiple times during the test if possible. A |
| 59 | + stop-start takes several seconds, so doing it several times blows up the |
| 60 | + runtime of the test. |
| 61 | +- Set the `self.setup_clean_chain` variable in `set_test_params()` to control whether |
| 62 | + or not to use the cached data directories. The cached data directories |
| 63 | + contain a 200-block pre-mined blockchain and wallets for four nodes. Each node |
| 64 | + has 25 mature blocks (25x50=1250 BTC) in its wallet. |
| 65 | +- When calling RPCs with lots of arguments, consider using named keyword |
| 66 | + arguments instead of positional arguments to make the intent of the call |
| 67 | + clear to readers. |
| 68 | +- Many of the core test framework classes such as `CBlock` and `CTransaction` |
| 69 | + don't allow new attributes to be added to their objects at runtime like |
| 70 | + typical Python objects allow. This helps prevent unpredictable side effects |
| 71 | + from typographical errors or usage of the objects outside of their intended |
| 72 | + purpose. |
| 73 | + |
| 74 | +#### RPC and P2P definitions |
| 75 | + |
| 76 | +Test writers may find it helpful to refer to the definitions for the RPC and |
| 77 | +P2P messages. These can be found in the following source files: |
| 78 | + |
| 79 | +- `/src/rpc/*` for RPCs |
| 80 | +- `/src/wallet/rpc*` for wallet RPCs |
| 81 | +- `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages |
| 82 | + |
| 83 | +#### Using the P2P interface |
| 84 | + |
| 85 | +- `messages.py` contains all the definitions for objects that pass |
| 86 | +over the network (`CBlock`, `CTransaction`, etc, along with the network-level |
| 87 | +wrappers for them, `msg_block`, `msg_tx`, etc). |
| 88 | + |
| 89 | +- P2P tests have two threads. One thread handles all network communication |
| 90 | +with the pivxd(s) being tested in a callback-based event loop; the other |
| 91 | +implements the test logic. |
| 92 | + |
| 93 | +- `P2PConnection` is the class used to connect to a pivxd. `P2PInterface` |
| 94 | +contains the higher level logic for processing P2P payloads and connecting to |
| 95 | +the Bitcoin Core node application logic. For custom behaviour, subclass the |
| 96 | +P2PInterface object and override the callback methods. |
| 97 | + |
| 98 | +- Can be used to write tests where specific P2P protocol behavior is tested. |
| 99 | +Examples tests are `p2p_unrequested_blocks.py`, `p2p_compactblocks.py`. |
| 100 | + |
| 101 | +### test-framework modules |
3 | 102 |
|
4 | | -### [test_framework/authproxy.py](test_framework/authproxy.py) |
| 103 | +#### [test_framework/authproxy.py](test_framework/authproxy.py) |
5 | 104 | Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). |
6 | 105 |
|
7 | | -### [test_framework/test_framework.py](test_framework/test_framework.py) |
8 | | -Base class for new regression tests. |
| 106 | +#### [test_framework/test_framework.py](test_framework/test_framework.py) |
| 107 | +Base class for functional tests. |
9 | 108 |
|
10 | | -### [test_framework/util.py](test_framework/util.py) |
| 109 | +#### [test_framework/util.py](test_framework/util.py) |
11 | 110 | Generally useful functions. |
12 | 111 |
|
13 | | -### [test_framework/mininode.py](test_framework/mininode.py) |
14 | | -Basic code to support p2p connectivity to a pivxd. |
| 112 | +#### [test_framework/mininode.py](test_framework/mininode.py) |
| 113 | +Basic code to support P2P connectivity to a pivxd. |
15 | 114 |
|
16 | | -### [test_framework/comptool.py](test_framework/comptool.py) |
| 115 | +#### [test_framework/comptool.py](test_framework/comptool.py) |
17 | 116 | Framework for comparison-tool style, p2p tests. |
18 | 117 |
|
19 | | -### [test_framework/script.py](test_framework/script.py) |
| 118 | +#### [test_framework/script.py](test_framework/script.py) |
20 | 119 | Utilities for manipulating transaction scripts (originally from python-bitcoinlib) |
21 | 120 |
|
22 | | -### [test_framework/blockstore.py](test_framework/blockstore.py) |
| 121 | +#### [test_framework/blockstore.py](test_framework/blockstore.py) |
23 | 122 | Implements disk-backed block and tx storage. |
24 | 123 |
|
25 | | -### [test_framework/key.py](test_framework/key.py) |
| 124 | +#### [test_framework/key.py](test_framework/key.py) |
26 | 125 | Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib) |
27 | 126 |
|
28 | | -### [test_framework/bignum.py](test_framework/bignum.py) |
| 127 | +#### [test_framework/bignum.py](test_framework/bignum.py) |
29 | 128 | Helpers for script.py |
30 | 129 |
|
31 | | -### [test_framework/blocktools.py](test_framework/blocktools.py) |
| 130 | +#### [test_framework/blocktools.py](test_framework/blocktools.py) |
32 | 131 | Helper functions for creating blocks and transactions. |
33 | 132 |
|
34 | | -P2P test design notes |
35 | | ---------------------- |
36 | | - |
37 | | -## Mininode |
38 | | - |
39 | | -* ```mininode.py``` contains all the definitions for objects that pass |
40 | | -over the network (```CBlock```, ```CTransaction```, etc, along with the network-level |
41 | | -wrappers for them, ```msg_block```, ```msg_tx```, etc). |
42 | | - |
43 | | -* P2P tests have two threads. One thread handles all network communication |
44 | | -with the pivxd(s) being tested (using python's asyncore package); the other |
45 | | -implements the test logic. |
46 | | - |
47 | | -* ```NodeConn``` is the class used to connect to a pivxd. If you implement |
48 | | -a callback class that derives from ```NodeConnCB``` and pass that to the |
49 | | -```NodeConn``` object, your code will receive the appropriate callbacks when |
50 | | -events of interest arrive. |
51 | | - |
52 | | -* You can pass the same handler to multiple ```NodeConn```'s if you like, or pass |
53 | | -different ones to each -- whatever makes the most sense for your test. |
54 | | - |
55 | | -* Call ```NetworkThread.start()``` after all ```NodeConn``` objects are created to |
56 | | -start the networking thread. (Continue with the test logic in your existing |
57 | | -thread.) |
58 | | - |
59 | | -* RPC calls are available in p2p tests. |
60 | | - |
61 | | -* Can be used to write free-form tests, where specific p2p-protocol behavior |
62 | | -is tested. Examples: ```p2p-accept-block.py```, ```maxblocksinflight.py```. |
63 | | - |
64 | | -## Comptool |
| 133 | +### Comptool |
65 | 134 |
|
66 | 135 | * Testing framework for writing tests that compare the block/tx acceptance |
67 | 136 | behavior of a pivxd against 1 or more other pivxd instances, or against |
|
0 commit comments