Looks like this runs the extended tests (no pruning) 4 times faster on my machine:

Total time: 314 s (accumulated: 1306 s)

Wow. This is a great idea.
Concept ACK.

Concept ACK, thanks for working on this!

Nice! Concept ACK. I wil review better and benchmark another day.

First run testing this (shortened tests),

ALL   | True   | 764 s (accumulated)

Runtime: 212 s

Concept ACK

@MarcoFalke This is great, thanks for tackling! Any idea what's causing the travis failure?

I think it would be nice to make the number of simultaneous jobs configurable on the command line; the default of 4 seems good but being able to tweak it for the machine it's running on (say if other developers run this script locally, as I do) would be nice.

I also noticed that by re-sorting the standard tests so that the slowest jobs start first, I was able to get the runtime of all the standard tests running in parallel down to the same as running the single longest test (walletbackup.py) -- awesome!

Hm, I just ran the extended tests and got an odd failure -- this test looks like it is supposed to be marked successful.

bip9-softforks.py:
Initializing test directory /tmp/testE9D_TH
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:13641
Test 1: PASS [143]
Test 2: PASS [287]
Test 3: PASS [431]
Test 4: PASS [574]
Test 5: PASS [575]
Test 6: PASS [575]
Initializing test directory /tmp/testE9D_TH
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:13641
Test 7: PASS [143]
Test 8: PASS [287]
Test 9: PASS [431]
Test 10: PASS [574]
Test 11: PASS [575]
Test 12: PASS [575]
Initializing test directory /tmp/testE9D_TH
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:13641
Test 13: PASS [143]
Test 14: PASS [287]
Test 15: PASS [431]
Test 16: PASS [574]
Test 17: PASS [575]
Test 18: PASS [575]
Initializing test directory /tmp/testE9D_TH
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:13641
Stopping nodes
Cleaning up
Tests successful

BDB3028 /tmp/testE9D_TH/blocks.db: unable to flush: No such file or directory

Pass: False, Duration: 41 s

Maybe that last line about "unable to flush" is going to stderr, causing this to be recorded as failure? If so then probably the test should be fixed.

Agree with @sdaftuar that there is benefit in sorting the standard tests.
Longest -> shortest:

ALL | True   | 857 s (accumulated)
Runtime: 218 s

Compared to as is:

ALL | True   | 845 s (accumulated)
Runtime: 238 s

ACK fa48ef8

Thanks!

Edit: I was checking it pre-python3, so ...

This needs to be updated to use http.client before it can merged, now that the Python3 change is in.

• Rebased and translated to python3
• made it possible to specify -parallel=n on the command line
• Moved walletbackup.py to first position
• --portseed is set in a deterministic manner for each rpc test

$ python3 ./qa/pull-tester/rpc-tests.py -parallel=20000
...
ALL                            | True   | 634 s (accumulated)
Runtime: 143 s

$ python3 ./qa/pull-tester/rpc-tests.py -parallel=20000 -extended
...
maxuploadtarget.py             | True   | 81 s
walletbackup.py                | True   | 135 s
smartfees.py                   | True   | 341 s
pruning.py                     | True   | 1503 s

ALL                            | False  | 3076 s (accumulated)

Runtime: 1503 s

Now I'm too lazzy to run them before this patch to compare...benchmark conclusion: this is great.

It also comes to mind that, in terms of time, only smartfees and pruning deserve to be in the extended tests group more then walletbackup. But probably moving walletbackup to the extended tests (and maybe some of the faster ones out of the extended group to the regular one at the same time?) is out of the scope of this PR.

Tested ACK.

This is great. Looking at a random travis before/after, this goes from 535sec to 164sec.
untested (locally) ACK.

This looks great, though there's still the issue with the bip9-softforks.py test outputting to stderr, causing the extended tests to fail. I'm trying to figure out a fix to suggest for that, but I'm not having any great ideas at the moment...

reACK ccccc59

@sdaftuar This seems to be a local problem or a problem with the bip9 test and is not caused by this pull, I assume?

Runs fine locally, just now:

TEST              | PASSED | DURATION

bip9-softforks.py | True   | 66 s

ALL               | True   | 66 s (accumulated)

Runtime: 66 s

Yep I didn't realize this was just affecting me, will open a separate issue.

ACK.