signet mining utility #19937

ajtowns · 2020-09-10T22:38:26Z

Adds contrib/signet/miner for mining signet blocks.

Adds bitcoin-util cli utility, with the idea being it can provide bitcoin related functionality that does not rely on the ability to access a running node. Only subcommand currently is "grind" which takes a hex-encoded header and grinds its nonce until its nBits is satisfied.

Updates getblocktemplate to include signet_challenge field, and makes getblocktemplate require the signet rule when invoked on the signet change. Removes connectivity and IBD checks from getblocktemplate when applied to a test chain (regtest, testnet, signet).

DrahtBot · 2020-09-19T13:39:39Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

ajtowns · 2020-09-21T22:48:42Z

EDIT: the following is no longer accurate, see contrib/signet/README.md instead

Some examples:

Generate blocks indefinitely, at 10 minute intervals, paying block reward to the given address, using bitcoin-util to do multi-threaded proof-of-work: $ ./contrib/signet/generate.py --cli='./bitcoin-cli' generate 10 --block-time=600 --address="tb1..." --grind-cmd='./bitcoin-util grind'
Generate 10 blocks, at 10 minute intervals, paying block reward to the given address, with timestamps starting at the genesis block, using single-threaded python code to do proof-of-work: $ ./contrib/signet/generate.py --cli='./bitcoin-cli' generate 10 --block-time=600 --address="tb1..." --backdate 0
Generate blocks indefinitely at 30 minute intervals, but only if there hasn't been a block in the meantime, paying the reward for block height X to the X'th address from the given descriptor: $ ./contrib/signet/generate.py --cli='./bitcoin-cli' generate -1 --block-time=1800 --descriptor="wpkh(...)#..." --secondary

Using a "--block-time=600" will get blocks exactly on 10 minute boundaries, with very little variation. You can alternatively use --mining-time=20 to target spending an average of 20 seconds of cpu time per block, which should get a mainnet-like distribution of block times once difficulty has stabilised, assuming only one miner.

You can also generate blocks more manually by running:

./bitcoin-cli -signet getblocktemplate '{"rules": ["signet","segwit"]}' | ../contrib/signet/generate.py --cli='./bitcoin-cli' genpsbt --address=tb1... -- to generate a psbt, encoding the signet txs and the raw block
./bitcoin-cli -signet -stdin walletprocesspsbt to sign the psbt from stdin, or some other method of signing a psbt
../contrib/signet/generate.py solvepsbt --grind-cmd='./bitcoin-util grind' | ./bitcoin-cli -signet submitblock to convert the signed psbt into a signet commitment and submit the block

Using the --cli option to specify ./bitcoin-cli -conf=/path/to/custom-signet.conf and having signet=1, signetchallenge and a custom datadir all specified in the conf file is probably sensible.

maflcko

Is there a risk in exposing the miner on RPC as well? Bitcoind is running anyway somewhere, and some miners might want to run everything in one process for convenience? No strong opinion.

contrib/signet/generate.py

maflcko · 2020-09-22T13:40:31Z

src/bitcoin-util.cpp

+        }
+
+        char* command = argv[1];
+        if (strcmp(command, "grind") == 0) {


Any reason to not use our ArgsManager here, so that -help documentation will be created?

I don't think ArgsManager supports subcommands that have different suboptions?

It would support -grind=block_header_hex. Is something else needed here that I am missing?

I was thinking it would make sense for bitcoin-util would have subcommands like git, each with their own different options (like --amend for git commit vs --continue for git rebase). bitcoin-util grind could have a -jobs=8 argument specifying how many jobs to run in parallel, eg, which presumably wouldn't be appropriate elsewhere.

The wallet tool is doing exactly that already with argsman. E.g. it has the info subcommand, or a dump subcommand with conditional options like -dumpfile (#19137)

This seems worth addressing still: e6473d2#r492742101

src/bitcoin-util.cpp

ajtowns · 2020-09-22T17:00:40Z

I don't think there's a risk in doing it over RPC; it's that it puts the wallet code in the middle of two mining codes bits (ie, generate a template, sign, grind proof of work). Merging mining and wallet code seemed a bit ugly, but keeping them distinct means you need a script to combine them for you, so it was easier to put everything in the script. Also seemed easier to add an RPC later if that makes sense, than to remove it if it doesn't make sense.

kallewoof

Tested ACK e077aa7

configure.ac

src/bitcoin-util.cpp

jsarenik · 2020-10-01T09:50:31Z

I still have no idea how to run a custom signet on this PR's commit f4c6ec1 (which has conflicts when I try to rebase it to master locally).

The documentation at bitcoin.it is out of date. Please help me so that I can test.

kallewoof · 2020-10-01T10:16:18Z

@jsarenik I've updated the running an issuer section of the docs. The rest are fine I think. Lemme know if not.

jsarenik · 2020-10-01T11:30:30Z

@kallewoof have a look at #15454 and add the createwallet lines to the doc please.

jsarenik · 2020-10-01T11:55:08Z

@kallewoof or start all bitcoind with -wallet command line argument.

contrib/signet/README.md

gruve-p · 2020-12-21T15:29:35Z

contrib/signet/README.md

MINER="..contrib/signet/miner" > MINER="../contrib/signet/miner"

laanwj · 2020-12-21T16:02:12Z

code review ACK f19148a9ddab01af80067982f2639dbee2cdbf52

nit: your new utility doesn't implement --version and no manual page is generated for it in contrib/devtools/gen-manpages.sh. No opinion on whether that's necessary here.

sipa · 2020-12-23T20:36:59Z

Concept ACK. I think it makes sense to have a bitcoin-util tool with subcommands for wallet- and node-free operations (and deprecate bitcoin-tx in favor of PSBT support in bitcoin-util).

gruve-p · 2020-12-24T15:58:45Z

tACK f19148a

Few docs nits.

practicalswift · 2020-12-27T23:03:54Z

Concept ACK

bitcoin-util makes sense!

laanwj · 2021-01-11T13:42:54Z

src/rpc/mining.cpp

What happens when signet rules are set when calling getblocktemplate on a non-signet network, I suppose this is (or should be) an error?

The way I read BIP9 is that by including "signet" in the rules when doing a GBT request, that just indicates you can handle it if signet rules are active -- if they're not active, then there's no reason why you can't handle that too. So adding signet to rules on mainnet should be fine as far as bitcoind is concerned -- it will just return a regular template without any signet stuff, just as if you'd indicated support for .. extension block mining or something.

Thanks, clear.

src/bitcoin-util.cpp

ajtowns · 2021-01-12T08:38:33Z

Rebased to fix the bad library ordering.

Suggest reviewing #20715 which is probably the future for bitcoin-util's argument parsing. I think we could update bitcoin-util to work that way either before or after merging this PR.

laanwj · 2021-01-12T11:53:32Z

code review ACK 595a34d

weedcoder · 2021-01-15T14:21:10Z

kallewoof · 2021-01-16T04:24:56Z

@weedcoder I'd say it is missing a lot of the logic required to do what liquid does, but I do believe this + a permanent elements chain with a 2-way-peg can do a great job as a test-chain for liquid. :)

…iscv 54ce4fa build: improve macro for testing -latomic requirement (fanquake) 2c010b9 add std::atomic include to bitcoin-util.cpp (fanquake) Pull request description: Since the merge of #19937, riscv builds have been failing, due to a link issue with [`std::atomic_exchange`](https://en.cppreference.com/w/cpp/atomic/atomic_exchange) in `bitcoin-util`: ```bash CXXLD bitcoin-util bitcoin_util-bitcoin-util.o: In function `grind_task': /home/ubuntu/build/bitcoin/distsrc-riscv64-linux-gnu/src/bitcoin-util.cpp:98: undefined reference to `__atomic_exchange_1' collect2: error: ld returned 1 exit status ``` We have a [macro](https://github.com/bitcoin/bitcoin/blob/master/build-aux/m4/l_atomic.m4) that tries to determine when `-latomic` is required, however it doesn't quite work well enough, as it's currently determining it isn't needed: ```bash ./autogen.sh ./configure --prefix=/home/ubuntu/bitcoin/depends/riscv64-linux-gnu ... checking whether std::atomic can be used without link library... yes ``` This PR adds a call to `std::atomic_exchange` to the macro, which will get us properly linked against `-latomic` on riscv: ```bash checking whether std::atomic can be used without link library... no checking whether std::atomic needs -latomic... yes ``` Also adds an `<atomic>` include to `bitcoin-util.cpp`. ACKs for top commit: laanwj: Tested ACK 54ce4fa Tree-SHA512: 963c875097ee96b131163ae8109bcf8fecf4451d20faa2f3d223f9938ea3d8d1ed5604e12ad82c2b4b1c605fd293a9b6b08fefc00dd3e68d09c49e95029c6f50

…g for riscv 54ce4fa build: improve macro for testing -latomic requirement (fanquake) 2c010b9 add std::atomic include to bitcoin-util.cpp (fanquake) Pull request description: Since the merge of bitcoin#19937, riscv builds have been failing, due to a link issue with [`std::atomic_exchange`](https://en.cppreference.com/w/cpp/atomic/atomic_exchange) in `bitcoin-util`: ```bash CXXLD bitcoin-util bitcoin_util-bitcoin-util.o: In function `grind_task': /home/ubuntu/build/bitcoin/distsrc-riscv64-linux-gnu/src/bitcoin-util.cpp:98: undefined reference to `__atomic_exchange_1' collect2: error: ld returned 1 exit status ``` We have a [macro](https://github.com/bitcoin/bitcoin/blob/master/build-aux/m4/l_atomic.m4) that tries to determine when `-latomic` is required, however it doesn't quite work well enough, as it's currently determining it isn't needed: ```bash ./autogen.sh ./configure --prefix=/home/ubuntu/bitcoin/depends/riscv64-linux-gnu ... checking whether std::atomic can be used without link library... yes ``` This PR adds a call to `std::atomic_exchange` to the macro, which will get us properly linked against `-latomic` on riscv: ```bash checking whether std::atomic can be used without link library... no checking whether std::atomic needs -latomic... yes ``` Also adds an `<atomic>` include to `bitcoin-util.cpp`. ACKs for top commit: laanwj: Tested ACK 54ce4fa Tree-SHA512: 963c875097ee96b131163ae8109bcf8fecf4451d20faa2f3d223f9938ea3d8d1ed5604e12ad82c2b4b1c605fd293a9b6b08fefc00dd3e68d09c49e95029c6f50

hebasto · 2021-02-01T19:49:10Z

src/bitcoin-util.cpp

+#include <stdio.h>
+#include <thread>
+
+#include <boost/algorithm/string.hpp>


Not sure if this include is required.

…in-wallet fa61b9d util: Add ArgsManager::GetCommand() and use it in bitcoin-wallet (MarcoFalke) 7777105 refactor: Move all command dependend checks to ExecuteWalletToolFunc (MarcoFalke) fa06bce test: Add tests (MarcoFalke) fac05cc wallet: [refactor] Pass ArgsManager to WalletAppInit (MarcoFalke) Pull request description: This not only moves the parsing responsibility out from the wallet tool, but it also makes it easier to implement bitcoin-util #19937 Fixes: #20902 ACKs for top commit: ajtowns: ACK fa61b9d fjahr: Code review ACK fa61b9d Tree-SHA512: 79622b806e8bf9dcd0dc24a8a6687345710df57720992e83a41cd8d6762a6dc112044ebc58fcf6e8fbf45de29a79b04873c5b8c2494a1eaaf902a2884703e47b

b3c712c contrib/signet/miner: remove debug code (Anthony Towns) 297e351 bitcoin-util: use AddCommand / GetCommand (Anthony Towns) b6d493f contrib/signet/README.md: Update miner description (Anthony Towns) e665438 contrib/signet/miner: Automatic timestamp for first block (Anthony Towns) a383ce5 contrib/signet/miner: --grind-cmd is required for calibrate (Anthony Towns) 1a45cd2 contrib/signet: Fix typos (Anthony Towns) Pull request description: Followups from #19937 ACKs for top commit: laanwj: Code review ACK b3c712c Tree-SHA512: a1003f9ee3697438114b60872b50f4300c8b52f0d58551566eb61c421d787525807ae75be205dcab2c24358cd568f53260120880109a9d728773405ff987596f

DrahtBot added Build system GUI Mining RPC/REST/ZMQ Scripts and tools Utils/log/libs Validation labels Sep 10, 2020

This was referenced Sep 19, 2020

net: Replace enum CConnMan::NumConnections with enum class ConnectionDirection #19771

Merged

Introduce deploymentstatus #19438

Merged

BIP-325: Signet [consensus] #18267

Merged

Allow whitelisting outgoing connections #17167

Closed

ajtowns force-pushed the 202009-signet-generate branch from 435cfff to e077aa7 Compare September 21, 2020 21:33

ajtowns removed GUI Validation labels Sep 21, 2020

ajtowns marked this pull request as ready for review September 21, 2020 22:26

maflcko removed Build system Mining RPC/REST/ZMQ labels Sep 22, 2020

maflcko reviewed Sep 22, 2020

View reviewed changes

maflcko added RPC/REST/ZMQ Mining labels Sep 22, 2020

kallewoof reviewed Sep 30, 2020

View reviewed changes

configure.ac Show resolved Hide resolved

src/bitcoin-util.cpp Outdated Show resolved Hide resolved

src/bitcoin-util.cpp Show resolved Hide resolved

maflcko reviewed Sep 30, 2020

View reviewed changes

src/bitcoin-util.cpp Outdated Show resolved Hide resolved

gruve-p reviewed Dec 20, 2020

View reviewed changes

contrib/signet/README.md Outdated Show resolved Hide resolved

contrib/signet/README.md Outdated Show resolved Hide resolved

ajtowns force-pushed the 202009-signet-generate branch from 69c6e47 to f19148a Compare December 21, 2020 02:19

gruve-p reviewed Dec 21, 2020

View reviewed changes

laanwj reviewed Jan 11, 2021

View reviewed changes

src/bitcoin-util.cpp Outdated Show resolved Hide resolved

ajtowns added 3 commits January 12, 2021 18:34

Add bitcoin-util command line utility

13762bc

contrib/signet: Add script for generating a signet chain

ff7dbdc

contrib/signet: Document miner script in README.md

595a34d

ajtowns force-pushed the 202009-signet-generate branch from f19148a to 595a34d Compare January 12, 2021 08:35

laanwj merged commit 7b97563 into bitcoin:master Jan 12, 2021

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jan 12, 2021

Merge bitcoin#19937: signet mining utility

0e8638b

bitcoin deleted a comment from bitmastercoin Jan 13, 2021

ajtowns mentioned this pull request Jan 13, 2021

signet miner followups #20923

Merged

fanquake mentioned this pull request Jan 15, 2021

build: fix linking against -latomic when building for riscv #20938

Merged

bitcoin deleted a comment Jan 19, 2021

hebasto reviewed Feb 1, 2021

View reviewed changes

maflcko mentioned this pull request Jun 17, 2021

test: Add bitcoin-util tests (+refactors) #22270

Merged

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

signet mining utility #19937

signet mining utility #19937

Uh oh!

Conversation

ajtowns commented Sep 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Sep 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

ajtowns commented Sep 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maflcko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajtowns commented Sep 22, 2020

Uh oh!

kallewoof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsarenik commented Oct 1, 2020

Uh oh!

kallewoof commented Oct 1, 2020

Uh oh!

jsarenik commented Oct 1, 2020

Uh oh!

jsarenik commented Oct 1, 2020

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj commented Dec 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Dec 23, 2020

Uh oh!

gruve-p commented Dec 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

practicalswift commented Dec 27, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajtowns commented Jan 12, 2021

Uh oh!

laanwj commented Jan 12, 2021

Uh oh!

weedcoder commented Jan 15, 2021

Uh oh!

kallewoof commented Jan 16, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

ajtowns commented Sep 10, 2020 •

edited

Loading

DrahtBot commented Sep 19, 2020 •

edited

Loading

ajtowns commented Sep 21, 2020 •

edited

Loading

laanwj commented Dec 21, 2020 •

edited

Loading

gruve-p commented Dec 24, 2020 •

edited

Loading