Skip to content

Conversation

@Sjors
Copy link
Member

@Sjors Sjors commented Dec 5, 2025

For Stratum v2 custom job declaration to be bandwidth efficient, the pool can request1 only the transactions that it doesn't know about.

The spec doesn't specify how this is achieved, but one method is to call the getrawtransaction RPC on each transaction id listed in DeclareMiningJob (or a subset if the pool software maintains a cache). Using RPC is inefficient, made worse by the need to make multiple calls. It also doesn't support queuing by witness id (yet, see #34013).

This PR introduces two new IPC methods:

  • getTransactions(): takes a list of Txid's
  • getTransactionsByWitnessID(): : takes a list of Wtxid's

Both return a list of serialised transactions. An empty element is returned for transactions that were not found.

The Txid variant, just like its RPC counterpart, takes advantage of -txindex if enabled. The Wtxid variant can't do that unless we add a -witnesstxindex, but at least sv2 has no use for that.

The RPC additionally allows providing a block_hash hint. I haven't implemented that here because I don't have a need or it. It can always be added later.

I thought about having a single (or overloaded) getTransactions() that works with both Txid and Wtxid, but I prefer that clients are intentional about which one they want.

A unit and functional test cover the new functionality.

Sv2 probably only needs getTransactionsByWitnessID(), but it's easy enough to just add both.

TODO:

Footnotes

  1. there's two reasons the pool requests these transactions: to approve the template and to broadcast the block if a solution is found (the miner will also broadcast via their template provider). See also https://github.com/stratum-mining/sv2-spec/issues/170

Add a simple test for both the Txid and Wtxid variants of GetTransaction().
@DrahtBot DrahtBot added the Mining label Dec 5, 2025
@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 5, 2025

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34020.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #34003 (test: interface_ipc.py minor fixes and cleanup by ryanofsky)
  • #33965 (mining: fix -blockreservedweight shadows IPC option by Sjors)
  • #33936 (mining: pass missing context to createNewBlock() and checkBlock() by Sjors)
  • #33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)
  • #29409 (multiprocess: Add capnp wrapper for Chain interface by ryanofsky)
  • #19461 (multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky)
  • #19460 (multiprocess: Add bitcoin-wallet -ipcconnect option by ryanofsky)
  • #10102 (Multiprocess bitcoin by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

  • represeent -> represent [spelling error in "represeent null pointers" makes the sentence harder to read]

2025-12-08

@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 5, 2025

🚧 At least one of the CI tasks failed.
Task lint: https://github.com/bitcoin/bitcoin/actions/runs/19973417058/job/57283535788
LLM reason (✨ experimental): Lint failure: a subtree directory was touched without performing a proper subtree merge.

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

@ryanofsky
Copy link
Contributor

figure out Treat std::shared_ptr nullptr as empty Data bitcoin-core/libmultiprocess#235 (this PR is draft pending that, subtree linter failure is expected)

This turned out to be an interesting problem, because in general it's not safe to treat empty Data fields the same as unset values. For example with std::optional<std::string> or std::string* we would want to interpret empty Data fields as empty strings not null values. Even for serializable types it's not really safe to treat empty Data the same as null since empty Data could be a valid serialization for some objects. For CTransaction objects though it is safe.

Another annoying thing is that if the goal is to be able to serialize vector<CTransactionRef> as List(Data) and allow the vector to contain null values, the most obvious way to do it would be to map null CTransactionRef values to null data values. But annoyingly although the Cap'n Proto wire format can represent null Data values in a list and has APIs to set null Data values and has APIs to distinguish between null and empty Data values in non-list contexts, it doesn't currently provide any way to distinguish between null and empty Data values when the Data value is in a List.

So your workaround of interpreting empty Data values as null CTransactionRef is probably the best approach for now. I put together an implementation that should be more robust in b147783 (tag). It limits the workaround to CTransactionRef data fields only and avoids making any assumptions in type-pointer.h about the type of pointers. It also includes a unit test that makes sure null values can be passed correctly.

@Sjors Sjors force-pushed the 2025/12/getrawtxs branch from ab851fc to 0ce5aba Compare December 8, 2025 08:40
@Sjors
Copy link
Member Author

Sjors commented Dec 8, 2025

@ryanofsky thanks, I swapped in your commit (subtree linter is still expected to fail).

So your workaround of interpreting empty Data values as null CTransactionRef is probably the best approach for now.

One reason I have some confidence in this approach is that the Python functional tests are able to read the result. Although it interprets the empty fields as b'' rather than None, that shouldn't be an issue for client developers. This is captured by your note:

//! [*] The Cap'n Proto wire format actually does distinguish between null and
//! empty Data values inside Lists, and the C++ API does allow distinguishing
//! between null and empty Data values in other contexts, just not the List
//! context, so this limitation could be removed in the future.

If this is fixed one day and we correctly encode null on the wire, it would technically be a breaking API change, but trivial to deal with for clients. E.g. the functional test would check for both b'' and None.

b147783 looks good to me, can you make a libmultiprocess PR, and another PR here that bumps the subtree to use it and add the tests? I'll rebase on that.

@DrahtBot
Copy link
Contributor

🐙 This pull request conflicts with the target branch and needs rebase.

@Sjors
Copy link
Member Author

Sjors commented Dec 19, 2025

Needs rebase after #34003, but I'll do that after there's a multiprocess PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants