Add key origin support to descriptors #14150

sipa · 2018-09-04T19:03:30Z

This adds support for key origin information to the descriptor parser, and exposes the resulting key path information through FlatSigningProvider.

There is no observable functionality from this right now, except having the scantxoutset RPC accept descriptors that include key origin information.

Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors).

src/script/descriptor.cpp

jonasschnelli · 2018-09-06T20:24:59Z

utACK e11d7ea24b7c67793ab8e84b5ad05d233f43e7da

Sjors

Concept ACK.

For me this would be easier to review when combined with a practical usage, like creating a PSBT with the origin info still in it. That could just be a WIP PR as it might involve quite a few changes, since descriptors currently are only used with scantxoutset.

src/test/descriptor_tests.cpp

src/script/sign.h

src/script/descriptor.cpp

Sjors · 2018-09-13T19:12:19Z

Perhaps @achow101 can use this in #14021?

achow101 · 2018-09-13T23:53:43Z

@Sjors I don't think this would be useful there. Instead we should have some new command that lets us import descriptors into the wallet.

practicalswift · 2018-09-25T19:31:43Z

src/script/descriptor.cpp

2018-09-25 20:23:55 clang(pr=14150): script/descriptor.cpp:179:69: warning: implicit conversion changes signedness: 'int' to 'std::vector<unsigned int, std::allocator<unsigned int> >::value_type' (aka 'unsigned int') [-Wsign-conversion]

sipa · 2018-10-12T23:20:40Z

Rebased, and expanded documentation to include key origin information.

ryanofsky · 2018-10-16T19:06:05Z

Was there a problem with the last rebase? It seems like some changes that were previously made, like #14150 (comment), have been reverted.

meshcollider · 2018-10-16T21:46:05Z

Concept ACK, will wait for the rebase to be fixed before reviewing then

sipa · 2018-10-17T01:18:25Z

@ryanofsky Wow, indeed, thanks for pointing that out. I don't know how that happened.

I went through all comments again, and reapplied the fixed I made earlier where necessary. I've also expanded the documentation change a bit.

ryanofsky

utACK 3fdc704f06bb5618cb63c248b20447026509e00d. Left some comments below that aren't critical and you should feel free to ignore.

src/script/descriptor.cpp

ryanofsky · 2018-10-17T19:31:56Z

doc/descriptors.md

This syntax is slightly weird to me because if you're looking at an expression like:

fingerprint/path1:xpub/path2

The fingerprint:path1 part only describes xpub, not xpub/path2, but there's no logical precedence of : and / operators that would lead me to see:

(fingerprint/path1:xpub)/path2

instead of:

(fingerprint/path1):(xpub/path2)

Maybe this is an unimportant aesthetic objection, but I'd almost rather there be an explicit origin function like:

origin(fingerprint/path1, xpub)/path2

or

origin(xpub, fingerprint/path1)/path2

that at least would group things together correctly.

src/script/descriptor.cpp

sipa · 2018-10-17T21:12:57Z

@ryanofsky Addressed all your comments except the syntax issue, which we should probably discuss at the PR level rather than buried inside a code comment.

I agree that fpr/path:key/path doesn't perfectly map with intuition about precedence of operation. On the other hand, I'd prefer to keep "key operations" as pure syntax, separated from "script operations" which are functions.

After thinking a bit more about it, how do you feel about fpr/path[key]/path? That way the entire expression maintains the order of path elements, and it just looks like you're "clarifying" the actual key somewhere inside the path itself, rather than as a separate thing prefixed onto it.

An alternative which is somewhat easier to parse I think is [fpr/path]key/path, which can still be read as "([fpr/path]key)/path".

ryanofsky

utACK 71cbcd9da8b31cb7cbe19001df32c8e1c46a603b. Only changes since last review were suggested changes (replacing +4's, adding a few test cases and a comment).

I think I like both of the [fpr/path]key/path and fpr/path[key]/path syntaxes more than the current fpr/path:key/path syntax and more than my origin() suggestion, and I agree that the [fpr/path]key/path version is probably more understandable.

I might also propose a more generalized syntax for adding attributes to keys like:

key[origin=fpr/path, device=myhardwarewallet]/path

Which might have other uses like:

key[birthday=timestamp]

But this might be too heavyweight. Anyway, you've thought about this a lot more than I have and I think whatever syntax you like, including the current syntax, should be fine.

ryanofsky · 2018-10-17T22:24:52Z

Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors).

Could you explain how these use-cases benefit from having origin information embedded inside descriptor expressions? It seems like it might be simpler if the descriptor language only described how things should be signed, and if key information was stored (or passed) in a separate a key id -> key metadata mapping.

sipa · 2018-10-17T22:27:48Z

@ryanofsky A PSBT updater needs to add key origin information to the PSBT file, or future signers may not be able to find the key to sign with.

I really think this information needs to be part of descriptors for them to be generally useful. Just listing the public keys is useless if those using the information don't know how to find the private key corresponding to said public key. In a way, that derivation information should from our point of view be treated as part of the public information about that key.

Another argument is so that you can 'specialize' a ranged descriptor to an individual one (like what may be desirable for #14503), without losing information. With origin support you can go from xpub/a/b/c/* to [fpr:a/b/c/i]pubkey when specializing for position i.

sipa · 2018-10-17T22:52:56Z

Switched to the [fpr/path]key/path notation, updating the parser, serializer, tests, and documentation.

As far as other attributes go, I'm not sure descriptors are the right place (this may be getting philosophical, though):

a timestamp is more a descriptor-level property than a key specific thing (you care about when any output generated by the descriptor can be used at the earliest, not differences in timestamps between the individual keys inside one).
the name of a hardware device is not portable (if included in descriptors, you wouldn't be able to copy the descriptor to another software/system that knows the hw device by a different name)

My preference is keeping those two as metadata on the descriptor (together with things like gap limit/range, whether the outputs are treated as change, ...). The same could be done with key origin information, but it feels more fundamental to not lose that information.

If there are generally useful properties of keys or path elements later we can still adopt such a syntax, though; it looks pretty readable.

instagibbs · 2018-10-18T16:16:02Z

Oops, I reviewed 2 days ago and didn't refresh this page to see updates!

instagibbs

utACK

doc/descriptors.md

instagibbs · 2018-10-18T16:34:03Z

test/functional/rpc_scantxoutset.py

        assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/1/1/1500)"])['total_amount'], Decimal("16.384"))
        assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/0)"])['total_amount'], Decimal("4.096"))
-        assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/1)"])['total_amount'], Decimal("8.192"))
+        assert_equal(self.nodes[0].scantxoutset("start", [ "combo([abcdef88/1/2'/3/4h]tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/1)"])['total_amount'], Decimal("8.192"))


Could there be a comment to point out what it's doing? As a reader of the test this is basically lost in the pile.

Not sure what you're asking. I've added 2 lines of comment on this block of tests.

Or is it specifically about this including of the key origin in the test? It has no effect, as for now key origins are not observable anyway. So this is just testing that you can in fact specify one, and nothing breaks.

I meant to a reader of the entire test itself, not the diff.

Ok, is what I added sufficient?

sipa · 2018-10-18T23:13:37Z

Addressed further comments in separate commits, will squash when ready.

instagibbs · 2018-10-18T23:29:27Z

re-ACK

meshcollider

utACK f4a8ccb

src/script/descriptor.cpp

doc/descriptors.md

ryanofsky · 2018-10-19T16:18:23Z

utACK 91cff98264ba5c6a86ca5d18b2591f759cffe793, just documentation fixes and two variable cleanups

DrahtBot · 2018-10-20T10:29:27Z

Reviewers, this pull request conflicts with the following ones:

test: Add linter to make sure single parameter constructors are marked explicit #14505 (Make all single parameter constructors explicit (C++11) by practicalswift)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

sipa · 2018-10-21T01:34:36Z

Squashed fixups, no tree changes.

sipa · 2018-10-21T03:32:52Z

Rebased afa9224b14 -> 8afb166

instagibbs · 2018-10-22T13:51:01Z

utACK 8afb166

ryanofsky

utACK 8afb166. Only change since last review is rebase/squash, resolving a merge conflict with #14161 in the documentation

8afb166 Update documentation to incude origin information (Pieter Wuille) ff37459 Add tests for key origin support (Pieter Wuille) 2c6281f Add key origin support to descriptors (Pieter Wuille) Pull request description: This adds support for [key origin](https://gist.github.com/sipa/e3d23d498c430bb601c5bca83523fa82#key-origin-identification) information to the descriptor parser, and exposes the resulting key path information through `FlatSigningProvider`. There is no observable functionality from this right now, except having the `scantxoutset` RPC accept descriptors that include key origin information. Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors). Tree-SHA512: 399828127b2e90a2f32d81ecc30a8a9261d08f4182d5d1744f05e46b25fde1bd383c54835b0820ca668e7d17353fa92c0fb2987e211ce269e0824c9395d210c2

Summary: 8afb166875 Update documentation to incude origin information (Pieter Wuille) ff37459abc Add tests for key origin support (Pieter Wuille) 2c6281f180 Add key origin support to descriptors (Pieter Wuille) Pull request description: This adds support for [key origin](https://gist.github.com/sipa/e3d23d498c430bb601c5bca83523fa82#key-origin-identification) information to the descriptor parser, and exposes the resulting key path information through `FlatSigningProvider`. There is no observable functionality from this right now, except having the `scantxoutset` RPC accept descriptors that include key origin information. Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors). Tree-SHA512: 399828127b2e90a2f32d81ecc30a8a9261d08f4182d5d1744f05e46b25fde1bd383c54835b0820ca668e7d17353fa92c0fb2987e211ce269e0824c9395d210c2 Backport of Core [[bitcoin/bitcoin#14150 | PR14150]] Test Plan: ninja ninja check ninja check-functional Reviewers: O1 Bitcoin ABC, #bitcoin_abc, deadalnix Reviewed By: O1 Bitcoin ABC, #bitcoin_abc, deadalnix Subscribers: deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6006

laanwj added the RPC/REST/ZMQ label Sep 6, 2018

promag reviewed Sep 6, 2018

View reviewed changes

src/script/descriptor.cpp Outdated Show resolved Hide resolved

src/script/descriptor.cpp Outdated Show resolved Hide resolved

sipa force-pushed the 201807_minedesc_origin branch from e6727aa to e11d7ea Compare September 6, 2018 16:45

Sjors reviewed Sep 7, 2018

View reviewed changes

src/test/descriptor_tests.cpp Outdated Show resolved Hide resolved

src/test/descriptor_tests.cpp Outdated Show resolved Hide resolved

practicalswift reviewed Sep 9, 2018

View reviewed changes

src/test/descriptor_tests.cpp Outdated Show resolved Hide resolved

practicalswift reviewed Sep 9, 2018

View reviewed changes

src/script/sign.h Outdated Show resolved Hide resolved

practicalswift reviewed Sep 9, 2018

View reviewed changes

src/script/descriptor.cpp Outdated Show resolved Hide resolved

practicalswift reviewed Sep 9, 2018

View reviewed changes

src/script/descriptor.cpp Outdated Show resolved Hide resolved

sipa force-pushed the 201807_minedesc_origin branch from e11d7ea to 78e5094 Compare September 10, 2018 01:36

practicalswift reviewed Sep 25, 2018

View reviewed changes

sipa force-pushed the 201807_minedesc_origin branch 2 times, most recently from 95897b6 to 2782910 Compare October 12, 2018 22:21

sipa force-pushed the 201807_minedesc_origin branch from 6053675 to 3fdc704 Compare October 17, 2018 01:17

Sjors mentioned this pull request Oct 17, 2018

Allow direct importing of xpubs #14449

Closed

matejcik mentioned this pull request Oct 17, 2018

scantxoutset does not identify UTXO origins #14503

Closed

ryanofsky approved these changes Oct 17, 2018

View reviewed changes

sipa force-pushed the 201807_minedesc_origin branch from 3fdc704 to 71cbcd9 Compare October 17, 2018 21:07

ryanofsky reviewed Oct 17, 2018

View reviewed changes

sipa force-pushed the 201807_minedesc_origin branch from 71cbcd9 to cfc8129 Compare October 17, 2018 22:43

instagibbs approved these changes Oct 18, 2018

View reviewed changes

meshcollider reviewed Oct 19, 2018

View reviewed changes

src/script/descriptor.cpp Outdated Show resolved Hide resolved

doc/descriptors.md Outdated Show resolved Hide resolved

sipa mentioned this pull request Oct 19, 2018

Add ability to convert solvability info to descriptor #14477

Merged

This was referenced Oct 20, 2018

test: Add linter to make sure single parameter constructors are marked explicit #14505

Merged

doc/descriptors.md tweaks #14161

Merged

sipa force-pushed the 201807_minedesc_origin branch from 91cff98 to afa9224 Compare October 21, 2018 01:34

sipa added 3 commits October 20, 2018 20:29

Add key origin support to descriptors

2c6281f

Add tests for key origin support

ff37459

Update documentation to incude origin information

8afb166

sipa force-pushed the 201807_minedesc_origin branch from afa9224 to 8afb166 Compare October 21, 2018 03:32

practicalswift mentioned this pull request Oct 22, 2018

Enable flake8 rule E225 which checks for missing whitespace around op… #14534

Closed

ryanofsky approved these changes Oct 22, 2018

View reviewed changes

sipa merged commit 8afb166 into bitcoin:master Oct 22, 2018

harding mentioned this pull request Nov 10, 2018

Doc: update release notes for changes since 0.17.0 branch #14688

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Add key origin support to descriptors #14150

Add key origin support to descriptors #14150

Uh oh!

Conversation

sipa commented Sep 4, 2018

Uh oh!

Uh oh!

Uh oh!

jonasschnelli commented Sep 6, 2018

Uh oh!

Sjors left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sjors commented Sep 13, 2018

Uh oh!

achow101 commented Sep 13, 2018

Uh oh!

practicalswift Sep 25, 2018

Choose a reason for hiding this comment

Uh oh!

sipa Oct 12, 2018

Choose a reason for hiding this comment

Uh oh!

sipa commented Oct 12, 2018

Uh oh!

ryanofsky commented Oct 16, 2018

Uh oh!

meshcollider commented Oct 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanofsky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanofsky Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sipa commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanofsky left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanofsky commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

instagibbs commented Oct 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

instagibbs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

instagibbs Oct 18, 2018

Choose a reason for hiding this comment

Uh oh!

sipa Oct 18, 2018

Choose a reason for hiding this comment

Uh oh!

meshcollider commented Oct 16, 2018 •

edited

Loading

sipa commented Oct 17, 2018 •

edited

Loading

sipa commented Oct 17, 2018 •

edited

Loading

ryanofsky left a comment •

edited

Loading

ryanofsky commented Oct 17, 2018 •

edited

Loading

sipa commented Oct 17, 2018 •

edited

Loading

sipa commented Oct 17, 2018 •

edited

Loading

instagibbs commented Oct 18, 2018 •

edited

Loading

DrahtBot commented Oct 20, 2018 •

edited

Loading

sipa commented Oct 21, 2018 •

edited

Loading