Skip to content

Conversation

@dexX7
Copy link
Contributor

@dexX7 dexX7 commented Mar 12, 2018

This pull request adds a flag "bip125-replaceable" to the mempool RPCs getrawmempool, getmempoolentry, getmempoolancestors and getmempooldescendants, which indicates whether an unconfirmed transaction might be replaced.

Initially the flag was added to the raw transaction RPCs, but thanks to @conscott, it was moved to the mempool RPCs, which actually have access to the mempool.

This pull request adds a flag "bip125-replaceable" to the RPCs "getrawtransaction" and "decoderawtransaction", which indicates, whether a transaction signals BIP 125 replaceability.

There was some discussion in #7817, whether showing replaceability in the UI could lead to the false assumption that transactions that don't signal BIP 125 are truely non-replaceable, but given that this PR tackles the raw transaction interface, which is a rather low level tool, I believe having this extra piece of information isn't bad.

@dexX7
Copy link
Contributor Author

dexX7 commented Mar 12, 2018

Please note, I copied SignalsOptInRBF() from policy/rbf.cpp into core_write.cpp.

This is probably not a good way to do, but it was done for now, because rbf.cpp isn't avaliable in LIBBITCOIN_COMMON. However, simply moving rbf.cpp from LIBBITCOIN_SERVER to LIBBITCOIN_COMMON doesn't work, because IsRBFOptIn() in rbf.cpp has mempool dependencies, which are also not available.

As alternative I could imagine to split rbf.cpp into something like rbf.cpp with SignalsOptInRBF(), which becomes part of LIBBITCOIN_COMMON and rbfstate.cpp with IsRBFOptIn(), which becomes part of LIBBITCOIN_SERVER?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo this should be the same key-value as the bip125-replaceable in the rpcwallet

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I updated the PR.

@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch 2 times, most recently from 5d7cb0f to e62c505 Compare March 12, 2018 13:55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SignalsOptInRBF is not bip125-replacable.

  • Confirmed transaction can never be replaced in the mempool
  • Unconfirmed transactions that depend on a mempool-transaction that signals opt in rbf can be replaced

Copy link
Contributor Author

@dexX7 dexX7 Mar 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you suggest to rename the flag to something like signals-rbf? Or rather do a stateful check, whether the transaction is actually replaceable, when the call is made?

The latter would probably be very useful, but given that TxToUniv isn't stateful and currently doesn't have any mempool relation, I don't see a straight forward way for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do this I'd say rename it to signals-rbf or such, which is also the name of the function already. And not make the raw transactions utility function decoding functionality depend on the mempool.

Copy link
Member

@maflcko maflcko Mar 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In which case I tend to Concept NACK. Just returning if the flag is set contradicts the bip125-replacability rules, which only adds confusion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to hijack.

I had considered adding a flag like this to the verbose output of getrawmempool. Assuming this idea were supported, based on your feedback @MarcoFalke - is it fair to say that you think it makes more sense to list a mempool transaction as replaceable (covering both explicit signaling and implicit based on ancestors) rather than just signals-rbf (just the explicit case)?

Do you think it would be appropriate to list that in getrawmempool?

@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch from e62c505 to 788885f Compare March 13, 2018 11:07
@dexX7 dexX7 changed the title Show "replaceable" flag, when decoding raw transactions Show "bip125-replaceable" flag, when retrieving mempool entries Mar 13, 2018
@dexX7
Copy link
Contributor Author

dexX7 commented Mar 13, 2018

@conscott brought up the idea of adding the flag to mempool entries and mempool RPCs. I really like the idea, because these calls are actually mempool aware and provide full access to the mempool and thus also allow to check, whether a transaction is really replaceable instead of simply signaling replaceablity.

I updated the PR accordingly, because I believe this is the way to go. @conscott if you want to submit this as PR, feel free to do so though, and I'll close this one.

Copy link
Member

@instagibbs instagibbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use brackets for multi-line conditionals

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, updated.

@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch from 788885f to eef1b43 Compare March 13, 2018 16:01
@instagibbs
Copy link
Member

re-ACK

Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK eef1b43

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test for unknown?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.. this is going to be a bit difficult: the "unknown" case is only triggered, if the transaction isn't found in the mempool.

In the case of the RPCs this path might even get removed, given that we iterate over mempool entries, which are indeed in the mempool and thus never "unknown".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a test might require mocking IsRBFOptIn in blockchain.cpp, but that seems overkill here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can just be an assert(bfState != RBF_TRANSACTIONSTATE_UNKNOWN). Looking at rbf.cpp, the unknown state should only occur if the transaction in question is not found in the mempool, and this function only processes transactions that are currently in the mempool (with mempool lock held).

If the above is correct (I believe it is), the bip125-replaceable field can just become a boolean.

Copy link
Member

@Sjors Sjors Mar 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of an assert here. It if it's really not possible, it's better to put the assert in rbf.cpp and remove this state from the enum. All this function should be concerned with is translating an enum to a string.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually: what about a situation where a descendant is missing from the mempool (because it somehow got evicted)?

Copy link
Contributor

@conscott conscott Mar 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have been more clear. In general it is possible to return RBF_TRANSACTIONSTATE_UNKNOWN from IsRBFOptIn(), however this call stack in particular (all mempool related rpc's) only calls IsRBFOptIn() on transactions we know to be in the mempool. So, the assert would belong here, not in rfb.cpp. I understand this still may not be the best option, I just wanted to make my language clear since my comment above was a bit ambiguous.

I am not sure I understand the problem/situation where 'a descendant is missing from the mempool'. Can you elaborate please?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of using assert in non-consensus critical code, given it's massive impact, if it's hit (e.g. by accident in the future etc.).

In the case of the RPCs "unknown" isn't triggered, because they iterate over mempool entries, which are inherently part of the mempool, but I kind of tend to leave it there for the potential case entryToJSON() is called in a different context. I'd be fine with removing this path though, too.

Actually: what about a situation where a descendant is missing from the mempool (because it somehow got evicted)?

Not an issue as far as I can see. We only care about ancestors and whether an unconfirmed ancestor signals RBF, which makes the chain replaceable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mempool.dat is sorted in dependency order, so ancestors always appear earlier. Further, the mempool is at all times internally consistent (when observable): all mempool transactions either spend a transaction output from the chainstate (so from a confirmed transaction), or another mempool output. There are also never any double spends in the mempool.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any opinion on listing unknown state @MarcoFalke ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion if the value of 'bip125-replacable' should be string (yes/no/unknown) or bool.

It seems entryToJSON can only be called when the entry is in the mempool (c.f. mempool.vTxHashes[e.vTxHashesIdx].first.ToString()). Thus, we can surely determine if it is replacable.

Instead of adding an assert, I'd prefer to throw some json exception that mentions that it never can be thrown.

I guess you could throw it in entryToJSON to make sure it is passed up to all potential callers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But does entryToJSON really care, whether it's in the mempool or not? I think this limits it's functionality in some way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entryToJSON specifically processes mempool transactions (CTxMemPoolEntry &e) - and I don't believe you should be able to have a CTxMemPoolEntry reference for something not currently in the mempool (it should guarantees this consistency), so in that sense, entryToJson should never be called in a "different context" (where the tx is not in mempool).

Anyhow, I am not trying to get pedantic on this point. I want this change (and requested it) and I am okay with the current implementation.

@conscott
Copy link
Contributor

conscott commented Mar 13, 2018

Concept ACK. Thanks for updating @dexX7 - just left a comment about the unknown state. Will test this out tomorrow.

@dexX7
Copy link
Contributor Author

dexX7 commented Mar 13, 2018

Regarding checking the "unknown" case: in the case of the RPCs this isn't triggered, because they iterate over mempool entries, which are inherently part of the mempool, but I kind of tend to leave it there for the potential case entryToJSON() is called in a different context. I'd be fine with removing this path though, too.

@meshcollider
Copy link
Contributor

Concept ACK

Copy link
Contributor

@conscott conscott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test ACK eef1b43b5860122bd7dc9270b5b5ba3f64b1dc0e

@dexX7
Copy link
Contributor Author

dexX7 commented Apr 24, 2018

Hey guys, just to clarify: given there were mixed reactions, do you insist on removing the branch for the "unknown" case and replace it with an exception or is this PR good to go as it is?

@conscott
Copy link
Contributor

@dexX7 - I was arguing over the unknown, but I am happy with it as is.

@maflcko
Copy link
Member

maflcko commented Apr 24, 2018

I just fail to see why it makes sense to burden rpc users with a third value that is never actually set.

@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch from eef1b43 to 5434aaa Compare April 25, 2018 05:37
@dexX7
Copy link
Contributor Author

dexX7 commented Apr 25, 2018

Alright, updated. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to encode a bool into a string, imo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good point.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch 2 times, most recently from e332b37 to 56301e0 Compare April 25, 2018 16:03
This affects getrawmempool, getmempoolentry, getmempoolancestors and getmempooldescendants.
@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch from 56301e0 to dc6f7eb Compare April 25, 2018 17:26
@dexX7 dexX7 force-pushed the rpc-raw-replaceable-flag branch from dc6f7eb to 870bd4c Compare May 22, 2018 06:23
bool rbfStatus = false;
RBFTransactionState rbfState = IsRBFOptIn(tx, mempool);
if (rbfState == RBFTransactionState::UNKNOWN) {
throw JSONRPCError(RPC_MISC_ERROR, "Transaction is not in mempool");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throwing here seems nasty. Just don't add a key?

Eg:

    switch (IsRBFOptIn(tx, mempool)) {
        case RBFTransactionState::FINAL:
            info.push_back(Pair("bip125-replaceable", false));
            break;
        case RBFTransactionState::REPLACEABLE_BIP125:
            info.push_back(Pair("bip125-replaceable", true));
            break;
        case RBFTransactionState::UNKNOWN:
            break;
    }

(Doing it this way also gets a compile-time warning if a new state is added later)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be nice to have this produce the same output as the wallet rpc calls that provide the same information (which appears to be "yes"/"no"/"unknown").

Copy link
Member

@achow101 achow101 Jul 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is even possible to hit this case since it requires the transaction to not be in the mempool (for the unknown state) yet every entry we are processing here comes from the mempool.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @achow101

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @sdaftuar

@achow101
Copy link
Member

achow101 commented Jul 9, 2018

utACK 870bd4c

@maflcko
Copy link
Member

maflcko commented Jul 10, 2018

utACK 870bd4c

@DrahtBot
Copy link
Contributor

The last travis run for this pull request was 61 days ago and is thus outdated. To trigger a fresh travis build, this pull request should be closed and re-opened.

@DrahtBot DrahtBot closed this Jul 22, 2018
@DrahtBot DrahtBot reopened this Jul 22, 2018
@laanwj laanwj added this to the 0.18.0 milestone Aug 7, 2018
@laanwj laanwj added the Feature label Aug 7, 2018
@laanwj
Copy link
Member

laanwj commented Aug 7, 2018

This seems mergeable, should do so after 0.17 branch as this is a feature

@laanwj laanwj merged commit 870bd4c into bitcoin:master Aug 25, 2018
laanwj added a commit that referenced this pull request Aug 25, 2018
… entries

870bd4c Update functional RBF test to check replaceable flag (dexX7)
820d31f Add "bip125-replaceable" flag to mempool RPCs (dexX7)

Pull request description:

  This pull request adds a flag "bip125-replaceable" to the mempool RPCs getrawmempool, getmempoolentry, getmempoolancestors and getmempooldescendants, which indicates whether an unconfirmed transaction might be replaced.

  Initially the flag was added to the raw transaction RPCs, but thanks to @conscott, it was moved to the mempool RPCs, which actually have access to the mempool.

  ~~This pull request adds a flag "bip125-replaceable" to the RPCs "getrawtransaction" and "decoderawtransaction", which indicates, whether a transaction signals BIP 125 replaceability.~~

  There was some discussion in #7817, whether showing replaceability in the UI could lead to the false assumption that transactions that don't signal BIP 125 are truely non-replaceable, but given that this PR tackles the raw transaction interface, which is a rather low level tool, I believe having this extra piece of information isn't bad.

Tree-SHA512: 1f5511957af2c20a9a6c79d80a335c3be37a2402dbf829c40cceaa01a24868eab81a9c1cdb0b3d77198fa3bb82799e3540a5c0ce7f35bbac80d73f7133ff7cbc
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.