Skip to content

Conversation

@xdustinface
Copy link

@xdustinface xdustinface commented Aug 7, 2020

While the idea of respecting fees in calculations brought up in #3588 and #3640 makes totally sense i still noticed while reviewing/testing the latter that the estimation of just saying use min fee for 148 bytes for inputs and min fee for 34 bytes for outputs and then sum it up isn’t good enough imo.

  1. It doen’t respect the general tx overhead
  2. It introduces rounding errors with the increasing numbers of inputs/outputs

So even with those estimations we end up in cases where the calculations are slightly off from the expected. This brings inaccuracy to transaction generation and leads to issues (e.g. the. one in TransactionRecord::decomposeTransaction).

As solution this PR introduces CTransactionBuilder in 13fa395eeb0050225ce2f1a77418d1af96137b0e, a class which wraps transaction creation and allows a simple and failproof creation of transactions with inputs defined by CompactTallyItem while also respecting the logics implemented in CWallet::CreateTransaction. For output generation it takes care of key generation, provides methods e.g. to add/try to add outputs, get the remaining amount available, and methods to create/commit the transaction to the wallet. To avoid running into the issues described above it does not calculate a seperate fee for the amount of bytes for input/outputs like it was before. Instead it uses bytes for all calculations and calculates a total fee for the transaction when required. This way it’s possible now to calculate the correct amounts/fees used for the created transactions.

Further on top this PR brings also some small fixes/modifications to the logic behind CreateDenominated and MakeCollateralAmounts in fba14a1 and 8e0191c so please review very carefully there that we can be sure i didn’t mess something here :)

3d00601 finally makes sure #3597 is not longer an issue. As it now should cover all the cases produced by MakeCollateralAmounts correctly.

@UdjinM6
Copy link

UdjinM6 commented Aug 9, 2020

Pls see https://github.com/UdjinM6/dash/commits/pr3657

tl;dr:

  1. 209f43f should be reverted cause it violates discard/dustrelay fee definitions
  2. I think you messed up "finished" conditions (39a60562aa95ef2a4dbd6a49e00c2cc0f61957a5)
  3. The rest is just some refactoring/cleanups.

@xdustinface
Copy link
Author

  1. 209f43f should be reverted cause it violates discard/dustrelay fee definitions

Isn't a partial revert like in 1365db1 enough? This way we can still say 100% sure that the same feerate is used for change dust consideration.

  1. I think you messed up "finished" conditions (39a6056)

Ops, good catch!

  1. The rest is just some refactoring/cleanups.

Picked them all 👍

@xdustinface xdustinface force-pushed the pr-privatesend-transaction-builder branch from ab18ddd to 7502fcb Compare August 11, 2020 01:32
@xdustinface
Copy link
Author

@UdjinM6 While reviewing your suggestions more precisely i realized that your refactoring of GetAmountUsed() in 2b01a09 introduced a rounding bug. Reason it didn't respect fees in the beginning was to always have only one fee calculation with the total number of bytes. With it respecting fees you need to call GetFee() twice with subtotals in TryAddOutput and TryAddOutputs https://github.com/dashpay/dash/blob/2b01a09c6e8263095f316961a62b8bfa514ec487/src/privatesend/privatesend-util.cpp#L169-L172 First call is in GetAmountLeft() which calls GetAmountUsed() https://github.com/dashpay/dash/blob/2b01a09c6e8263095f316961a62b8bfa514ec487/src/privatesend/privatesend-util.cpp#L198-L206 This twice calling GetFee() results in a rounding error due to integer cutoffs here https://github.com/dashpay/dash/blob/2b01a09c6e8263095f316961a62b8bfa514ec487/src/policy/feerate.cpp#L28

Lets assume GetBytesTotal() == 200 and nBytesOutput == 34 and the feerate 1234 duff/KB

With 2b01a09 applied:
1234*200/1000 = 246,8 (246 in integer world)
1234*34/1000 = 41,956 (41 in integer world)
= 287 duffs fee

With the initial commits:
1234*234/1000 = 288,756 (288 in integer world)
= 288 duffs fee

With the force push i dropped 2b01a09, 7c30d5d, f90e626 again. The latter two because they bring no drawbacks but IMO without 2b01a09 it just brings more readability to keep them.. still feel free to complain again if you prefer to have them dropped for some reason. There are also the new commits 94529da, 3cb7d2d and 7502fcb

@UdjinM6
Copy link

UdjinM6 commented Aug 11, 2020

Makes sense. Still don't see a reason for CoinControl::m_discard_feerate though abbb195ff5.

@xdustinface
Copy link
Author

Makes sense. Still don't see a reason for CoinControl::m_discard_feerate though abbb195.

Okaaay... just one more time a bit more detailed in case you didn't get the reasoning i have in mind for it: I can't say for sure that there is no chance for the code down in the deeps of GetDiscardRate() to give different results when it's called twice with some time elapsed in between, can you? So for me thats the reason to have CCoinControl::m_discard_rate as it makes sure we still use the same feerate in both cases (CTransactionBuilder::IsDust and CWallet::CreateTransaction). What may happen if the results are different is that CTransactionBuilder::Commit fails. Sure, then it tries it one more time but what if there is only one utxo left with a dust change, this continues to happen and the mixing starts to stuck because of it? I know, it sounds not very likely 😆 but at the end it's a potential unexpected scenario we could avoid with few lines of code. I rather prefer to prevent such edge cases if i know about them. But if you are sure that can't happen or if you think this is an acceptable case just give me your 👍 and i will drop it.

Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of nits

@UdjinM6
Copy link

UdjinM6 commented Aug 11, 2020

Oh, by 👍 I meant that it makes sense and I understand the reason now 🙈 pls ignore my previous suggestion and drop 5fb050b

@xdustinface
Copy link
Author

Oh, by 👍 I meant that it makes sense and I understand the reason now 🙈 pls ignore my previous suggestion and drop 5fb050b

Ahhh ops, i took it as the "go drop it thumbs up" 😅

@xdustinface xdustinface force-pushed the pr-privatesend-transaction-builder branch from 5fb050b to 7502fcb Compare August 11, 2020 11:30
@UdjinM6
Copy link

UdjinM6 commented Aug 13, 2020

Pls see https://github.com/UdjinM6/dash/commits/pr3657 (starting with 703922203ecfa8f18a2c0c9ed853d91ab7c3fc368b19c551fd85840aeadeaf64f04ffe655cd6072f)

EDIT: tl;dr: change outputs should be handled automatically, size calculations need to be more precise, some minor tweaks (off-by-one, types, logs, comments)

@xdustinface
Copy link
Author

xdustinface commented Aug 13, 2020

Overall nice suggestions but im not a big fan of 703922203ecfa8f18a2c0c9ed853d91ab7c3fc36 yet, convince me :D

Why does it matter in this case if the change is not handled automatically? The idea of doing it manually was to have two outputs with the same amount in this case as this imo reduces potential false positive detection even more for the "make collateral amounts tx type".

Edit: Also there is no "Make sure its not denominated" anymore for the change if its done automatically as the amount is not set through getAdjustedAmount

@UdjinM6
Copy link

UdjinM6 commented Aug 14, 2020

Why does it matter in this case if the change is not handled automatically?

Added f6f0f8ac72. Try mixing with --paytxfee=0.001 for example and you'll see that it fails without 7039222 - when feerate is much greater than discard feerate, wallet tries to create new change output in such cases and fails with "not enough funds".

Also there is no "Make sure its not denominated" anymore for the change if its done automatically as the amount is not set through getAdjustedAmount.

Right, it's not really used after 7039222 (wasn't ever needed in case3, the smallest denom is 10x larger than GetCollateralAmount) and it can be dropped together with UpdateAmount ced2939491.

@xdustinface
Copy link
Author

Nice catch with the extra byte due to the compact serialize size 👍 I added some commits related to that to wrap it up a bit and to also respect this when trying to add outputs in TryAddOutput and TryAddOutputs. Let me know what you think.

8b19c551fd85840aeadeaf64f04ffe655cd6072f is also a good catch! But i modified it a bit in 05ee734, thoughts?

And i picked the others except 703922203ecfa8f18a2c0c9ed853d91ab7c3fc36. But i will have a better look at it and your points again later if i get the chance or tomorrow!

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a limited code review, didn't review everything.

Big takeaway so far:
I'm not a huge fan of adding ~320 LOC, but that's probably okay.
CTransactionBuilder really needs a good amount of unit tests

CTransactionBuilderOutput* out = txBuilder.AddOutput();
out->UpdateAmount(getAdjustedAmount(txBuilder.GetAmountLeft()));

assert(CPrivateSend::IsCollateralAmount(out->GetAmount()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These asserts kindof scare me tbh...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why exactly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could imagine a situation where somehow something gets changed in such a way that it's too much or too little to pass this assert, and this isn't caught in testing. I would rather log an error, and return false than assert and crash the application. You could maybe use something like

bool AbortNode(const std::string& strMessage, const std::string& userMessage="")
if you want to somewhat gracefully shutdown the application

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts @UdjinM6 ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@UdjinM6 I would like your thoughts here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm imo its nothing to worry about, we have unit tests and

if (!txBuilder.CouldAddOutput(CPrivateSend::GetCollateralAmount())) {
} else if (txBuilder.CouldAddOutputs({CPrivateSend::GetCollateralAmount(), CPrivateSend::GetCollateralAmount()})) {

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo it's ok

Comment on lines +204 to +214
CAmount CTransactionBuilder::GetAmountLeft(const CAmount nAmountInitial, const CAmount nAmountUsed, const CAmount nFee)
{
return nAmountInitial - nAmountUsed - nFee;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the point of this function at this point is it's just a glorified subtraction function...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jup its just a subtraction helper. https://github.com/dashpay/dash/blob/e7dc9da05c9e6cf6be4173ad65aecbc611cd1992/src/privatesend/privatesend-util.h#L119 imo it improves the readability of e.g.

https://github.com/dashpay/dash/blob/e7dc9da05c9e6cf6be4173ad65aecbc611cd1992/src/privatesend/privatesend-util.cpp#L169-L174

As it wraps the subtrahends in GetAmountLeft() so you can just see by reading the wrappers name what the point of the return is. I don't see any drawback by doing it like this as the compiler should be smart enough to take care of it, but i neither will fight for keeping it lol.. so if you see any problems with this, just complain again.

Comment on lines +298 to +309

fKeepKeys = true;

strResult = wtx.GetHash().ToString();

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say remove a few of these empty lines, probably line 300 at least

@xdustinface
Copy link
Author

Added f6f0f8a, ee0e3ee, e42c43f
Dropped 8d4dcd2 (moved it into a4620e4), 6801983 (not longer needed if #3668 gets merged), part of dfebc25

@UdjinM6 are you good now with #3668 instead of 7039222?

@xdustinface
Copy link
Author

Rebased on develop + e755205

UdjinM6
UdjinM6 previously approved these changes Aug 26, 2020
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be working as expected (together with #3668 )

ACK

@xdustinface xdustinface force-pushed the pr-privatesend-transaction-builder branch from b857b1c to 7814ff3 Compare August 29, 2020 19:05
@xdustinface
Copy link
Author

see https://github.com/PastaPastaPasta/dash/commits/pr_3657

Picked 87b7431, 2543a4c and bcdcb13 (squashed the 3 typo commits) 👍

About the others:

71d49f9bd57dc0937358001f5638f1c506b114dc and 6c34988801d640a91d624a3ab8cc7ae70ab0cf58: does it really bring any improvement / make a difference to use explicit casts there? IMO this just blows it up unnecessarily. We will never reach anything near INT_MAX in any of those cases, do we?

eaeb50e7a61dd6125f9cde76977e76770fed29be: why?

@PastaPastaPasta
Copy link
Member

PastaPastaPasta commented Aug 29, 2020

see https://github.com/PastaPastaPasta/dash/commits/pr_3657

Picked 87b7431, 2543a4c and bcdcb13 (squashed the 3 typo commits) +1

About the others:

71d49f9 and 6c34988: does it really bring any improvement / make a difference to use explicit casts there? IMO this just blows it up unnecessarily. We will never reach anything near INT_MAX in any of those cases, do we?

eaeb50e: why?

eaeb50e resolves a clang-tidy warning

71d49f9#diff-7d55c5a40c548772705e07a8dff7a49eR188-R189 L189 specifically resolves a clang-tidy, and the rest imo it's good standard to validate when going from unsigned int -> int or long -> int or really any narrowing conversion, that we aren't having something weird happen. I think the explicit cast on L249 also resolves clang-tidy warning

6c34988 resolves clang-tidy warning

@xdustinface
Copy link
Author

Ok, seems like im just not a big fan of some of those clang-tidy recommendations lol.. not the first time i noticed that haha.. and i really wonder if we should always follow them.. especially eaeb50e is one i can't understand at all 😆 anyway, they don't do anything bad so i just picked them.

@PastaPastaPasta
Copy link
Member

Ok, seems like im just not a big fan of some of those clang-tidy recommendations lol.. not the first time i noticed that haha.. and i really wonder if we should always follow them.. especially eaeb50e is one i can't understand at all laughing anyway, they don't do anything bad so i just picked them.

see https://stackoverflow.com/a/46292715

@xdustinface
Copy link
Author

see https://stackoverflow.com/a/46292715

It's not that i didn't get that it doesn't make a difference :D I just don't see the point why clang-tidy prefers to introduce inconsistencies between declaration and definition. I mean

static CAmount GetAmountLeft(CAmount, CAmount, CAmount);

also works, but you also don't leave variable names out of the declaration only because they are ignored by the compiler and they don't have an impact on the definition, or is this also something clang-tidy recommends?

@PastaPastaPasta
Copy link
Member

see https://stackoverflow.com/a/46292715

It's not that i didn't get that it doesn't make a difference :D I just don't see the point why clang-tidy prefers to introduce inconsistencies between declaration and definition. I mean

static CAmount GetAmountLeft(CAmount, CAmount, CAmount);

also works, but you also don't leave variable names out of the declaration only because they are ignored by the compiler and they don't have an impact on the definition, or is this also something clang-tidy recommends?

Seems like most of the thought process is the fact that something is const doesn't change how someone is going to call that function, whereas knowing the names is pretty valuable

UdjinM6
UdjinM6 previously approved these changes Aug 30, 2020
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@PastaPastaPasta
Copy link
Member

I think I'm ready to ACK once those comments become doxygen

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK, code review, as well as mixed a good bit on testnet

@UdjinM6 UdjinM6 merged commit a206af4 into dashpay:develop Sep 1, 2020
xdustinface added a commit to xdustinface/dash that referenced this pull request Sep 12, 2020
…CollateralAmounts (dashpay#3657)

* wallet: Add m_dust_feerate to CCoinControl / Use it in CreateTransaction

* privatesend: Introduce CTransactionBuilder/CTransactionBuilderOutput

Builder classes for transactions from type

Inputs: Defined by CompactTallyItem
Outputs: Simple outputs with lose reserve keys

This takes fully takes care of fee calculations and makes sure calculations are the same like those happening when actually create the transaction with CreateTransaction.

* privatesend: Improve amount/fee calculation in CreateDenominated

* privatesend: Improve amount/fee calculation in MakeCollateralAmounts

* qt: Fix decomposeTransaction's MakeCollateralAmounts detection

Align it with the three cases in CPrivateSendClientSession::MakeCollateralAmounts

* Refactor GetFee

The fee rate is always coinControl.m_feerate, also it's not used outside so should be a private method

* Simplify nBytesOutput calculations

* Drop unused GetBytesOutput()

* Make Clear(), GetBytesTotal() and GetAmountUsed() private

* Drop unused GetCoinControl()

* Make ToString() const

* Refactor `CTransactionBuilder::Commit()`

* Reorder cases in decomposeTransaction

* Fix "finished" conditions in CreateDenominated

* Fix typo

* wallet|privatesend: Refactor CCoinControl's m_dust_feerate -> m_discard_feerate

* privatesend: Drop unused member CTransactionBuilder::dustFeeRate

* privatesend: Improve CTransactionBuilder's readability

* privatesend: Make the static CTransactionBuilder::GetAmountLeft private

* wallet: Recover previous code style

* Update src/privatesend/privatesend-util.cpp

Co-authored-by: UdjinM6 <[email protected]>

* Tweak CTransactionBuilder to respect potential compact size diffs

* Tweak GetFee param type

* Trivial log/comments tweaks

* privatesend: Fix countPossibleOutputs

- Fix off by one issue
- Respect max outputs threshold

* privatesend: Use GetSizeOfCompactSizeDiff in CTransactionBuilder

* Apply suggestions from code review

Co-authored-by: PastaPastaPasta <[email protected]>

* privatesend: Rename TryAdd to CouldAdd in CTransactionBuilder

* wallet: Reset m_discard_feerate in CCoinControl::SetNull

* Respect `-paytxfee` and `settxfee`

* privatesend: Check for denominated amount only where really required

* qt: Remove obsolete IsCollateralAmount() checks

* privatesend: Don't accept negative amounts in CTransactionBuilder

* privatesend: Remove unused CConnman parameter

* use emplace_back instead of push_back

Signed-off-by: pasta <[email protected]>

* fix typos

Signed-off-by: pasta <[email protected]>

* make GetAmount const

Signed-off-by: pasta <[email protected]>

* privatesend: Explicit capture __func__ in needMoreOutputs lambda

* privatesend: Update CTransactionBuilder::UpdateAmount

* remove const on parameter in declaration

Signed-off-by: pasta <[email protected]>

* handle unsigned int -> int conversions a bit better

Signed-off-by: pasta <[email protected]>

* explicitly cast to int from unsigned int.

estimateSmartFee handles it if negative

Signed-off-by: pasta <[email protected]>

* Make CTransactionBuilderOutput::GetScript const

Co-authored-by: UdjinM6 <[email protected]>

* privatesend: Update comments to follow doxygen

* privatesend: Add class descriptions

Co-authored-by: UdjinM6 <[email protected]>
Co-authored-by: PastaPastaPasta <[email protected]>
Co-authored-by: pasta <[email protected]>
gades pushed a commit to cosanta/cosanta-core that referenced this pull request Mar 8, 2022
…CollateralAmounts (dashpay#3657)

* wallet: Add m_dust_feerate to CCoinControl / Use it in CreateTransaction

* privatesend: Introduce CTransactionBuilder/CTransactionBuilderOutput

Builder classes for transactions from type

Inputs: Defined by CompactTallyItem
Outputs: Simple outputs with lose reserve keys

This takes fully takes care of fee calculations and makes sure calculations are the same like those happening when actually create the transaction with CreateTransaction.

* privatesend: Improve amount/fee calculation in CreateDenominated

* privatesend: Improve amount/fee calculation in MakeCollateralAmounts

* qt: Fix decomposeTransaction's MakeCollateralAmounts detection

Align it with the three cases in CPrivateSendClientSession::MakeCollateralAmounts

* Refactor GetFee

The fee rate is always coinControl.m_feerate, also it's not used outside so should be a private method

* Simplify nBytesOutput calculations

* Drop unused GetBytesOutput()

* Make Clear(), GetBytesTotal() and GetAmountUsed() private

* Drop unused GetCoinControl()

* Make ToString() const

* Refactor `CTransactionBuilder::Commit()`

* Reorder cases in decomposeTransaction

* Fix "finished" conditions in CreateDenominated

* Fix typo

* wallet|privatesend: Refactor CCoinControl's m_dust_feerate -> m_discard_feerate

* privatesend: Drop unused member CTransactionBuilder::dustFeeRate

* privatesend: Improve CTransactionBuilder's readability

* privatesend: Make the static CTransactionBuilder::GetAmountLeft private

* wallet: Recover previous code style

* Update src/privatesend/privatesend-util.cpp

Co-authored-by: UdjinM6 <[email protected]>

* Tweak CTransactionBuilder to respect potential compact size diffs

* Tweak GetFee param type

* Trivial log/comments tweaks

* privatesend: Fix countPossibleOutputs

- Fix off by one issue
- Respect max outputs threshold

* privatesend: Use GetSizeOfCompactSizeDiff in CTransactionBuilder

* Apply suggestions from code review

Co-authored-by: PastaPastaPasta <[email protected]>

* privatesend: Rename TryAdd to CouldAdd in CTransactionBuilder

* wallet: Reset m_discard_feerate in CCoinControl::SetNull

* Respect `-paytxfee` and `settxfee`

* privatesend: Check for denominated amount only where really required

* qt: Remove obsolete IsCollateralAmount() checks

* privatesend: Don't accept negative amounts in CTransactionBuilder

* privatesend: Remove unused CConnman parameter

* use emplace_back instead of push_back

Signed-off-by: pasta <[email protected]>

* fix typos

Signed-off-by: pasta <[email protected]>

* make GetAmount const

Signed-off-by: pasta <[email protected]>

* privatesend: Explicit capture __func__ in needMoreOutputs lambda

* privatesend: Update CTransactionBuilder::UpdateAmount

* remove const on parameter in declaration

Signed-off-by: pasta <[email protected]>

* handle unsigned int -> int conversions a bit better

Signed-off-by: pasta <[email protected]>

* explicitly cast to int from unsigned int.

estimateSmartFee handles it if negative

Signed-off-by: pasta <[email protected]>

* Make CTransactionBuilderOutput::GetScript const

Co-authored-by: UdjinM6 <[email protected]>

* privatesend: Update comments to follow doxygen

* privatesend: Add class descriptions

Co-authored-by: UdjinM6 <[email protected]>
Co-authored-by: PastaPastaPasta <[email protected]>
Co-authored-by: pasta <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants