Skip to content

Conversation

@codablock
Copy link

@codablock codablock commented Sep 27, 2019

Limiting the orphan TX map by TX count is suboptimal because orphan counts can go up significantly for short periods of time. The old limit of 100 TXs resulted in a maximum memory usage of 10mb, based on the max size of TXs being 100kb. If we limit the map by the total TX size sum, we can store many thousand TXs in the same max memory.

@codablock codablock added this to the 14.1 milestone Sep 27, 2019
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 👍 Pls see a few suggestions below.

@codablock
Copy link
Author

Rebased on develop and applied suggestions

Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concept ACK, utACK. Any good way to test this? how would you suggest generating orphan txes 😂

Copy link

@nmarley nmarley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concept ACK

Test values seems a little high since the maxorphantx value wasn't adjusted, not sure if this is intentional.

def set_test_params(self):
self.num_nodes = 2
self.extra_args = [["-maxorphantx=1000"], ["-maxorphantx=1000", "-limitancestorcount=5"]]
self.extra_args = [["-maxorphantxsize=1000"], ["-maxorphantxsize=1000", "-limitancestorcount=5"]]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1000 MB would be 1 GB, right? Does this not seem a little high (even for this test script)?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is a "little" high, but 1000 orphan TX count was also high. The intent here was most likely to just disable the limit and keeping the number 1000 makes it easier later when conflicts arise (no need to think about that number)

self.add_nodes(3, extra_args=[["-maxorphantx=1000", "-whitelist=127.0.0.1"],
["-blockmaxsize=17000", "-maxorphantx=1000"],
["-blockmaxsize=8000", "-maxorphantx=1000"]])
self.add_nodes(3, extra_args=[["-maxorphantxsize=1000", "-whitelist=127.0.0.1"],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as above.

@codablock codablock merged commit a8fa5cf into dashpay:develop Sep 30, 2019
@codablock codablock deleted the pr_maxorphantxsize branch September 30, 2019 13:34
codablock added a commit to codablock/dash that referenced this pull request Nov 19, 2019
MIPPL pushed a commit to biblepay/biblepay that referenced this pull request Nov 24, 2019
* commit '8b14adb206d76fbc6307385999a1c512052e93fa': (21 commits)
  [v0.14.0.x] Update release notes with change log (dashpay#3213)
  [v0.14.0.x] Bump version to 0.14.0.4 and draft release notes (dashpay#3203)
  Circumvent BIP69 sorting in fundrawtransaction.py test (dashpay#3100)
  Fix compile issues
  Merge bitcoin#11397: net: Improve and document SOCKS code
  Slightly optimize ApproximateBestSubset and its usage for PS txes (dashpay#3184)
  Update activemn if protx info changed (dashpay#3176)
  Actually update spent index on DisconnectBlock (dashpay#3167)
  Only track last seen time instead of first and last seen time (dashpay#3165)
  Merge bitcoin#17118: build: depends macOS: point --sysroot to SDK (dashpay#3161)
  Simulate BlockConnected/BlockDisconnected for PS caches
  Few fixes related to SelectCoinsGroupedByAddresses (dashpay#3144)
  Various fixes for mixing queues (dashpay#3138)
  Also consider txindex for transactions in AlreadyHave() (dashpay#3126)
  Ignore recent rejects filter for locked txes (dashpay#3124)
  Make orphan TX map limiting dependent on total TX size instead of TX count (dashpay#3121)
  Update/modernize macOS plist (dashpay#3074)
  Fix bip69 vs change position issue (dashpay#3063)
  Partially revert 3061 (dashpay#3150)
  Fix SelectCoinsMinConf to allow instant respends (dashpay#3061)
  ...

# Conflicts:
#	configure.ac
#	doc/man/biblepay-cli.1
#	doc/man/biblepay-qt.1
#	doc/man/biblepay-tx.1
#	doc/man/biblepayd.1
#	doc/release-notes.md
#	src/clientversion.h
#	src/wallet/wallet.cpp
PastaPastaPasta added a commit that referenced this pull request Aug 4, 2025
, bitcoin#25683, bitcoin#23679, bitcoin#26216, bitcoin#26295, bitcoin#26513, bitcoin#26515, bitcoin#26727, bitcoin#25374, bitcoin#25272, bitcoin#26408, bitcoin#25112, bitcoin#26702, bitcoin#25932, bitcoin#26822 (auxiliary backports: part 27)

f45de22 merge bitcoin#26822: don't allow past absolute timestamp in `setban` (Kittywhiskers Van Gogh)
920e85d test: adjust mocktime bump to match upstream in `p2p_disconnect_ban` (Kittywhiskers Van Gogh)
b1a95d8 merge bitcoin#25932: Simplify backtrack logic (Kittywhiskers Van Gogh)
eff27a4 merge bitcoin#26702: drop unused `FindWalletTx` parameter and rename (Kittywhiskers Van Gogh)
62998f3 merge bitcoin#25112: Move error message formatting of NonFatalCheckError to cpp (Kittywhiskers Van Gogh)
27db044 merge bitcoin#26408: Remove spam from debug log (Kittywhiskers Van Gogh)
12acde7 merge bitcoin#25272: guard and alert about a wallet invalid state during chain sync (Kittywhiskers Van Gogh)
385306d merge bitcoin#25374: remove unused `create_confirmed_utxos` helper (Kittywhiskers Van Gogh)
a8d29f9 merge bitcoin#26727: remove optional from fStateStats fields (Kittywhiskers Van Gogh)
a0701e9 merge bitcoin#26515: skip getpeerinfo for a peer without CNodeStateStats (Kittywhiskers Van Gogh)
c9eedc8 merge bitcoin#26513: Make static nLastFlush and nLastWrite Chainstate members (Kittywhiskers Van Gogh)
859d64e txorphanage: adapt block-based orphan reprocessing to refactor (Kittywhiskers Van Gogh)
4172102 merge bitcoin#26295: Replace global g_cs_orphans lock with local (Kittywhiskers Van Gogh)
3665919 fix: protect `m_orphan_tx_size` with `g_cs_orphans` (Kittywhiskers Van Gogh)
f12e6cc fix: scan outputs in `GetCandidatesForBlock`, add regression test (Kittywhiskers Van Gogh)
42aa8a7 merge bitcoin#26216: Limit outpoints.size in txorphan target to avoid OOM (Kittywhiskers Van Gogh)
b376c09 merge bitcoin#23679: Sanitize port in `addpeeraddress()` (Kittywhiskers Van Gogh)
cd6c2b5 merge bitcoin#25683: log `nEvicted` message in LimitOrphans then return void (Kittywhiskers Van Gogh)
6cd09ef merge bitcoin#25641: Fix `-Wparentheses` gcc warning (Kittywhiskers Van Gogh)
bc69f80 merge bitcoin#25624: Fix assert bug in txorphan target (Kittywhiskers Van Gogh)
fc68188 merge bitcoin#25447: add low-level target for txorphanage (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional information

  * Since [dash#3121](#3121), Dash has limited the orphan transaction map by the accumulated size of all transactions in the map instead of transaction count (like upstream). [bitcoin#25447](bitcoin#25447) introduces a new fuzzing harness that operates on upstream assumptions.

    To bridge the gap, `DEFAULT_MAX_ORPHAN_TRANSACTIONS` has been defined in the harness to mean the highest transaction count _assuming_ each transaction is of maximum size, which means the effective worst-case count is 100 transactions. This is already documented in `net_processing.h` ([source](https://github.com/dashpay/dash/blob/bd589dd6f434c0c6cc8f0c3e2cf299785bad6d07/src/net_processing.h#L31-L32)) but has also been explicitly documented for the harness ([source](https://github.com/dashpay/dash/blob/fc68188e3227dc0912ef3c508f6d45ff1d7614ee/src/test/fuzz/txorphan.cpp#L27-L28)).

  * Dash introduced an additional trigger for orphan reprocessing in [dash#3139](#3139) to allow for faster processing when parents appear in a block. Currently, the implementation scans through the _inputs_ of each transaction to check against the outpoint map ([source](https://github.com/dashpay/dash/blob/86071f9df4d052de6f1df64a5d269a67a6756d29/src/txorphanage.cpp#L186-L192)).

    This does not seem to be intended behavior as it's using the input index to search through outpoints when it should be iterating through outpoints like `AddChildrenToWorkSet()` ([source](https://github.com/dashpay/dash/blob/86071f9df4d052de6f1df64a5d269a67a6756d29/src/txorphanage.cpp#L148C19-L159)). Though despite this, the code still worked _if_ either of the following conditions were met.

    * There are an even number of inputs and outputs (in which case a match is found as if iterating through the set of outputs); or
    * There are more inputs than outputs (in which case a match is found even if all `find()`s `i > vout.size()` are wasted operations)

    The test case introduced in [dash#3139](#3139) met this criteria and passed though the code would _not_ work as expected if

    * There are more outputs than inputs (in which case the loop will bail prematurely as `vin.size() > vout.size()` and we're scanning `vins` for outpoint positions)

    This has been rectified by modifying `GetCandidatesForBlock()` to use `AddChildrenToWorkSet()` and adding a unit test to document this behavior.

  * [bitcoin#26295](bitcoin#26295) does a substantial refactor of the transaction "orphanage" and part of that refactor is solidifying `ProcessOrphanTx()` as dealing with `Peer`s (as it's located in `net_processing` and not available through the interface). This conflicts with the capability introduced in [dash#3139](#3139) as there is no peer to attribute it to, which is relevant as the newer code introduces a map between peer ID and the orphan work map ([source](https://github.com/bitcoin/bitcoin/blob/7082ce3e88d77456d60a5a66bd38807fbd66f311/src/txorphanage.h#L75-L76)).

    This has been worked around by using the ID `-1` to mean block connection triggered orphan work ([source](https://github.com/dashpay/dash/blob/859d64e98fc1b5bf613f526ef76e5fc66851fe13/src/txorphanage.cpp#L204-L209), [source](https://github.com/dashpay/dash/blob/859d64e98fc1b5bf613f526ef76e5fc66851fe13/src/net_processing.cpp#L2015-L2024)). This is because no valid peer can be assigned a negative ID as it starts from 0 ([source](https://github.com/dashpay/dash/blob/bd589dd6f434c0c6cc8f0c3e2cf299785bad6d07/src/net.h#L1735)) and increments ([source](https://github.com/dashpay/dash/blob/bd589dd6f434c0c6cc8f0c3e2cf299785bad6d07/src/net.cpp#L3897-L3900)). An example of this kind of usage is InstantSend code ([source](https://github.com/dashpay/dash/blob/859d64e98fc1b5bf613f526ef76e5fc66851fe13/src/instantsend/signing.cpp#L252)).

    * This change, alongside relevant changes to `ProcessOrphanTx()` arguments and other Dash-specific code has been segmented out as a distinct commit to allow for ease of review. This also means that the commit that backports [bitcoin#26295](bitcoin#26295) cannot be compiled on its own due to build errors from not-yet adapted code.

    * Additionally, `ProcessOrphanTx()` assumes `Peer`-triggered orphan resolution and doesn't account for the possibility that the processing of a transaction may further populate the orphan set. This is a key assumption in block-based orphan processing and to retain this intended functionality, `TxOrphanage::HaveMoreWork()` has been defined to influence the return value of `ProcessOrphanTx()` to affect the resolution loop ([source](https://github.com/dashpay/dash/blob/859d64e98fc1b5bf613f526ef76e5fc66851fe13/src/net_processing.cpp#L2015-L2024)) to exhaust all work before continuing ahead.

  ## Breaking changes

  None expected.

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  knst:
    utACK f45de22
  UdjinM6:
    utACK f45de22

Tree-SHA512: 8f6a46984091d6f43d4fb97d13edc9dfa7affa1eb79ab37e14f49078b5d738308c6044f9b2b3f5e0d1fe2efde27b7bae14727e14e1ccd5d21eb695162d9c4615
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants