-
Notifications
You must be signed in to change notification settings - Fork 38.7k
rest: allow reading partial block data from storage #33657
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33657. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
11e3a3c to
787d6df
Compare
|
Some of the PR description is reused from the previous PR, but is now stale.
This does not support fetching an N-th transaction, only a slice of a block.
There is no way to use this in non-binary form (except hex I suppose). |
787d6df to
c4bf7a7
Compare
Good catch, thanks! |
c4bf7a7 to
c9bfd29
Compare
|
Concept ACK c9bfd298be422de7e989fe244fb4281c507068a3 |
c9bfd29 to
301116e
Compare
|
tACK 301116e855 What was done:
If there are any additional test you'd like me to help with let me know. |
|
Many thanks @ubbabeck!
It would be interesting to compare the performance for single transaction fetching, similar to how it was done in #32541 (comment), comparing |
Here are the results from testing. I used hey to benchmark and your rust getrawtransaction. Was this somewhat you were looking for? If not let me know and I'll redo them correctly. Maybe not relevant to test, but fetching the whole block using blockpart.
DetailsUsing a random offset and size of 310 bytes to simulate fetching a transaction.
DetailsOne runner
DetailsRust getrawtransaction benchmark |
|
Tested the performance of the new REST endpoint using
So fetching a typical transaction is expected to take <40us on my machine. @ubbabeck could you please test it too? |
|
Compared your command fetching SIZE=788 and block 850000 with a 788 byte tx in that block 2d2dcc80195541dd44209dcfeb25393c8c8710f262360368a618b7ff3fa3f08c using I got Compared with RPC where I got Using binary encoding |
Sure here her my results for the same test using the same params and ApacheBench, Version 2.3 Details
Fetching on average tx on my system is expected to take <0.05 ms For fetching
|
sedited
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made sure this also doesn't slow down normal block retrieval. There is a bit of extra logic in the hot path for ReadRawBlock, but I think this should be fine.
301116e to
c30647c
Compare
sedited
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK c30647c4d34c2941696729704854467b30657c43
|
@romanz do you have a proof-of-concept for the tx indexer that will utilize this? It would be good to make sure this will be sufficient and performant enough for your requirements. |
Yes - I am working on a PR to adapt Will test it on mainnet as well :) |
|
Tested mainnet query performance using https://mempool.space/address/1BitcoinEaterAddressDontSendf59kuE. |
c30647c to
9a4c016
Compare
389eafc to
e4f460b
Compare
It will allow fetching specific transactions using an external index, following bitcoin#32541 (comment). Co-authored-by: Hodlinator <[email protected]> Co-authored-by: Lőrinc <[email protected]>
e4f460b to
0713529
Compare
|
Applied most of the recommendations above & removed a trailing space in https://github.com/bitcoin/bitcoin/compare/e4f460bbe44ba11e3997da80daf39cbb2515fefc..07135290c1720a14c9d2f18a5700bb6565ae7a10. |
|
Nice. Only non-test change is adding the missing review ACK 0713529 🏪 Show signatureSignature: |
hodlinator
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 0713529
|
|
||
| if (block_part) { | ||
| const auto [offset, size]{*block_part}; | ||
| if (size == 0 || offset >= blk_size || size > blk_size - offset) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Agree on reducing the number of direct conditions by using SaturatingAdd as sugggested at the beginning of the thread (passes current unit tests).
No strong opinion for/against disallowing 0-size ranges.
|
ACK 0713529 Since my last ACK: deduplicated & sorted the unit test cases, moved non-const return value in rpc rfm? |
|
Many thanks! |
938d7aacab Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage 597b8be223 Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling d155fc12a0 Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate 07135290c1 rest: allow reading partial block data from storage 4e2af1c065 blockstorage: allow reading partial block data from storage f2fd1aa21c blockstorage: return an error code from `ReadRawBlock()` 5be20c380d Merge bitcoin/bitcoin#34033: scripted-diff: Unify error and warning log formatting b31f786695 Merge bitcoin/bitcoin#34045: test: Log IP of download server in get_previous_releases.py b26762bdcb Merge bitcoin/bitcoin#33805: merkle: migrate `path` arg to reference and drop unused args 0f6d8a347a Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs c2975f26d6 Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation cdaf25f9c3 test: Log IP of download server in get_previous_releases.py c1f0a89d9c Merge bitcoin/bitcoin#34040: test: Detect truncated download in get_previous_releases.py fa75480c84 test: Detect truncated download in get_previous_releases.py 56ce78d5f6 Merge bitcoin/bitcoin#34031: net: Remove "tor" as a network specification 500862b2d4 Merge bitcoin/bitcoin#33423: qa: Improvements to debug_assert_log + busy_wait_for_debug_log 5f5c1ea019 net: Cache -capturemessages setting cca113f5b0 Merge bitcoin/bitcoin#34008: log: don't rate-limit "new peer" with -debug=net 2c44c41984 Merge bitcoin/bitcoin#33553: validation: Improve warnings in case of chain corruption 6eb5ba5691 refactor: extract shared `SipHash` state into `SipHashState` 118d22ddb4 optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs` 9ca52a4cbe optimization: migrate `SipHashUint256` to `PresaltedSipHasher` ec11b9fede optimization: introduce `PresaltedSipHasher` for repeated hashing d23d49ee3f Merge bitcoin/bitcoin#31823: tests: Add witness commitment if we have a witness transaction in `FullBlockTest.update_block()` 20330548cf refactor: extract `SipHash` C0-C3 constants to class scope 9f9eb7fbc0 test: rename k1/k2 to k0/k1 in `SipHash` consistency tests 29ed608dc7 Merge bitcoin/bitcoin#33961: script: Add a separate ScriptError for empty pubkeys encountered in Tapscript d2a199bca7 Merge bitcoin/bitcoin#33909: doc, ci: Make the max number of commits tested explicit dbc8928069 Merge bitcoin/bitcoin#33993: init: point out -stopatheight may be imprecise d4d184eda9 log: don't rate-limit "new peer" with -debug=net e7ac5a133c doc: add release note for 34031 c4c70a256e netbase: Remove "tor" as a network specification fa89f60e31 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning() fa6c7a1954 scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug() d5c8199b79 Merge bitcoin/bitcoin#34006: Add util::Expected (std::expected) 77248e8496 Merge bitcoin/bitcoin#33771: refactor: C++20 operators 36073d56db Merge bitcoin/bitcoin#33952: depends: update freetype and document remaining `bitcoin-qt` runtime libs f09ae5f96f Merge bitcoin/bitcoin#33950: guix: reduce allowed exported symbols cea443e246 net: Pass time to InactivityChecks fuctions 89dc82295e Merge bitcoin/bitcoin#29641: scripted-diff: Use LogInfo over LogPrintf eb19a2dac5 Merge bitcoin/bitcoin#34017: fuzz: Add a test case for `ParseByteUnits()` faa23738fc refactor: Enable clang-tidy bugprone-unused-return-value fa114be27b Add util::Expected (std::expected) e68517208b Merge bitcoin/bitcoin#33995: depends: Propagate native C compiler to `sqlite` package 091cae6fdf Merge bitcoin/bitcoin#33939: contrib: Count entry differences in asmap-tool diff summary 57b888ce0e fuzz: Add a test case for `ParseByteUnits()` b8e66b901d Merge bitcoin/bitcoin#33858: test: add unit test coverage for the empty leaves path in MerkleComputation 0c9ab0f8f8 Merge bitcoin/bitcoin#33956: net: fix use-after-free with v2->v1 reconnection logic fa4395dffd refactor: Remove unused LogPrintf fa05181d90 scripted-diff: LogPrintf -> LogInfo 9890058b37 Merge bitcoin/bitcoin#33723: chainparams: remove dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us 9e02f78089 Merge bitcoin/bitcoin#33774: cmake: Move IPC tests to `ipc/test` ad452a1e65 Merge bitcoin/bitcoin#33528: wallet: don't consider unconfirmed TRUC coins with ancestors ff06e2468a init: point out -stopatheight may be imprecise 9a29b2d331 Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 69e66efe45 Merge bitcoin/bitcoin#32882: index: remove unnecessary locator cleaning in BaseIndex::Init() 6581ac5d9f Merge bitcoin/bitcoin#33996: contrib: fix manpage generation 39ca015259 Merge bitcoin/bitcoin#33140: test: Avoid shutdown race in NetworkThread e9536faaee contrib: fix manpage generation bcf794d5f3 Merge bitcoin/bitcoin#30455: test: assumeutxo: add missing tests in wallet_assumeutxo.py af0e6a65c9 Merge bitcoin/bitcoin#33702: contrib: Remove brittle, confusing and redundant UTF8 encoding from Python IO 4b47113698 validation: Reword CheckForkWarningConditions and call it also during IBD and at startup 2f51951d03 p2p: Add warning message when receiving headers for blocks cached as invalid 4c784b25c4 Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight 710031ebef Revert "guix: sqlite wants tcl" 4cf5ea6c3d depends: Propagate native C compiler to `sqlite` package ce771726f3 Merge bitcoin/bitcoin#33960: log: Use more severe log level (warn/err) where appropriate cb7d5bfe4a test, assumeutxo: loading a wallet (backup) on a pruned node 7a365244f8 test, refactor snapshot import and background validation e0ba6bbed9 Merge bitcoin/bitcoin#33591: Cluster mempool followups b8d279a81c doc: add comment to explain correctness of GatherClusters() aba7500a30 Fix parameter name in getmempoolcluster rpc 6c1325a091 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology bc2eb931da Require mempool lock to be held when invoking TRUC checks 957ae23241 Improve comments for getTransactionAncestry to reference cluster counts instead of descendants d97d6199ce Fix comment to reference cluster limits, not chain limits a1b341ef98 Sanity check feerate diagram in CTxMemPool::check() 23d6f457c4 rpc: improve getmempoolcluster output d2dcd37aac Avoid using mapTx.modify() to update modified fees d84ffc24d2 doc: add release notes snippet for cluster mempool b0417ba944 doc: Add design notes for cluster mempool and explain new mempool limits 804329400a fuzz: gate mempool entry based on weight 6356041e58 Merge bitcoin/bitcoin#33972: cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 7d7cb1bb48 Merge bitcoin/bitcoin#33971: cmake: Set `WITH_ZMQ` to `ON` in Windows presets 2d88966e43 miner: replace "package" with "chunk" 6f3e8eb300 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler b5f245f6f2 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB 1dac54d506 Use cluster size limit instead of ancestor size limit in txpackage unit test 04f65488ca Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits 634291a7dc Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits fc18ef1f3f Remove ancestor and descendant vsize limits from MemPoolLimits ed8e819121 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect 80d8df2d47 Invoke removeUnchecked() directly in removeForBlock() 9292570f4c Rewrite GetChildren without sets 3e39ea8c30 Rewrite removeForReorg to avoid using sets a3c31dfd71 scripted-diff: rename AddToMempool -> TryAddToMempool a5a7905d83 Simplify removeRecursive 01d8520038 Remove unused argument to RemoveStaged fe1815d48f cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 49c6728535 cmake: Set `WITH_ZMQ` to `ON` in Windows presets ec8eb013a9 doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 48496caa12 ci: Remove redundant `DEP_OPTS` from “Windows-cross UCRT” job f6acbef108 Merge bitcoin/bitcoin#33764: ci: Add Windows + UCRT jobs for cross-compiling and native testing b5a7a685bb ci: Make the max number of commits tested explicit 9d5021a05b script: add SCRIPT_ERR_TAPSCRIPT_EMPTY_PUBKEY 7b90b4f5bb guix: reduce allowed exported symbols 41e657aacf guix: add bitcoin-qt runtime libs doc in symbol-check ef4ce19a15 depends: freetype 2.11.1 808f1d972b Merge bitcoin/bitcoin#32009: contrib: turn off compression of macOS SDK to fix determinism (across distros) 4de26b111f Merge bitcoin/bitcoin#33514: ci: clear out space on CentOS, depends, gui GHA job fa45a1503e log: Use LogWarning for non-critical logs fa0018d011 log: Use LogError for fatal errors 22229de728 doc: Fix typo in init log 38c8474d0d Merge bitcoin/bitcoin#33914: Change Parse descriptor argument to string_view 4b25b274de Merge bitcoin/bitcoin#33951: test: check for output to stdout in `TestShell` test 167df7a98c net: fix use-after-free with v2->v1 reconnection logic 52230a7f69 test: check for output to stdout in `TestShell` test 85d058dc53 Merge bitcoin/bitcoin#33946: interfaces: remove redundant mempool lock in ChainImpl::isInMempool() fd4ce55121 contrib: Count entry differences in asmap-tool diff summary e07e57368e ci: clear out space on centos job 79d6e874e1 Merge bitcoin/bitcoin#32587: test: Fix reorg patterns in tests to use proper fork-based approach e249ea7da6 Merge bitcoin/bitcoin#33945: depends: latest config.guess & config.sub 3e01b5d0e7 contrib: rename gen-sdk to gen-sdk.py c1213a35ab macdeploy: disable compression in macOS gen-sdk script a33d034545 contrib: more selectively pick files for macOS SDK fad6118586 test: Fix "typo" in written invalid content fab085c15f contrib: Use text=True in subprocess over manual encoding handling fa71c15f86 scripted-diff: Bump copyright headers after encoding changes fae612424b contrib: Remove confusing and redundant encoding from IO fa7d72bd1b lint: Drop check to enforce encoding to be specified in Python scripts faf39d8539 test: Clarify that Python UTF-8 mode is the default today for most systems fa83e3a81d lint: Do not allow locale dependent shell scripts 70d9e8f0a1 fix: reorg behaviour in mempool tests to match real one 540ed333f6 Move the create_empty_fork method to the test framework's blocktools.py module to enable reuse across multiple tests. 2909655fba fix: remove redundant mempool lock in ChainImpl::isInMempool() d5ed4ba9d8 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin 3e4355314b depends: latest config.sub 04eb84fe3f depends: latest config.guess b30262dcaa Merge bitcoin/bitcoin#33903: ci: Remove redundant busybox option 1a5f1eb080 Merge bitcoin/bitcoin#33921: doc: clarify and cleanup macOS fuzzing notes 72cb8cef97 Merge bitcoin/bitcoin#33862: txgraph: drop move assignment operator bc64013e6f Remove unused variable (cacheMap) in mempool ade0397f59 txgraph: drop move assignment operator 5336bcd578 Merge bitcoin/bitcoin#33855: kernel: add btck_block_tree_entry_equals 4f65a1c5db Merge bitcoin/bitcoin#33917: clang-format: Set Bitcoin Core IncludeCategories 902717b66d Merge bitcoin/bitcoin#33918: depends: Update Qt download link 68ab2b65bf Merge bitcoin/bitcoin#33919: ci: Run GUI unit tests in cross-Windows task 7e129b644e Merge bitcoin/bitcoin#33893: test: add `-alertnotify` test for large work invalid chain warning 5fe753b56f Merge bitcoin/bitcoin#32655: depends: sqlite 3.50.4; switch to autosetup ff8c2f3749 Merge bitcoin/bitcoin#33932: ci: Use latest Xcode that the minimum macOS version allows fa283d28e2 Merge bitcoin/bitcoin#33629: Cluster mempool 2e27bd9c3a ci: Add Windows + UCRT jobs for cross-compiling and native testing 238c1c8933 Merge bitcoin-core/gui#914: Revert "gui, qt: brintToFront workaround for Wayland" 8343a9ffcc test: add `-alertnotify` test for large work invalid chain warning c34bc01b2f doc: clarify and cleanup macOS fuzzing notes fa9537cde1 ci: Use latest Xcode that the minimum macOS version allows 17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general 315e43e5d8 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() de2e9a24c4 test: extend package rbf functional test to larger clusters 4ef4ddb504 doc: update policy/packages.md for new package acceptance logic 79f73ad713 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() a86ac11768 Update comments for CTxMemPool class 9567eaa66d Invoke TxGraph::DoWork() at appropriate times bd130db994 ci: Rename items specific to Windows + MSVCRT 0672e727bf Revert "gui, qt: brintToFront workaround for Wayland" fa7ea497c3 ci: Run GUI unit tests in cross-Windows task fa0fee44a8 ci: Remove redundant busybox option fa102ec69f doc: Shorten ci name fa7e222a23 clang-format: Set Bitcoin Core IncludeCategories 2222223780 doc: Remove bash -c wrapper 50cbde3295 depends: Update Qt download link c0bfe72f6e Change Parse descriptor argument to string_view 8558902e57 depends: Add patch for Windows11Style plugin 6c5c44f774 test: add functional test for new cluster mempool RPCs 72f60c877e doc: Update mempool_replacements.md to reflect feerate diagram checks 21693f031a Expose cluster information via rpc 72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing f107417490 bench: add more mempool benchmarks 7976eb1ae7 Avoid violating mempool policy limits in tests 84de685cf7 Stop tracking parents/children outside of txgraph 88672e205b Rewrite GatherClusters to use the txgraph implementation 1ca4f01090 Fix miniminer_tests to work with cluster limits 1902111e0f Eliminate CheckPackageLimits, which no longer does anything 3a646ec462 Rework RBF and TRUC validation 19b8479868 Make getting parents/children a function of the mempool, not a mempool entry 5560913e51 Rework truc_policy to use descendants, not children a4458d6c40 Use txgraph to calculate descendants c8b6f70d64 Use txgraph to calculate ancestors 241a3e666b Simplify ancestor calculation functions b9cec7f0a1 Make removeConflicts private 0402e6c780 Remove unused limits from CalculateMemPoolAncestors 08be765ac2 Remove mempool logic designed to maintain ancestor/descendant state fc4e3e6bc1 Remove unused members from CTxMemPoolEntry ff3b398d12 mempool: eliminate accessors to mempool entry ancestor/descendant cached state b9a2039f51 Eliminate use of cached ancestor data in miniminer_tests and truc_policy ba09fc9774 mempool: Remove unused function CalculateDescendantMaximum 8e49477e86 wallet: Replace max descendant count with cluster_count e031085fd4 Eliminate Single-Conflict RBF Carve Out cf3ab8e1d0 Stop enforcing descendant size/count limits 89ae38f489 test: remove rbf carveout test from mempool_limit.py c0bd04d18f Calculate descendant information for mempool RPC output on-the-fly bdcefb8a8b Use mempool/txgraph to determine if a tx has descendants 69e1eaa6ed Add test case for cluster size limits to TRUC logic 9cda64b86c Stop enforcing ancestor size/count limits 1f93227a84 Remove dependency on cached ancestor data in mini-miner 9fbe0a4ac2 rpc: Calculate ancestor data from scratch for mempool rpc calls 7961496dda Reimplement GetTransactionAncestry() to not rely on cached data feceaa42e8 Remove CTxMemPool::GetSortedDepthAndScore 21b5cea588 Use cluster linearization for transaction relay sort order 6445aa7d97 Remove the ancestor and descendant indices from the mempool 216e693729 Implement new RBF logic for cluster mempool ff8f115dec policy: Remove CPFP carveout rule c3f1afc934 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits 47ab32fdb1 Select transactions for blocks based on chunk feerate dec138d1dd fuzz: remove comparison between mini_miner block construction and miner 6c2bceb200 bench: rewrite ComplexMemPool to not create oversized clusters 1ad4590f63 Limit mempool size based on chunk feerate b11c89cab2 Rework miner_tests to not require large cluster limit 95a8297d48 Check cluster limits when using -walletrejectlongchains 95762e6759 Do not allow mempool clusters to exceed configured limits edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters 435fd56711 test: update feature_rbf.py replacement test 34e32985e8 Add new (unused) limits for cluster size/count 838d7e3553 Add transactions to txgraph, but without cluster dependencies a7c96f874d tests: Add witness commitment if we have a witness transaction in FullBlockTest.update_block() 096924d39d kernel: add btck_block_tree_entry_equals ffcae82a68 test: exercise TransactionMerklePath with empty block; targets the MerkleComputation empty-leaves path that was only reached by fuzz tests d5ed9cb3eb Add accessor for sigops-adjusted weight 1bf3b51396 Add sigops adjusted weight calculator c18c68a950 Create a txgraph inside CTxMemPool 29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref 92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying 24ed820d4f merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` 63d640fa6a merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` be270551df merkle: migrate `path` arg of `MerkleComputation` to a reference 866bbb98fd cmake, test: Improve locality of `bitcoin_ipc_test` library description ae2e438b25 cmake: Move IPC tests to `ipc/test` 48840bfc2d refactor: Prefer `<=>` over multiple relational operators 5a0f49bd26 refactor: Remove all `operator!=` definitions 0ac969cddf validation: don't reallocate cache for short-lived CCoinsViewCache c8f5e446dc coins: reduce lookups in dbcache layer propagation 1db7491470 depends: sqlite 3.50.4 286f3e49c8 guix: sqlite wants tcl b0c706795c Remove unreliable seed from chainparams.cpp, and the associated README 6c73e47448 mempool: Store iterators into mapTx in mapNextTx 51430680ec Allow moving an Epoch::Marker dcd42d6d8f [test] wallet send 3 generation TRUC e753fadfd0 [wallet] never try to spend from unconfirmed TRUC that already has ancestors fa6db79302 test: Avoid shutdown race in NetworkThread a1f7623020 qa: Only complain about expected messages that were not found 1e54125e2e refactor(qa): Avoid unnecessary string operations a9021101dc qa: Replace always-escaped regexps with "X in Y" 5c16e4631c doc: Remove no longer correct comment facd01e6ff refactor: remove redundant locator cleanup in BaseIndex::Init() c1e554d3e5 refactor: consolidate 3 separate locks into one block 41479ed1d2 test: add test for periodic flush inside ActivateBestChain 84820561dc validation: periodically flush dbcache during reindex-chainstate git-subtree-dir: depend/bitcoin git-subtree-split: 938d7aacabd0bb3784bb3e529b1ed06bb2891864
There was an extra indentation level (found during bitcoin#33657): ``` $ git show -U0 0713529 | ./contrib/devtools/clang-format-diff.py -p1 -i -v ```
Use `SaturatingAdd` following bitcoin#33657 (comment).
| resp = self.test_rest_request(f"/blockpart/{blockhash}", status=status, | ||
| req_type=req_type, ret_type=RetType.BYTES, **kwargs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just want to note that I saw CI failures here on windows after this was merged where the request for /block/ on line 463 seemed to succeed, but the /blockpart/ request immediately following it returned a 404 on line 466 (this line). Unfortunately the only error information was Expected: 200, Got: 404 - Response: b'' because line 87 above is only printing resp.status not resp.reason, so cause of the 404 is not clear.
The failures happened in 4 different runs:
https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933
https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933
https://github.com/bitcoin/bitcoin/actions/runs/20166532530/job/57895147947?pr=19460#step:13:3913
https://github.com/bitcoin/bitcoin/actions/runs/20166532530/job/57895147999?pr=19460#step:13:3917
And the logs look exactly the same exactly the same except for the blockhashes:
test TestFramework (INFO): Test the /blockpart URI
node0 [http] [httpserver.cpp:307] [http] Received a POST request for / from 127.0.0.1:56381
node0 [httpworker.0] [rpc/request.cpp:243] [void JSONRPCRequest::parse(const UniValue&)] [rpc] ThreadRPCServer method=getbestblockhash user=__cookie__
test TestFramework (DEBUG): GET /rest/block/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin
node0 [http] [httpserver.cpp:307] [http] Received a GET request for /rest/block/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin from 127.0.0.1:57139
test TestFramework (DEBUG): GET /rest/blockpart/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin?offset=0&size=652
node0 [http] [httpserver.cpp:307] [http] Received a GET request for /rest/blockpart/7bb6d8cde00d88f6a4342f0273a5981d2fdf4d397b35954e22c015fc55e19b7b.bin?offset=0&size=6 from 127.0.0.1:57140
test TestFramework (ERROR): Unexpected exception
Traceback (most recent call last):
File "D:\a\bitcoin\bitcoin\test\functional\test_framework\test_framework.py", line 142, in main
self.run_test()
~~~~~~~~~~~~~^^
File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 473, in run_test
assert_equal(block_bin, get_block_part(query_params={"offset": 0, "size": len(block_bin)}))
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 466, in get_block_part
resp = self.test_rest_request(f"/blockpart/{blockhash}", status=status,
req_type=req_type, ret_type=RetType.BYTES, **kwargs)
File "D:\a\bitcoin\bitcoin/test/functional/interface_rest.py", line 87, in test_rest_request
assert resp.status == status, f"Expected: {status}, Got: {resp.status} - Response: {str(resp.read())}"
^^^^^^^^^^^^^^^^^^^^^
AssertionError: Expected: 200, Got: 404 - Response: b''One thing that is weird here about the logs is the failing request shows up as ?offset=0&size=652 in the test logs but ?offset=0&size=6 in the node logs, where 652 is somehow truncated to 6. I don't think the truncation explains the error, because it doesn't look like there is a way for an incorrect size to trigger a 404, but the problems may be related.
Also, even though the same failure happened in the two PR's above, it did NOT happen in a third PR based on the other two (#19461) and pushed at the same time, which suggests the failure might happen randomly. I also don't think there's any code changes in any of the PR that would be related to this test or should change its behavior.
Not sure if this report is actionable, but wanted to say:
- If anyone has any idea what might cause these type of failures on windows CI, that would be helpful.
- It would be nice if the
assert resp.status == statuserror includedresp.reason, since knowing the message associated with the status would probably be helpful for debugging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
where
652is somehow truncated to6
Actually this appears to be normal because requests are truncated at 100 characters and 6 is the last character that fits
Line 308 in 41bf8f2
| RequestMethodString(hreq->GetRequestMethod()), SanitizeString(hreq->GetURI(), SAFE_CHARS_URI).substr(0, 100), hreq->GetPeer().ToStringAddrPort()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also wonder if there may be a CI bug causing this, where windows jobs were using old bitcoind binaries that did not have the /blockpart/ endpoint defined so it returned a 404?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I understand what's going on, but can we assume that it will happen again if it's a real problem (and not just some commit order mixup or whatever)?
doing a
curl 'https://productionresultssa2.blob.core.windows.net/actions-results/b958a1af-d4e2-49e6-9411-5feedc7e3f3a/workflow-job-run-1192ebfe-48b0-5433-a77e-092eca6802f2/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-12-15T19%3A49%3A59Z&sig=LJxVtw7oMg%2Byssj%2FbCpdhXU3T3PfIVYuOxqJwuu2FyE%3D&ske=2025-12-16T06%3A42%3A59Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-12-15T18%3A42%3A59Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2025-12-15T19%3A39%3A54Z&sv=2025-11-05' | grep 'Registering HTTP handler for /rest/block'shows no Registering HTTP handler for /rest/blockpart/ for https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933:
2025-12-12T14:35:29.4245912Z node0 2025-12-12T14:35:23.889620Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/block/notxdetails/ (exactmatch 0)
2025-12-12T14:35:29.4248088Z node0 2025-12-12T14:35:23.889630Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/block/ (exactmatch 0)
2025-12-12T14:35:29.4249218Z node0 2025-12-12T14:35:23.889640Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockfilter/ (exactmatch 0)
2025-12-12T14:35:29.4250381Z node0 2025-12-12T14:35:23.889650Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockfilterheaders/ (exactmatch 0)
2025-12-12T14:35:29.4259046Z node0 2025-12-12T14:35:23.889720Z [init] [httpserver.cpp:761] [void RegisterHTTPHandler(const std::string&, bool, const HTTPRequestHandler&)] [http] Registering HTTP handler for /rest/blockhashbyheight/ (exactmatch 0)
so a 404 is indeed expected.
Also, the build seems to be for GITHUB_SHA: 369ecaca0e89f90647edc41faa73b84ca31eebfa which doesn't seem to contain this PR - why are we even running those test, is it an invalid checkout, some leftovers that we forgot to clean up?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #33657 (comment)
shows no
Registering HTTP handler for /rest/blockpart/forhttps://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:13:3933:
Nice, thanks! That seems like a clear indication that the CI job was running the new test with an old bitcoind binary.
Not sure I understand what's going on, but can we assume that it will happen again if it's a real problem (and not just some commit order mixup or whatever)?
From what I can tell it seems like this is a real problem and it's probably fair to assume it will happen again at some point. But maybe it will happen rarely enough that it doesn't matter. It seems like I was just unlucky pushing my PR's around the same time that this PR was merged.
I do think it would be good if windows CI jobs that do not build their own binaries could check whether the binaries came from the same commit being tested, and produce an explicit error or warning if they don't. (Or maybe they could just check out the same commit that the binaries came from.) No idea how difficult these things would be to implement, though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ryanofsky do you mean that even with
GITHUB_SHA: 369ecaca0e89f90647edc41faa73b84ca31eebfasince we're fetching by PR number:
"C:\Program Files\Git\bin\git.exe" checkout --progress --force refs/remotes/pull/10102/merge
we might switch versions mid-run if they push?
2025-12-12T14:21:28.8214717Z HEAD is now at e3a830d Merge 3638b89090f539c360c5d4d358c4ed17db59c25e into 938d7aacabd0bb3784bb3e529b1ed06bb2891864
bitcoin/.github/workflows/ci.yml
Lines 153 to 154 in d2a199b
| # Ensure the latest merged pull request state is used, even on re-runs. | |
| ref: &CHECKOUT_REF_TMPL ${{ github.event_name == 'pull_request' && github.ref || '' }} |
should
github.ref be GITHUB_SHA - or maybe we should fail if the two don't coincide?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the issue is that we merge into different commits on the master branch:
https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57892614796?pr=10102#step:3:79
https://github.com/bitcoin/bitcoin/actions/runs/20166531437/job/57891206284?pr=10102#step:3:79
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #33657 (comment)
we might switch versions mid-run if they push?
Yes that seems like what happened. Test appears to be from e3a830d9d810b085eeb78e3e99c697508df52696 while binaries were built from 369ecaca0e89f90647edc41faa73b84ca31eebfa (GITHUB_SHA)
should
github.refbeGITHUB_SHA- or maybe we should fail if the two don't coincide?
It does seem like using GITHUB_SHA would fix this, but might have other negative consequences. I don't know enough to know what the right fix should be, or if this is worth fixing. The problem seems easy enough to work around for now by pushing again (which I need to do anyway because the linter job failed).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was intentionally left as-is, but a clean fix would be to record the commit id and check out the exact commit id later in the second task. see #33303 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in #34080
…dpoint 59b93f1 rest: print also HTTP response reason in case of an error (Roman Zeyde) 7fe94a0 rest: add a test for unsuported `/blockpart/` request type (Roman Zeyde) 55d0d19 rest: deduplicate `interface_rest.py` negative tests (Roman Zeyde) 89eb531 rest: update release notes for `/blockpart/` endpoint (Roman Zeyde) 41118e1 blockstorage: simplify partial block read validation (Roman Zeyde) 599effd rest: reformat `uri_prefixes` initializer list (Roman Zeyde) Pull request description: The commits below should resolve a few leftovers from #33657. ACKs for top commit: l0rinc: ACK 59b93f1 hodlinator: re-ACK 59b93f1 Tree-SHA512: ae45e08edd315018e11283b354fb32f9658f5829c956554dc662a81c2e16397def7c3700e6354e0a91ff03c850def35638a69ec2668b7c015d25d6fed42b92bb
…4ddc2dced57 94ddc2dced57 Merge bitcoin/bitcoin#34113: refactor: [rpc] Remove confusing and brittle integral casts c575990651d6 Merge bitcoin/bitcoin#34147: scripted-diff: refactor: wallet: Delete duplicate IsCrypted() eb0594e23f0c Merge bitcoin/bitcoin#33891: kernel: Expose reusable `PrecomputedTransactionData` in script validation e7033605775c Merge bitcoin/bitcoin#32997: index: Deduplicate HashKey / HeightKey handling ec4ff99a22b1 Merge bitcoin/bitcoin#33892: policy: allow <minrelay txns in package context if paid for by cpfp 48c9ba1e974b Merge bitcoin/bitcoin#34137: test: Avoid hard time.sleep(1) in feature_init.py 11ce5cf7997e scripted-diff: refactor: wallet: Delete IsCrypted 44e006d43831 [kernel] Expose reusable PrecomputedTransactionData in script valid fa727e3ec984 test: Avoid hard time.sleep(1) in feature_init.py d861c3820528 Merge bitcoin/bitcoin#33636: wallet: Expand MuSig test coverage and follow-ups 25636500c232 Merge bitcoin/bitcoin#32737: rpc, doc: clarify the response of listtransactions RPC d018876696cf Merge bitcoin/bitcoin#34039: test: address self-announcement 5bbc7c8cc1f2 Merge bitcoin/bitcoin#33810: ci: Add IWYU job 695e2b94ecd9 Merge bitcoin/bitcoin#33353: log: show reindex progress in `ImportBlocks` 1f151e73c00a Merge bitcoin/bitcoin#32929: qa: Clarify assert_start_raises_init_error output 315fdb406658 Merge bitcoin/bitcoin#34079: kernel: Remove non-kernel module includes d3a479cb077d kernel: Move BlockInfo to a kernel file d69a582e72ea kernel: Remove some unnecessary non-kernel includes e44dec027cee add release note about supporing non-TRUC <minrelay txns 7f295e1d9b44 Merge bitcoin/bitcoin#34084: scripted-diff: [doc] Unify stale copyright headers 5e7931af3573 Merge bitcoin/bitcoin#34095: refactor: enable `readability-container-contains` clang-tidy rule c80fd910f96c Merge bitcoin/bitcoin#33732: ci: Call docker exec from Python script to fix word splitting acba51101bbc Merge bitcoin/bitcoin#34107: build: Update minimum required Boost version fa66e2d07a4b refactor: [rpc] Remove confusing and brittle integral casts 74d6efe9c28b Merge bitcoin/bitcoin#34106: doc: add missing copyright headers 0c862bc7ea28 Merge bitcoin/bitcoin#32545: Replace cluster linearization algorithm with SFL 1e94e562f76e refactor: enable `readability-container-contains` clang-tidy rule fd9f1accbda9 Fix compilation for old Boost versions 75bdb925f404 clusterlin: drop support for improvable chunking (simplification) 91399a79122c clusterlin: remove unused MergeLinearizations (cleanup) 5ce280074512 clusterlin: randomize equal-feerate parts of linearization (privacy) 13aad26b7848 clusterlin: randomize various decisions in SFL (feature) ddbfa4dfac7b clusterlin: keep FIFO queue of improvable chunks (preparation) 3efc94d6564d clusterlin: replace cluster linearization with SFL (feature) 6a8fa821b80c clusterlin: add support for loading existing linearization (feature) da48ed9f348a clusterlin: ReadLinearization for non-topological (tests) c461259fb629 clusterlin: add class implementing SFL state (preparation) f480c1e71777 build: Update minimum required Boost version 95bfe7d574cf clusterlin: replace benchmarks with SFL-hard ones (bench) 86dd550a9b70 clusterlin: add known-correct optimal linearization tests (tests) aeb7ccb937bb doc: add missing copyright headers 68a7cb8f8be8 contrib: output copyright in generate-seeds.py 516ae5ede44a Merge bitcoin/bitcoin#31533: fuzz: Add fuzz target for block index tree and related validation events 9272fd517fd3 Merge bitcoin/bitcoin#34105: kernel: revert accidentally removed copyright header 85314dc0bf87 kernel: revert accidentally removed copyright header fa4cb13b5203 test: [doc] Manually unify stale headers 1841bf9cb67b test: address self-announcement 1ed8e7616527 rpc, doc: clarify the response of listtransactions RPC 09a1fa190ea3 Merge bitcoin/bitcoin#34094: chore: bump checkout to v6 80b1b5917dd1 Merge bitcoin/bitcoin#34088: log: Use `__func__` for -logsourcelocations 3a2807ad953c Merge bitcoin/bitcoin#33875: qa: Account for unset errno in ConnectionResetError 8d38b6f5f10b Merge bitcoin/bitcoin#34091: fuzz: doc: remove any mention to `address_deserialize_v2` cd98caea438a Update ci.yml ab513103df8d Merge bitcoin/bitcoin#33192: refactor: unify container presence checks 56750c4f87d0 iwyu, clang-format: Sort includes 2c78814e0e18 ci: Add IWYU job 94e4f04d7cf4 cmake: Fix target name 0f81e005197f cmake: Make `codegen` target dependent on `generate_build_info` 73f7844cdb1e iwyu: Add patch to prefer C++ headers over C counterparts 7a65437e2370 iwyu: Add patch to prefer angled brackets over quotes for includes facd3d56ccbe log: Use `__func__` for -logsourcelocations fe0e31f1efca Merge bitcoin/bitcoin#34053: lint: Remove confusing, redundant, and brittle lint-spelling e5c600dc0e06 Merge bitcoin/bitcoin#34063: Make `transaction_indentifier` hex string constructor evaluated at comptime 41f2cc6d3d59 Merge bitcoin-core/gui#919: move-only: MAX_BLOCK_TIME_GAP to src/qt 7c7cd8c296a5 Merge bitcoin/bitcoin#34089: contrib: asmap-tool.py - Don't write binary to TTY e3a4cb127f0d Merge bitcoin/bitcoin#34080: ci: Pin native tests on cross-builds to same commit a005fdff6c7a Merge bitcoin/bitcoin#34074: A few followups after introducing `/rest/blockpart/` endpoint caf4843a59a9 fuzz: doc: remove any mention to address_deserialize_v2 fa5ed16aa4d9 move-only: MAX_BLOCK_TIME_GAP to src/qt 356883f0e48b qa-tests: Log expected output in debug 7427a03b5ac9 qa-tests: Add test for timeouts due to missing init errors d7f703c1f1a8 refactor(qa-tests): Extract InternalDurationTestMixin for use in next commit 69bcfcad8c3d fix(qa-tests): Bring back decoding of exception field fb43b2f8cc4c qa: Improve assert_start_raises_init_error output 59b93f11e860 rest: print also HTTP response reason in case of an error 7fe94a04934a rest: add a test for unsuported `/blockpart/` request type fa5f29774872 scripted-diff: [doc] Unify stale copyright headers faa8ee62f5c1 ci: Pin native tests on cross-builds to same commit db2d39f64297 fuzz: add subtest for re-downloading a previously pruned block 45f5b2dac330 fuzz: Add fuzzer for block index c011e3aa5426 test: Wrap validation functions with TestChainstateManager 13891a8a685d Merge bitcoin/bitcoin#34050: fuzz: exercise `ComputeMerkleRoot` without `mutated` parameter ab643efc0a70 Merge bitcoin/bitcoin#34003: test: interface_ipc.py minor fixes and cleanup 4f11ef058b08 Merge bitcoin/bitcoin#30214: refactor: Improve assumeutxo state representation cbafd3ddf8a2 Merge bitcoin/bitcoin#34060: test: fix race condition in p2p_v2_misbehaving.py peerid assertion 55d0d19b5c02 rest: deduplicate `interface_rest.py` negative tests 89eb531024d9 rest: update release notes for `/blockpart/` endpoint 41bf8f2d5ece Merge bitcoin-core/gui#877: Add a menu action to restore then migrate a legacy wallet 2210feb4466e Merge bitcoin/bitcoin#34051: log: Remove brittle and confusing LogPrintLevel 58251bf9fa4b Merge bitcoin/bitcoin#34061: fuzz: Fix bugs in `clusterlin_postlinearize_tree` target 41118e17f875 blockstorage: simplify partial block read validation 599effdeab4d rest: reformat `uri_prefixes` initializer list 5ac35795206d refactor: Add compile-time-checked hex txid fa8a5d215c5a log: Remove brittle and confusing LogPrintLevel fac24bbec85f test: Clarify logging_SeverityLevels test f2731676619d ipc: separate log statements per level 94c51ae54072 libevent: separate log statements per level a70a14a3f4f4 refactor: Separate out logic for building a tree-shaped dependency graph ce29d7d6262c fuzz: Fix variable in `clusterlin_postlinearize_tree` check 876e2849b4ec fuzz: Fix incorrect loop bounds in `clusterlin_postlinearize_tree` 09dfa4d3f8df test: fix race condition in p2p_v2_misbehaving.py peerid assertion 938d7aacabd0 Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage 82be652e40ec doc: Improve ChainstateManager documentation, use consistent terms 597b8be223d4 Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling af455dcb39db refactor: Simplify pruning functions ae85c495f1b5 refactor: Delete ChainstateManager::GetAll() method 6a572dbda92c refactor: Add ChainstateManager::ActivateBestChains() method 491d827d5284 refactor: Add ChainstateManager::m_chainstates member e514fe611681 refactor: Delete ChainstateManager::SnapshotBlockhash() method ee35250683ab refactor: Delete ChainstateManager::IsSnapshotValidated() method d9e82299fc4e refactor: Delete ChainstateManager::IsSnapshotActive() method 4dfe38391276 refactor: Convert ChainstateRole enum to struct 352ad27fc1b1 refactor: Add ChainstateManager::ValidatedChainstate() method a229cb9477e6 refactor: Add ChainstateManager::CurrentChainstate() method a9b7f5614c24 refactor: Add Chainstate::StoragePath() method 840bd2ef230e refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation 1598a15aedb9 refactor: Deduplicate Chainstate activation code 9fe927b6d654 refactor: Add Chainstate m_assumeutxo and m_target_utxohash members 6082c84713f4 refactor: Add Chainstate::m_target_blockhash member de00e87548f7 test: Fix broken chainstatemanager_snapshot_init check fa904fc683c0 lint: Remove confusing, redundant, and brittle lint-spelling 14371fd1fca5 gui: Add a menu item to restore then migrate a wallet file f11a7d248cf5 gui: Add restore_and_migrate function to restore then migrate a wallet 16ab6dfc1074 gui: Move actual migration part of migrate() to its own function 4ec2d18a0734 wallet, interfaces, gui: Expose load_after_restore parameter d155fc12a0c7 Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate 07135290c172 rest: allow reading partial block data from storage 4e2af1c06547 blockstorage: allow reading partial block data from storage f2fd1aa21c76 blockstorage: return an error code from `ReadRawBlock()` 5be20c380dcb Merge bitcoin/bitcoin#34033: scripted-diff: Unify error and warning log formatting b31f7866952a Merge bitcoin/bitcoin#34045: test: Log IP of download server in get_previous_releases.py 7e9de20c0c14 fuzz: exercise `ComputeMerkleRoot` without mutated parameter b26762bdcb94 Merge bitcoin/bitcoin#33805: merkle: migrate `path` arg to reference and drop unused args 0f6d8a347aec Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs c2975f26d69f Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation cdaf25f9c3e5 test: Log IP of download server in get_previous_releases.py c1f0a89d9cae Merge bitcoin/bitcoin#34040: test: Detect truncated download in get_previous_releases.py fa75480c84ff test: Detect truncated download in get_previous_releases.py 56ce78d5f62c Merge bitcoin/bitcoin#34031: net: Remove "tor" as a network specification 500862b2d4a1 Merge bitcoin/bitcoin#33423: qa: Improvements to debug_assert_log + busy_wait_for_debug_log 5f5c1ea01955 net: Cache -capturemessages setting cca113f5b022 Merge bitcoin/bitcoin#34008: log: don't rate-limit "new peer" with -debug=net 2c44c41984e0 Merge bitcoin/bitcoin#33553: validation: Improve warnings in case of chain corruption 6eb5ba569141 refactor: extract shared `SipHash` state into `SipHashState` 118d22ddb4ba optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs` 9ca52a4cbece optimization: migrate `SipHashUint256` to `PresaltedSipHasher` ec11b9fede2a optimization: introduce `PresaltedSipHasher` for repeated hashing d23d49ee3f23 Merge bitcoin/bitcoin#31823: tests: Add witness commitment if we have a witness transaction in `FullBlockTest.update_block()` 20330548cf5f refactor: extract `SipHash` C0-C3 constants to class scope 9f9eb7fbc053 test: rename k1/k2 to k0/k1 in `SipHash` consistency tests 29ed608dc75e Merge bitcoin/bitcoin#33961: script: Add a separate ScriptError for empty pubkeys encountered in Tapscript d2a199bca73b Merge bitcoin/bitcoin#33909: doc, ci: Make the max number of commits tested explicit dbc892806912 Merge bitcoin/bitcoin#33993: init: point out -stopatheight may be imprecise d4d184eda9c0 log: don't rate-limit "new peer" with -debug=net e7ac5a133cc3 doc: add release note for 34031 c4c70a256ed8 netbase: Remove "tor" as a network specification fa89f60e31d1 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning() fa6c7a1954ea scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug() d8fe5f0326c5 test: improve interface_ipc.py waitNext tests a5e61b1917af test: interface_ipc.py minor fixes and cleanup d5c8199b7904 Merge bitcoin/bitcoin#34006: Add util::Expected (std::expected) 77248e849699 Merge bitcoin/bitcoin#33771: refactor: C++20 operators 36073d56db0d Merge bitcoin/bitcoin#33952: depends: update freetype and document remaining `bitcoin-qt` runtime libs f09ae5f96fe8 Merge bitcoin/bitcoin#33950: guix: reduce allowed exported symbols cea443e24618 net: Pass time to InactivityChecks fuctions 89dc82295ebd Merge bitcoin/bitcoin#29641: scripted-diff: Use LogInfo over LogPrintf eb19a2dac5c7 Merge bitcoin/bitcoin#34017: fuzz: Add a test case for `ParseByteUnits()` faa23738fc25 refactor: Enable clang-tidy bugprone-unused-return-value fa114be27b17 Add util::Expected (std::expected) e68517208b4c Merge bitcoin/bitcoin#33995: depends: Propagate native C compiler to `sqlite` package 091cae6fdf89 Merge bitcoin/bitcoin#33939: contrib: Count entry differences in asmap-tool diff summary 57b888ce0ebd fuzz: Add a test case for `ParseByteUnits()` b8e66b901d56 Merge bitcoin/bitcoin#33858: test: add unit test coverage for the empty leaves path in MerkleComputation 0c9ab0f8f8c8 Merge bitcoin/bitcoin#33956: net: fix use-after-free with v2->v1 reconnection logic fa4395dffd43 refactor: Remove unused LogPrintf fa05181d904d scripted-diff: LogPrintf -> LogInfo 5646e6c0d358 index: restrict index helper function to namespace 032f3503e3fe index, refactor: deduplicate LookUpOne a67d3eb91d5e index: deduplicate Hash / Height handling 9890058b37b8 Merge bitcoin/bitcoin#33723: chainparams: remove dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us 9e02f7808909 Merge bitcoin/bitcoin#33774: cmake: Move IPC tests to `ipc/test` ad452a1e655e Merge bitcoin/bitcoin#33528: wallet: don't consider unconfirmed TRUC coins with ancestors ff06e2468a5d init: point out -stopatheight may be imprecise ded11fb04d82 test: fix interface_ipc.py template destruction 9a29b2d331ee Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 69e66efe45a0 Merge bitcoin/bitcoin#32882: index: remove unnecessary locator cleaning in BaseIndex::Init() d9319b06cf82 refactor: unify container presence checks - non-trivial counts 039307554eb3 refactor: unify container presence checks - trivial counts 8bb9219b6301 refactor: unify container presence checks - find 6581ac5d9f93 Merge bitcoin/bitcoin#33996: contrib: fix manpage generation 39ca01525977 Merge bitcoin/bitcoin#33140: test: Avoid shutdown race in NetworkThread e9536faaee2b contrib: fix manpage generation bcf794d5f35b Merge bitcoin/bitcoin#30455: test: assumeutxo: add missing tests in wallet_assumeutxo.py af0e6a65c928 Merge bitcoin/bitcoin#33702: contrib: Remove brittle, confusing and redundant UTF8 encoding from Python IO 4b4711369880 validation: Reword CheckForkWarningConditions and call it also during IBD and at startup 2f51951d03cc p2p: Add warning message when receiving headers for blocks cached as invalid 4c784b25c478 Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight 710031ebef83 Revert "guix: sqlite wants tcl" 4cf5ea6c3d2a depends: Propagate native C compiler to `sqlite` package ce771726f3e7 Merge bitcoin/bitcoin#33960: log: Use more severe log level (warn/err) where appropriate cb7d5bfe4a59 test, assumeutxo: loading a wallet (backup) on a pruned node 7a365244f839 test, refactor snapshot import and background validation e0ba6bbed97b Merge bitcoin/bitcoin#33591: Cluster mempool followups b8d279a81c16 doc: add comment to explain correctness of GatherClusters() aba7500a30ee Fix parameter name in getmempoolcluster rpc 6c1325a0913e Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology bc2eb931da30 Require mempool lock to be held when invoking TRUC checks 957ae232414b Improve comments for getTransactionAncestry to reference cluster counts instead of descendants d97d6199ce50 Fix comment to reference cluster limits, not chain limits a1b341ef9875 Sanity check feerate diagram in CTxMemPool::check() 23d6f457c4c0 rpc: improve getmempoolcluster output d2dcd37aac1e Avoid using mapTx.modify() to update modified fees d84ffc24d2dc doc: add release notes snippet for cluster mempool b0417ba94437 doc: Add design notes for cluster mempool and explain new mempool limits 804329400a73 fuzz: gate mempool entry based on weight 2d88966e43c6 miner: replace "package" with "chunk" 6f3e8eb3001a Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler b5f245f6f219 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB 1dac54d506b5 Use cluster size limit instead of ancestor size limit in txpackage unit test 04f65488ca3e Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits 634291a7dc44 Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits fc18ef1f3f33 Remove ancestor and descendant vsize limits from MemPoolLimits ed8e819121d7 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect 80d8df2d47c2 Invoke removeUnchecked() directly in removeForBlock() 9292570f4cb8 Rewrite GetChildren without sets 3e39ea8c3070 Rewrite removeForReorg to avoid using sets a3c31dfd71de scripted-diff: rename AddToMempool -> TryAddToMempool a5a7905d83df Simplify removeRecursive 01d8520038ea Remove unused argument to RemoveStaged ec8eb013a9bf doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 48496caa1235 ci: Remove redundant `DEP_OPTS` from “Windows-cross UCRT” job b5a7a685bba3 ci: Make the max number of commits tested explicit 9d5021a05bd3 script: add SCRIPT_ERR_TAPSCRIPT_EMPTY_PUBKEY 7b90b4f5bb10 guix: reduce allowed exported symbols 41e657aacfa6 guix: add bitcoin-qt runtime libs doc in symbol-check ef4ce19a1545 depends: freetype 2.11.1 fa45a1503eee log: Use LogWarning for non-critical logs fa0018d01102 log: Use LogError for fatal errors e7e51952dc24 contrib: Avoid outputting binary data to TTY 22229de7288f doc: Fix typo in init log 167df7a98c85 net: fix use-after-free with v2->v1 reconnection logic fd4ce55121e7 contrib: Count entry differences in asmap-tool diff summary 1488315d76ee policy: Allow any transaction version with < minrelay fad61185861a test: Fix "typo" in written invalid content fab085c15f72 contrib: Use text=True in subprocess over manual encoding handling fa71c15f8610 scripted-diff: Bump copyright headers after encoding changes fae612424b3e contrib: Remove confusing and redundant encoding from IO fa7d72bd1be9 lint: Drop check to enforce encoding to be specified in Python scripts faf39d8539c9 test: Clarify that Python UTF-8 mode is the default today for most systems fa83e3a81ddb lint: Do not allow locale dependent shell scripts 217dbbbb5e38 test: Add musig failure scenarios fa336053aada Move ci_exec to the Python script fa83555d163f ci: Require rsync to pass eeee02ea53dd ci: Untangle CI_EXEC bash function fa21fd1dc2e5 ci: Move macos snippet under DANGER_RUN_CI_ON_HOST fa37559ac5b7 ci: Document the retry script in PATH 666675e95fe8 ci: Move folder creation and docker kill to Python script bc64013e6fad Remove unused variable (cacheMap) in mempool c9519c260b7a musig: Check session id reuse e755614be586 sign: Remove duplicate sigversion check 0f7f0692ca1e musig: Move MUSIG_CHAINCODE to musig.cpp a7c96f874de1 tests: Add witness commitment if we have a witness transaction in FullBlockTest.update_block() 76e0e6087d03 qa: Account for errno not always being set for ConnectionResetError ffcae82a6810 test: exercise TransactionMerklePath with empty block; targets the MerkleComputation empty-leaves path that was only reached by fuzz tests 24ed820d4f0d merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` 63d640fa6a70 merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` be270551df30 merkle: migrate `path` arg of `MerkleComputation` to a reference 866bbb98fd36 cmake, test: Improve locality of `bitcoin_ipc_test` library description ae2e438b257f cmake: Move IPC tests to `ipc/test` 48840bfc2d7b refactor: Prefer `<=>` over multiple relational operators 5a0f49bd2661 refactor: Remove all `operator!=` definitions 0ac969cddfdb validation: don't reallocate cache for short-lived CCoinsViewCache c8f5e446dc95 coins: reduce lookups in dbcache layer propagation b0c706795ce6 Remove unreliable seed from chainparams.cpp, and the associated README dcd42d6d8f16 [test] wallet send 3 generation TRUC e753fadfd01c [wallet] never try to spend from unconfirmed TRUC that already has ancestors fa6db79302d2 test: Avoid shutdown race in NetworkThread a1f762302096 qa: Only complain about expected messages that were not found 1e54125e2e00 refactor(qa): Avoid unnecessary string operations a9021101dc63 qa: Replace always-escaped regexps with "X in Y" 5c16e4631c00 doc: Remove no longer correct comment facd01e6ffbb refactor: remove redundant locator cleanup in BaseIndex::Init() d7de5b109f69 logs: show reindex progress in `ImportBlocks` c1e554d3e583 refactor: consolidate 3 separate locks into one block 41479ed1d23e test: add test for periodic flush inside ActivateBestChain 84820561dcb2 validation: periodically flush dbcache during reindex-chainstate git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 94ddc2dced5736612e358a3b80f2bc718fbd8161
…dc2dced 94ddc2dced Merge bitcoin/bitcoin#34113: refactor: [rpc] Remove confusing and brittle integral casts c575990651 Merge bitcoin/bitcoin#34147: scripted-diff: refactor: wallet: Delete duplicate IsCrypted() eb0594e23f Merge bitcoin/bitcoin#33891: kernel: Expose reusable `PrecomputedTransactionData` in script validation e703360577 Merge bitcoin/bitcoin#32997: index: Deduplicate HashKey / HeightKey handling ec4ff99a22 Merge bitcoin/bitcoin#33892: policy: allow <minrelay txns in package context if paid for by cpfp 48c9ba1e97 Merge bitcoin/bitcoin#34137: test: Avoid hard time.sleep(1) in feature_init.py 11ce5cf799 scripted-diff: refactor: wallet: Delete IsCrypted 44e006d438 [kernel] Expose reusable PrecomputedTransactionData in script valid fa727e3ec9 test: Avoid hard time.sleep(1) in feature_init.py d861c38205 Merge bitcoin/bitcoin#33636: wallet: Expand MuSig test coverage and follow-ups 25636500c2 Merge bitcoin/bitcoin#32737: rpc, doc: clarify the response of listtransactions RPC d018876696 Merge bitcoin/bitcoin#34039: test: address self-announcement 5bbc7c8cc1 Merge bitcoin/bitcoin#33810: ci: Add IWYU job 695e2b94ec Merge bitcoin/bitcoin#33353: log: show reindex progress in `ImportBlocks` 1f151e73c0 Merge bitcoin/bitcoin#32929: qa: Clarify assert_start_raises_init_error output 315fdb4066 Merge bitcoin/bitcoin#34079: kernel: Remove non-kernel module includes d3a479cb07 kernel: Move BlockInfo to a kernel file d69a582e72 kernel: Remove some unnecessary non-kernel includes e44dec027c add release note about supporing non-TRUC <minrelay txns 7f295e1d9b Merge bitcoin/bitcoin#34084: scripted-diff: [doc] Unify stale copyright headers 5e7931af35 Merge bitcoin/bitcoin#34095: refactor: enable `readability-container-contains` clang-tidy rule c80fd910f9 Merge bitcoin/bitcoin#33732: ci: Call docker exec from Python script to fix word splitting acba51101b Merge bitcoin/bitcoin#34107: build: Update minimum required Boost version fa66e2d07a refactor: [rpc] Remove confusing and brittle integral casts 74d6efe9c2 Merge bitcoin/bitcoin#34106: doc: add missing copyright headers 0c862bc7ea Merge bitcoin/bitcoin#32545: Replace cluster linearization algorithm with SFL 1e94e562f7 refactor: enable `readability-container-contains` clang-tidy rule fd9f1accbd Fix compilation for old Boost versions 75bdb925f4 clusterlin: drop support for improvable chunking (simplification) 91399a7912 clusterlin: remove unused MergeLinearizations (cleanup) 5ce2800745 clusterlin: randomize equal-feerate parts of linearization (privacy) 13aad26b78 clusterlin: randomize various decisions in SFL (feature) ddbfa4dfac clusterlin: keep FIFO queue of improvable chunks (preparation) 3efc94d656 clusterlin: replace cluster linearization with SFL (feature) 6a8fa821b8 clusterlin: add support for loading existing linearization (feature) da48ed9f34 clusterlin: ReadLinearization for non-topological (tests) c461259fb6 clusterlin: add class implementing SFL state (preparation) f480c1e717 build: Update minimum required Boost version 95bfe7d574 clusterlin: replace benchmarks with SFL-hard ones (bench) 86dd550a9b clusterlin: add known-correct optimal linearization tests (tests) aeb7ccb937 doc: add missing copyright headers 68a7cb8f8b contrib: output copyright in generate-seeds.py 516ae5ede4 Merge bitcoin/bitcoin#31533: fuzz: Add fuzz target for block index tree and related validation events 9272fd517f Merge bitcoin/bitcoin#34105: kernel: revert accidentally removed copyright header 85314dc0bf kernel: revert accidentally removed copyright header fa4cb13b52 test: [doc] Manually unify stale headers 1841bf9cb6 test: address self-announcement 1ed8e76165 rpc, doc: clarify the response of listtransactions RPC 09a1fa190e Merge bitcoin/bitcoin#34094: chore: bump checkout to v6 80b1b5917d Merge bitcoin/bitcoin#34088: log: Use `__func__` for -logsourcelocations 3a2807ad95 Merge bitcoin/bitcoin#33875: qa: Account for unset errno in ConnectionResetError 8d38b6f5f1 Merge bitcoin/bitcoin#34091: fuzz: doc: remove any mention to `address_deserialize_v2` cd98caea43 Update ci.yml ab513103df Merge bitcoin/bitcoin#33192: refactor: unify container presence checks 56750c4f87 iwyu, clang-format: Sort includes 2c78814e0e ci: Add IWYU job 94e4f04d7c cmake: Fix target name 0f81e00519 cmake: Make `codegen` target dependent on `generate_build_info` 73f7844cdb iwyu: Add patch to prefer C++ headers over C counterparts 7a65437e23 iwyu: Add patch to prefer angled brackets over quotes for includes facd3d56cc log: Use `__func__` for -logsourcelocations fe0e31f1ef Merge bitcoin/bitcoin#34053: lint: Remove confusing, redundant, and brittle lint-spelling e5c600dc0e Merge bitcoin/bitcoin#34063: Make `transaction_indentifier` hex string constructor evaluated at comptime 41f2cc6d3d Merge bitcoin-core/gui#919: move-only: MAX_BLOCK_TIME_GAP to src/qt 7c7cd8c296 Merge bitcoin/bitcoin#34089: contrib: asmap-tool.py - Don't write binary to TTY e3a4cb127f Merge bitcoin/bitcoin#34080: ci: Pin native tests on cross-builds to same commit a005fdff6c Merge bitcoin/bitcoin#34074: A few followups after introducing `/rest/blockpart/` endpoint caf4843a59 fuzz: doc: remove any mention to address_deserialize_v2 fa5ed16aa4 move-only: MAX_BLOCK_TIME_GAP to src/qt 356883f0e4 qa-tests: Log expected output in debug 7427a03b5a qa-tests: Add test for timeouts due to missing init errors d7f703c1f1 refactor(qa-tests): Extract InternalDurationTestMixin for use in next commit 69bcfcad8c fix(qa-tests): Bring back decoding of exception field fb43b2f8cc qa: Improve assert_start_raises_init_error output 59b93f11e8 rest: print also HTTP response reason in case of an error 7fe94a0493 rest: add a test for unsuported `/blockpart/` request type fa5f297748 scripted-diff: [doc] Unify stale copyright headers faa8ee62f5 ci: Pin native tests on cross-builds to same commit db2d39f642 fuzz: add subtest for re-downloading a previously pruned block 45f5b2dac3 fuzz: Add fuzzer for block index c011e3aa54 test: Wrap validation functions with TestChainstateManager 13891a8a68 Merge bitcoin/bitcoin#34050: fuzz: exercise `ComputeMerkleRoot` without `mutated` parameter ab643efc0a Merge bitcoin/bitcoin#34003: test: interface_ipc.py minor fixes and cleanup 4f11ef058b Merge bitcoin/bitcoin#30214: refactor: Improve assumeutxo state representation cbafd3ddf8 Merge bitcoin/bitcoin#34060: test: fix race condition in p2p_v2_misbehaving.py peerid assertion 55d0d19b5c rest: deduplicate `interface_rest.py` negative tests 89eb531024 rest: update release notes for `/blockpart/` endpoint 41bf8f2d5e Merge bitcoin-core/gui#877: Add a menu action to restore then migrate a legacy wallet 2210feb446 Merge bitcoin/bitcoin#34051: log: Remove brittle and confusing LogPrintLevel 58251bf9fa Merge bitcoin/bitcoin#34061: fuzz: Fix bugs in `clusterlin_postlinearize_tree` target 41118e17f8 blockstorage: simplify partial block read validation 599effdeab rest: reformat `uri_prefixes` initializer list 5ac3579520 refactor: Add compile-time-checked hex txid fa8a5d215c log: Remove brittle and confusing LogPrintLevel fac24bbec8 test: Clarify logging_SeverityLevels test f273167661 ipc: separate log statements per level 94c51ae540 libevent: separate log statements per level a70a14a3f4 refactor: Separate out logic for building a tree-shaped dependency graph ce29d7d626 fuzz: Fix variable in `clusterlin_postlinearize_tree` check 876e2849b4 fuzz: Fix incorrect loop bounds in `clusterlin_postlinearize_tree` 09dfa4d3f8 test: fix race condition in p2p_v2_misbehaving.py peerid assertion 938d7aacab Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage 82be652e40 doc: Improve ChainstateManager documentation, use consistent terms 597b8be223 Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling af455dcb39 refactor: Simplify pruning functions ae85c495f1 refactor: Delete ChainstateManager::GetAll() method 6a572dbda9 refactor: Add ChainstateManager::ActivateBestChains() method 491d827d52 refactor: Add ChainstateManager::m_chainstates member e514fe6116 refactor: Delete ChainstateManager::SnapshotBlockhash() method ee35250683 refactor: Delete ChainstateManager::IsSnapshotValidated() method d9e82299fc refactor: Delete ChainstateManager::IsSnapshotActive() method 4dfe383912 refactor: Convert ChainstateRole enum to struct 352ad27fc1 refactor: Add ChainstateManager::ValidatedChainstate() method a229cb9477 refactor: Add ChainstateManager::CurrentChainstate() method a9b7f5614c refactor: Add Chainstate::StoragePath() method 840bd2ef23 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation 1598a15aed refactor: Deduplicate Chainstate activation code 9fe927b6d6 refactor: Add Chainstate m_assumeutxo and m_target_utxohash members 6082c84713 refactor: Add Chainstate::m_target_blockhash member de00e87548 test: Fix broken chainstatemanager_snapshot_init check fa904fc683 lint: Remove confusing, redundant, and brittle lint-spelling 14371fd1fc gui: Add a menu item to restore then migrate a wallet file f11a7d248c gui: Add restore_and_migrate function to restore then migrate a wallet 16ab6dfc10 gui: Move actual migration part of migrate() to its own function 4ec2d18a07 wallet, interfaces, gui: Expose load_after_restore parameter d155fc12a0 Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate 07135290c1 rest: allow reading partial block data from storage 4e2af1c065 blockstorage: allow reading partial block data from storage f2fd1aa21c blockstorage: return an error code from `ReadRawBlock()` 5be20c380d Merge bitcoin/bitcoin#34033: scripted-diff: Unify error and warning log formatting b31f786695 Merge bitcoin/bitcoin#34045: test: Log IP of download server in get_previous_releases.py 7e9de20c0c fuzz: exercise `ComputeMerkleRoot` without mutated parameter b26762bdcb Merge bitcoin/bitcoin#33805: merkle: migrate `path` arg to reference and drop unused args 0f6d8a347a Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs c2975f26d6 Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation cdaf25f9c3 test: Log IP of download server in get_previous_releases.py c1f0a89d9c Merge bitcoin/bitcoin#34040: test: Detect truncated download in get_previous_releases.py fa75480c84 test: Detect truncated download in get_previous_releases.py 56ce78d5f6 Merge bitcoin/bitcoin#34031: net: Remove "tor" as a network specification 500862b2d4 Merge bitcoin/bitcoin#33423: qa: Improvements to debug_assert_log + busy_wait_for_debug_log 5f5c1ea019 net: Cache -capturemessages setting cca113f5b0 Merge bitcoin/bitcoin#34008: log: don't rate-limit "new peer" with -debug=net 2c44c41984 Merge bitcoin/bitcoin#33553: validation: Improve warnings in case of chain corruption 6eb5ba5691 refactor: extract shared `SipHash` state into `SipHashState` 118d22ddb4 optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs` 9ca52a4cbe optimization: migrate `SipHashUint256` to `PresaltedSipHasher` ec11b9fede optimization: introduce `PresaltedSipHasher` for repeated hashing d23d49ee3f Merge bitcoin/bitcoin#31823: tests: Add witness commitment if we have a witness transaction in `FullBlockTest.update_block()` 20330548cf refactor: extract `SipHash` C0-C3 constants to class scope 9f9eb7fbc0 test: rename k1/k2 to k0/k1 in `SipHash` consistency tests 29ed608dc7 Merge bitcoin/bitcoin#33961: script: Add a separate ScriptError for empty pubkeys encountered in Tapscript d2a199bca7 Merge bitcoin/bitcoin#33909: doc, ci: Make the max number of commits tested explicit dbc8928069 Merge bitcoin/bitcoin#33993: init: point out -stopatheight may be imprecise d4d184eda9 log: don't rate-limit "new peer" with -debug=net e7ac5a133c doc: add release note for 34031 c4c70a256e netbase: Remove "tor" as a network specification fa89f60e31 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning() fa6c7a1954 scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug() d8fe5f0326 test: improve interface_ipc.py waitNext tests a5e61b1917 test: interface_ipc.py minor fixes and cleanup d5c8199b79 Merge bitcoin/bitcoin#34006: Add util::Expected (std::expected) 77248e8496 Merge bitcoin/bitcoin#33771: refactor: C++20 operators 36073d56db Merge bitcoin/bitcoin#33952: depends: update freetype and document remaining `bitcoin-qt` runtime libs f09ae5f96f Merge bitcoin/bitcoin#33950: guix: reduce allowed exported symbols cea443e246 net: Pass time to InactivityChecks fuctions 89dc82295e Merge bitcoin/bitcoin#29641: scripted-diff: Use LogInfo over LogPrintf eb19a2dac5 Merge bitcoin/bitcoin#34017: fuzz: Add a test case for `ParseByteUnits()` faa23738fc refactor: Enable clang-tidy bugprone-unused-return-value fa114be27b Add util::Expected (std::expected) e68517208b Merge bitcoin/bitcoin#33995: depends: Propagate native C compiler to `sqlite` package 091cae6fdf Merge bitcoin/bitcoin#33939: contrib: Count entry differences in asmap-tool diff summary 57b888ce0e fuzz: Add a test case for `ParseByteUnits()` b8e66b901d Merge bitcoin/bitcoin#33858: test: add unit test coverage for the empty leaves path in MerkleComputation 0c9ab0f8f8 Merge bitcoin/bitcoin#33956: net: fix use-after-free with v2->v1 reconnection logic fa4395dffd refactor: Remove unused LogPrintf fa05181d90 scripted-diff: LogPrintf -> LogInfo 5646e6c0d3 index: restrict index helper function to namespace 032f3503e3 index, refactor: deduplicate LookUpOne a67d3eb91d index: deduplicate Hash / Height handling 9890058b37 Merge bitcoin/bitcoin#33723: chainparams: remove dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us 9e02f78089 Merge bitcoin/bitcoin#33774: cmake: Move IPC tests to `ipc/test` ad452a1e65 Merge bitcoin/bitcoin#33528: wallet: don't consider unconfirmed TRUC coins with ancestors ff06e2468a init: point out -stopatheight may be imprecise ded11fb04d test: fix interface_ipc.py template destruction 9a29b2d331 Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 69e66efe45 Merge bitcoin/bitcoin#32882: index: remove unnecessary locator cleaning in BaseIndex::Init() d9319b06cf refactor: unify container presence checks - non-trivial counts 039307554e refactor: unify container presence checks - trivial counts 8bb9219b63 refactor: unify container presence checks - find 6581ac5d9f Merge bitcoin/bitcoin#33996: contrib: fix manpage generation 39ca015259 Merge bitcoin/bitcoin#33140: test: Avoid shutdown race in NetworkThread e9536faaee contrib: fix manpage generation bcf794d5f3 Merge bitcoin/bitcoin#30455: test: assumeutxo: add missing tests in wallet_assumeutxo.py af0e6a65c9 Merge bitcoin/bitcoin#33702: contrib: Remove brittle, confusing and redundant UTF8 encoding from Python IO 4b47113698 validation: Reword CheckForkWarningConditions and call it also during IBD and at startup 2f51951d03 p2p: Add warning message when receiving headers for blocks cached as invalid 4c784b25c4 Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight 710031ebef Revert "guix: sqlite wants tcl" 4cf5ea6c3d depends: Propagate native C compiler to `sqlite` package ce771726f3 Merge bitcoin/bitcoin#33960: log: Use more severe log level (warn/err) where appropriate cb7d5bfe4a test, assumeutxo: loading a wallet (backup) on a pruned node 7a365244f8 test, refactor snapshot import and background validation e0ba6bbed9 Merge bitcoin/bitcoin#33591: Cluster mempool followups b8d279a81c doc: add comment to explain correctness of GatherClusters() aba7500a30 Fix parameter name in getmempoolcluster rpc 6c1325a091 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology bc2eb931da Require mempool lock to be held when invoking TRUC checks 957ae23241 Improve comments for getTransactionAncestry to reference cluster counts instead of descendants d97d6199ce Fix comment to reference cluster limits, not chain limits a1b341ef98 Sanity check feerate diagram in CTxMemPool::check() 23d6f457c4 rpc: improve getmempoolcluster output d2dcd37aac Avoid using mapTx.modify() to update modified fees d84ffc24d2 doc: add release notes snippet for cluster mempool b0417ba944 doc: Add design notes for cluster mempool and explain new mempool limits 804329400a fuzz: gate mempool entry based on weight 2d88966e43 miner: replace "package" with "chunk" 6f3e8eb300 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler b5f245f6f2 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB 1dac54d506 Use cluster size limit instead of ancestor size limit in txpackage unit test 04f65488ca Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits 634291a7dc Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits fc18ef1f3f Remove ancestor and descendant vsize limits from MemPoolLimits ed8e819121 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect 80d8df2d47 Invoke removeUnchecked() directly in removeForBlock() 9292570f4c Rewrite GetChildren without sets 3e39ea8c30 Rewrite removeForReorg to avoid using sets a3c31dfd71 scripted-diff: rename AddToMempool -> TryAddToMempool a5a7905d83 Simplify removeRecursive 01d8520038 Remove unused argument to RemoveStaged ec8eb013a9 doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 48496caa12 ci: Remove redundant `DEP_OPTS` from “Windows-cross UCRT” job b5a7a685bb ci: Make the max number of commits tested explicit 9d5021a05b script: add SCRIPT_ERR_TAPSCRIPT_EMPTY_PUBKEY 7b90b4f5bb guix: reduce allowed exported symbols 41e657aacf guix: add bitcoin-qt runtime libs doc in symbol-check ef4ce19a15 depends: freetype 2.11.1 fa45a1503e log: Use LogWarning for non-critical logs fa0018d011 log: Use LogError for fatal errors e7e51952dc contrib: Avoid outputting binary data to TTY 22229de728 doc: Fix typo in init log 167df7a98c net: fix use-after-free with v2->v1 reconnection logic fd4ce55121 contrib: Count entry differences in asmap-tool diff summary 1488315d76 policy: Allow any transaction version with < minrelay fad6118586 test: Fix "typo" in written invalid content fab085c15f contrib: Use text=True in subprocess over manual encoding handling fa71c15f86 scripted-diff: Bump copyright headers after encoding changes fae612424b contrib: Remove confusing and redundant encoding from IO fa7d72bd1b lint: Drop check to enforce encoding to be specified in Python scripts faf39d8539 test: Clarify that Python UTF-8 mode is the default today for most systems fa83e3a81d lint: Do not allow locale dependent shell scripts 217dbbbb5e test: Add musig failure scenarios fa336053aa Move ci_exec to the Python script fa83555d16 ci: Require rsync to pass eeee02ea53 ci: Untangle CI_EXEC bash function fa21fd1dc2 ci: Move macos snippet under DANGER_RUN_CI_ON_HOST fa37559ac5 ci: Document the retry script in PATH 666675e95f ci: Move folder creation and docker kill to Python script bc64013e6f Remove unused variable (cacheMap) in mempool c9519c260b musig: Check session id reuse e755614be5 sign: Remove duplicate sigversion check 0f7f0692ca musig: Move MUSIG_CHAINCODE to musig.cpp a7c96f874d tests: Add witness commitment if we have a witness transaction in FullBlockTest.update_block() 76e0e6087d qa: Account for errno not always being set for ConnectionResetError ffcae82a68 test: exercise TransactionMerklePath with empty block; targets the MerkleComputation empty-leaves path that was only reached by fuzz tests 24ed820d4f merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` 63d640fa6a merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` be270551df merkle: migrate `path` arg of `MerkleComputation` to a reference 866bbb98fd cmake, test: Improve locality of `bitcoin_ipc_test` library description ae2e438b25 cmake: Move IPC tests to `ipc/test` 48840bfc2d refactor: Prefer `<=>` over multiple relational operators 5a0f49bd26 refactor: Remove all `operator!=` definitions 0ac969cddf validation: don't reallocate cache for short-lived CCoinsViewCache c8f5e446dc coins: reduce lookups in dbcache layer propagation b0c706795c Remove unreliable seed from chainparams.cpp, and the associated README dcd42d6d8f [test] wallet send 3 generation TRUC e753fadfd0 [wallet] never try to spend from unconfirmed TRUC that already has ancestors fa6db79302 test: Avoid shutdown race in NetworkThread a1f7623020 qa: Only complain about expected messages that were not found 1e54125e2e refactor(qa): Avoid unnecessary string operations a9021101dc qa: Replace always-escaped regexps with "X in Y" 5c16e4631c doc: Remove no longer correct comment facd01e6ff refactor: remove redundant locator cleanup in BaseIndex::Init() d7de5b109f logs: show reindex progress in `ImportBlocks` c1e554d3e5 refactor: consolidate 3 separate locks into one block 41479ed1d2 test: add test for periodic flush inside ActivateBestChain 84820561dc validation: periodically flush dbcache during reindex-chainstate git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 94ddc2dced5736612e358a3b80f2bc718fbd8161
There was an extra indentation level (found during bitcoin#33657): ``` $ git show -U0 0713529 | ./contrib/devtools/clang-format-diff.py -p1 -i -v ```
Use `SaturatingAdd` following bitcoin#33657 (comment).
It allows fetching specific transactions using an external index, following #32541 (comment) and will be used by https://github.com/romanz/bindex (allowing much more efficient encoding).
Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.
However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction1.
Other indexers use a
txindexto fetch a transaction using its txid 234. This approach has significant storage and CPU overhead, since thetxidis a pseudo-random 32-byte value. Also, mainnettxindextakes ~60GB today.This PR is adding support for using the transaction's position within its block to be able to fetch it directly using REST API, using the following HTTP request:
The offsets' index can be encoded much more efficiently (~1.3GB today).
Address history query performance can be tested on mainnet using 1BitcoinEaterAddressDontSendf59kuE - assuming warm OS block cache, it takes <1s to fetch 5200 txs, i.e. <0.2ms per tx with bindex.
Only binary and hex response formats are supported.
Footnotes
https://github.com/romanz/electrs/blob/master/doc/schema.md ↩
https://github.com/Blockstream/electrs/blob/new-index/doc/schema.md#txstore ↩
https://github.com/spesmilo/electrumx/blob/master/docs/HOWTO.rst#prerequisites ↩
https://github.com/cculianu/Fulcrum/blob/master/README.md#requirements ↩