-
Notifications
You must be signed in to change notification settings - Fork 38.6k
contrib: turn off compression of macOS SDK to fix determinism (across distros) #32009
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32009. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste |
|
Is determinism of that tarball necessary? IIRC it doesn't actually affect the build. |
I don't see why we wouldn't want it. If we don't care about it, we should remove: "The |
|
Concept ACK, while it might not be strictly necessary for this tarball to be deterministic, I think matching hashes of tarballs is the simplest way to verify that you've followed the same procedure as others to build the macOS sdk and have gotten the same result. 150MB seems to me a small price to pay to fix #31873 and reduce surface area for nondeterminism in generating the macOS sdk. |
|
Concept ACK for the same reasoning as in #32009 (comment). |
|
Concept ACK. I don't think a few hundred MB more disk space required for (power-)users doing guix builds will be a great concern to anyone, and I agree with @davidgumberg that having as many steps in the guix workflow as possible be reproducible, is probably best. |
|
Concept ACK, imo determinism is good because it makes it possible to check if mistakes have been made in the process at earlier steps. One comment i was about to give is "why use a .tar.gz instead of .tar, that's the only way to be sure to remove dependency on zlib", but that's a much more spread out code change. This is most likely fine. edit: Could do that when we have to upgrade the MacOS SDK anyway, but definitely not for 29.0 |
|
I'm not too worried about the size increase. A typical guix build eats 30GB on my Ubuntu VM (3 GB for the outputs).
Agreed. It seems simple enough to backport, especially because we're not bumping the version. I'll see if I can reproduce the new hashes. |
|
One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip I still have the original xip on one of my machines, I'll upload it somewhere for testing. But we should probably bump it. (sha256 checksum is in Update: XCode 15 is still there, see below. I was able to reproduce the hash for the first commit 6998e933f935a379c3ad55c2fb16eca9b854f40b, but not for the second commit 20778eb0235df70397fc285f9e3b72270bd4aaf4. I get It looks like Apple Silicon has hardware acceleration for zip, maybe that's a factor? I also get |
I'm still seeing it here: https://developer.apple.com/download/all/?q=Xcode%2015 , and with this download link: https://download.developer.apple.com/Developer_Tools/Xcode_15/Xcode_15.xip, it's labeled |
|
@davidgumberg ah indeed, I see it as well. You have to click on the download link in order to for the download URL to work at all, so that's probably why it didn't work for me. So that just leaves the hash mismatch to figure out. |
|
At commit 20778eb0235df70397fc285f9e3b72270bd4aaf4 I get the following guix output: env HOSTS="x86_64-apple-darwin arm64-apple-darwin" ./contrib/guix/guix-build
<snip>
$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
30aa32906f879c5347be27bd9cb1a65b3d839acd72805fe3185276f788971fdb guix-build-20778eb0235d/output/arm64-apple-darwin/SHA256SUMS.part
0763a089bcc5d1d22191414f556bf461cefa8ea8a18971b171cf6fc5570b8790 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
18cd712ac4cbd63122483a083a9f29d7c184ffaead45078fb27b3d6381b691a6 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
c3fdc3b50077f2a4b772d4c1cf50e230f358542844e800e77194bf85dee9eb28 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.zip
b97da95ff464b8bdcb9ce31dc783f947dd24d93eb6bfdfea71c3070d51991058 guix-build-20778eb0235d/output/dist-archive/bitcoin-20778eb0235d.tar.gz
e180a9ef657ea3f0a6e42c4babe7b38a8e585902fbac301a1f16bccb0c53a18b guix-build-20778eb0235d/output/x86_64-apple-darwin/SHA256SUMS.part
5b032a2b6dd2dad11f3884194bcb85aea37237246162dfad27099e6c5175071d guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
39d23ec95f34cd7d8bbdb6228e55f7f8a0f48a3997c3dcb146a2ff3744f37629 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
76e36fa1af2f788dea631a7fcc93451fdf794a40dded0e3c7c12f74ab136aefb guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.zip
$ eza -al guix-build-20778eb0235d/output/*/*apple*.tar.gz
.rw-r--r-- 52M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
.rw-r--r-- 35M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
.rw-r--r-- 56M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
.rw-r--r-- 38M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
|
|
I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various
git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)@@ -2050,7 +2050,7 @@
00008010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00008020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00008030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-00008040: 0000 0000 0000 0000 00a0 815f 7e58 636f ..........._~Xco
+00008040: 0000 0000 0000 0000 0000 d0ff 2f58 636f ............/Xco
00008050: 6465 2d31 352e 302d 3135 4132 3430 642d de-15.0-15A240d-
00008060: 6578 7472 6163 7465 642d 5344 4b2d 7769 extracted-SDK-wi
00008070: 7468 2d6c 6962 6378 782d 6865 6164 6572 th-libcxx-header
@@ -4124,30 +4124,30 @@
000101b0: 6129 2041 5050 4c45 5f41 5243 4849 5645 a) APPLE_ARCHIVE
000101c0: 5f53 5749 4654 5f50 5249 5641 5445 3b0a _SWIFT_PRIVATE;.
000101d0: 0a23 6966 6465 6620 5f5f 6370 6c75 7370 .#ifdef __cplusp
-000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 608e lus.}.#endif..`.
-000101f0: 9f71 0000 0000 0000 0000 0000 0000 0000 .q..............
+000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 0000 lus.}.#endif....
+000101f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00010200: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00010210: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00010220: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00010230: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-00010240: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-00010250: 0000 5863 6f64 652d 3135 2e30 2d31 3541 ..Xcode-15.0-15A
-00010260: 3234 3064 2d65 7874 7261 6374 6564 2d53 240d-extracted-S
-00010270: 444b 2d77 6974 682d 6c69 6263 7878 2d68 DK-with-libcxx-h
-00010280: 6561 6465 7273 2f75 7372 2f69 6e63 6c75 eaders/usr/inclu
-00010290: 6465 2f41 7070 6c65 4172 6368 6976 652f de/AppleArchive/
-000102a0: 4141 456e 7472 7958 4154 426c 6f62 2e68 AAEntryXATBlob.h
-000102b0: 0000 0000 0000 3030 3030 3634 3400 3030 ......0000644.00
-000102c0: 3030 3030 3000 3030 3030 3030 3000 3030 00000.0000000.00
-000102d0: 3030 3030 3135 3437 3400 3030 3030 3030 000015474.000000
-000102e0: 3030 3030 3000 3032 3631 3631 0020 3000 00000.026161. 0.
+00010240: 0000 0000 0000 0000 0000 0000 0058 636f .............Xco
+00010250: 6465 2d31 352e 302d 3135 4132 3430 642d de-15.0-15A240d-
+00010260: 6578 7472 6163 7465 642d 5344 4b2d 7769 extracted-SDK-wi
+00010270: 7468 2d6c 6962 6378 782d 6865 6164 6572 th-libcxx-header
+00010280: 732f 7573 722f 696e 636c 7564 652f 4170 s/usr/include/Ap
+00010290: 706c 6541 7263 6869 7665 2f41 4145 6e74 pleArchive/AAEnt
+000102a0: 7279 5841 5442 6c6f 622e 6800 0000 0000 ryXATBlob.h.....
+000102b0: 0030 3030 3036 3434 0030 3030 3030 3030 .0000644.0000000
+000102c0: 0030 3030 3030 3030 0030 3030 3030 3031 .0000000.0000001
+000102d0: 3534 3734 0030 3030 3030 3030 3030 3030 5474.00000000000
+000102e0: 0030 3236 3136 3100 2030 0000 0000 0000 .026161. 0......
|
| image | python --version |
python -c "import zlib; print(zlib.ZLIB_VERSION)" |
|---|---|---|
fedora:39 |
3.12.7 | 1.2.13 |
fedora:40 |
3.12.9 | 1.3.1-zlib-ng |
fedora:41 |
3.13.2 | 1.3.1-zlib-ng |
ubuntu:24.04 |
3.12.3 | 1.3 |
ubuntu:24.10 |
3.12.7 | 1.3.1 |
archlinux:latest |
3.13.2 | 1.3.1 |
Failed to reproduce:
| image | python --version |
python -c "import zlib; print(zlib.ZLIB_VERSION)" |
hash |
|---|---|---|---|
debian:bookworm |
3.11.2 | 1.2.13 | 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 |
rockylinux:8.9 |
3.6.8 | 1.2.11 | e779914636e6a3a417bf2a19dbce6f0bf8fab10b16717df769d107a5aad6aa2e |
rockylinux:9.3 |
3.9.18 | 1.2.11 | 07b12c2a489c241bbc8c853fe78f2e92faf8ff51631311d142aeb8c7e20e7268 |
I downgraded to python 3.11.3 on fedora 41 with zlib 1.3, and got a bad hash, upgraded to python 3.12.0 with zlib-ng 1.3 on debian:bookworm and got a good hash.
custom zlib pyenv setup
Also see #31873 (comment)
dnf install -y gcc git make openssl-devel xz-devel git
curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..
curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"
pyenv install 3.11.3
pyenv global 3.11.3
# verify zlib version
python -c "import zlib; print(zlib.ZLIB_VERSION)"
# 1.3
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# get something bad
pyenv install 3.12.0
pyenv global 3.12.0
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gzDon't have super strong evidence for this, but I suspect that this might be a second source of nondeterministic behavior that was already present before, as in my previous testing of master when I reported this issue, when I was trying different combinations of python and zlib, I also saw a similar pattern where when changing to random python, some of the hashes were randomly mismatched, and the rest were either of two hashes, one right and one wrong. I avoided sharing the random hashes in my table in #31873, and all of the values I share are for python 3.12.x
A wild guess: Could this be related to python/cpython#95385? |
This seems to suggest the divergence is in the tarring, not the gzipping. Do you still see a difference if you |
|
Maybe keep the filename the same, so people can optionally compress it on their own (or even use the current file) without breaking the build? |
20778eb to
fc81a75
Compare
|
Rebased and added a change to skip more files to the first commit, which reduces the final uncompressed size from ~230mb to ~157mb. |
|
nit: can drop "RFC:" from the second commit?
Tested this again against a2bf3b03b19a0e668bf86eea7de3e3a592b4b23f and fc81a759413230daf3e05b448b357e62915bb3ba on my M4 machine running macOS 15.5 with Python 3.10.14 (via PyEnv). Again the hash matches in the first commit, but with second commit I get: If I use Python 3.11.11 instead I get the same hash as with 3.10.14. @davidgumberg wrote:
With Python 3.12.9 I indeed get We could bump the version in |
fc81a75 to
0a9b63b
Compare
Dropped. |
Wholesale bumping to 3.12 seems a bit early. Maybe just add an option to this tool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 71b0f47, Concept ACK on disabling compression to increase determinism.
71b0f47 seems like a strict win, no need to include what we don't use. I think conceptually 0a9b63b also makes sense: the increased size of the tarball should not be a blocked for increased determinism. However, on my machines (arm64 macOS 15.6 and x86_64 Debian Bookworm ) determinism seems to be reduced without compression across Python versions (as also reported by other reviewers). I'm not sure whether increased determinism across distros is more important than determinism across (supported) Python versions?
(note: below results are from macOS, the hashes are the same for Debian but the filesizes vary a bit)
With selective picking and no compression (0a9b63b) different sha256sum hashes are produced by python 3.{10,11}, 3.{12, 13} and 3.{14}:
py10: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
py11: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
py12: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
py13: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
py14: ea4684c568bbf63ca9fa4d1a9116751ba5b870728c4e9f72862f1b0d453ef05d (161mb)
With selective picking and compression (71b0f47) the results are consistent:
py10: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py11: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py12: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py13: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py14: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
On master (e872a56) the results are consistent:
py10: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py11: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py12: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py13: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py14: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
I don't think that's true. Our guix build documentation states that the tarball needs to be extracted into |
5513cd0 to
e7c22c8
Compare
|
ACK e7c22c8eb446727df00d1cb2ce554be9324c7fbd once the new .tar SDK is uploaded |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't yet have experience with this part of the code, left a few nits.
I arrived here while investigating an upgrade to Xcode 16.3 in #29415 (comment) and noticed a couple of potential issues around the argparse usage and the -o handling.
I have tested the following changes locally:
diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
index 426d82e46c..3bf9154887 100755
--- a/contrib/macdeploy/gen-sdk.py
+++ b/contrib/macdeploy/gen-sdk.py
@@ -20,12 +20,12 @@ def run():
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
- parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
- parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
+ parser.add_argument('xcode_app', metavar='XCODEAPP')
+ parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
args = parser.parse_args()
- xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
+ xcode_app = pathlib.Path(args.xcode_app).resolve()
assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
@@ -46,10 +46,10 @@ def run():
out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
if args.out_sdkt:
- out_sdkt_path = pathlib.Path(args.out_sdkt_path)
+ out_sdkt_path = pathlib.Path(args.out_sdkt)
else:
# Construct our own out_sdkt if not specified on the command line
- out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
+ out_sdkt_path = pathlib.Path(f"./{out_name}.tar")
def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
"""Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'This removes the unnecessary nargs=1 (so the positional is a scalar string instead of a 1-element list), fixes the -o code path to actually use args.out_sdkt, and updates the metavar to reflect that we now produce a .tar instead of .tar.gz.
Ran
python3 contrib/macdeploy/gen-sdk.py /Applications/Xcode.app -o /tmp/Xcode-sdk-test.tar
it shows:
Found Xcode (version: 26.1.1, build id: 17B100)
Found MacOSX SDK (version: 26.1, build id: 25B74)
Creating output .tar file...
Adding MacOSX SDK 26.1 files...
Done! Find the resulting tarball at:
/private/tmp/Xcode-sdk-test.tar
e7c22c8 to
266a2bf
Compare
Only include what we really need. Skip 100s of mb of manpages, swiftmodules, modulemaps. Note that System/Library is only needed for the Qt build.
Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output. For now, or until a better solution exists, disable compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit). See: https://docs.python.org/3/library/gzip.html#gzip.compress Co-authored-by: stickies-v <[email protected]>
This puts it in scope for the Python linters.
266a2bf to
3e01b5d
Compare
|
ACK 3e01b5d modulo the new .tar SDK being uploaded |
|
New tarball is now available. Open/close to kick the CI. |
|
Tested ACK 3e01b5d This fixes #31873.
Versions of python <3.9.0 are still nondeterministic but this is fine, since we have a minimum python >= 3.10. I bisected to this commit that fixed the non-determinism: python/cpython@674935b, prior to which python was incorrectly writing the device id of the target file to the tar header, which obviously varies between systems. Using the same methodology of testing this branch, I tested master and it exhibited inconsistent hashes depending on which version of zlib python was built with, as discussed here and in #31873.
master hashes
|
|
Guix Build (x86_64): cc72adc0e567f33605050a98cd7cd08db2b0c7da1aa11dd5f73b9077653b5e00 guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/SHA256SUMS.part
15d23cda22ba4afceddb733820893e4ef3a5307717002403cdc5e66d3561bafb guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu-debug.tar.gz
2d0e852259c802ea3c2411a38b93148dcaa2415b5d5d34d1f6362df39108e718 guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu.tar.gz
b9c4572af1a6312575f01414b0bb1530250c3c52da5dc53638134c399ca1048b guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/SHA256SUMS.part
ee2d60c4b260107b36933e66d7eb8c2a1b69f323661348ac461a66056d2522d4 guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf-debug.tar.gz
900d4bc2e57448087e3bdda96873d98eeceb9ac05d48ca33843eee3c2a60e2bc guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf.tar.gz
324e3036941d34dae95579ba7f4ca006514d2a7a99431deb8333c61d61e2a527 guix-build-3e01b5d0e7be/output/arm64-apple-darwin/SHA256SUMS.part
fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
8088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59 guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
1b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415 guix-build-3e01b5d0e7be/output/dist-archive/bitcoin-3e01b5d0e7be.tar.gz
e5758de2f2acc35d2ef953b80b1e8bf618e1e157f325b5d9bfa94f74e7be9c86 guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/SHA256SUMS.part
5800c2135a6148bba763cb68f23708ef7fc62c850dac507a544381cccba49fa9 guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu-debug.tar.gz
eae287e264b5e0ae88e0aee92a520e13aa00341266239a5ec257577f4c1a2d5c guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu.tar.gz
eafdaacbc9540f9e9706db281ec9f6db859e80dcb8e0e79efdc7e536799029d7 guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/SHA256SUMS.part
fba989ba1b4890e1161e98b05d6ebf53cd19aab5f5be44a0081cd9eed1499138 guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu-debug.tar.gz
c0e4f20087abb4ad0a58b06a7718913dd851ff46e092f95384618ad74cebff58 guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu.tar.gz
78a684a96bc9c0d901cc77601263277cf142cb20659b99451247f671b11afafc guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/SHA256SUMS.part
04713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18 guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
3ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3 guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176 guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
d95954c5818d7d8036c5f8353063739e2c1fe65dc40eb1224e00a3dc3f4b56b4 guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/SHA256SUMS.part
d8cf149a5d762da733f0efdcd67d09fbde8db601d1cd4d22ee57e87a2dcf6ce2 guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu-debug.tar.gz
ffc1fb0b72f22ae574628e15dbfcd52bc37d5c37bdfd1665ecc320cca4830eae guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu.tar.gz
ba812f2cd3b757c0096c6070e5bf072442a74a5614b1dfc92a8af39c6afbffdf guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/SHA256SUMS.part
4dac4c0f9bc0109e2beca6ec3c14d650538d1a3271bae92bb84a4d05612665ff guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-codesigning.tar.gz
06f08c02fc90e461ce837f4b7cc4d0a32ada0bc3a1ebec3eae4507b00e4ca8af guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-debug.zip
ef1ce4596f1d98f684604c58c8e207e2cb30858d90232ea7e3978e03c7545a85 guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-setup-unsigned.exe
301f9ee12bb95e109b0f278e3ffaace61c5c218effdf6dc475dbb358bf4116fc guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-unsigned.zip |
|
post-merge ACK 3e01b5d |
…356041e58d1 6356041e58d1 Merge bitcoin/bitcoin#33972: cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 7d7cb1bb48f1 Merge bitcoin/bitcoin#33971: cmake: Set `WITH_ZMQ` to `ON` in Windows presets fe1815d48f0c cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 49c672853503 cmake: Set `WITH_ZMQ` to `ON` in Windows presets f6acbef1084e Merge bitcoin/bitcoin#33764: ci: Add Windows + UCRT jobs for cross-compiling and native testing 808f1d972be3 Merge bitcoin/bitcoin#32009: contrib: turn off compression of macOS SDK to fix determinism (across distros) 4de26b111f4d Merge bitcoin/bitcoin#33514: ci: clear out space on CentOS, depends, gui GHA job 38c8474d0d77 Merge bitcoin/bitcoin#33914: Change Parse descriptor argument to string_view 4b25b274de66 Merge bitcoin/bitcoin#33951: test: check for output to stdout in `TestShell` test 52230a7f697f test: check for output to stdout in `TestShell` test 85d058dc537e Merge bitcoin/bitcoin#33946: interfaces: remove redundant mempool lock in ChainImpl::isInMempool() e07e57368e9f ci: clear out space on centos job 79d6e874e1da Merge bitcoin/bitcoin#32587: test: Fix reorg patterns in tests to use proper fork-based approach e249ea7da6c2 Merge bitcoin/bitcoin#33945: depends: latest config.guess & config.sub 3e01b5d0e7be contrib: rename gen-sdk to gen-sdk.py c1213a35abed macdeploy: disable compression in macOS gen-sdk script a33d03454508 contrib: more selectively pick files for macOS SDK 70d9e8f0a15d fix: reorg behaviour in mempool tests to match real one 540ed333f6c8 Move the create_empty_fork method to the test framework's blocktools.py module to enable reuse across multiple tests. 2909655fba91 fix: remove redundant mempool lock in ChainImpl::isInMempool() d5ed4ba9d862 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin 3e4355314b1a depends: latest config.sub 04eb84fe3f73 depends: latest config.guess b30262dcaa28 Merge bitcoin/bitcoin#33903: ci: Remove redundant busybox option 1a5f1eb08067 Merge bitcoin/bitcoin#33921: doc: clarify and cleanup macOS fuzzing notes 72cb8cef9778 Merge bitcoin/bitcoin#33862: txgraph: drop move assignment operator ade0397f59f2 txgraph: drop move assignment operator 5336bcd57849 Merge bitcoin/bitcoin#33855: kernel: add btck_block_tree_entry_equals 4f65a1c5db84 Merge bitcoin/bitcoin#33917: clang-format: Set Bitcoin Core IncludeCategories 902717b66dc1 Merge bitcoin/bitcoin#33918: depends: Update Qt download link 68ab2b65bfac Merge bitcoin/bitcoin#33919: ci: Run GUI unit tests in cross-Windows task 7e129b644ec9 Merge bitcoin/bitcoin#33893: test: add `-alertnotify` test for large work invalid chain warning 5fe753b56f45 Merge bitcoin/bitcoin#32655: depends: sqlite 3.50.4; switch to autosetup ff8c2f37497f Merge bitcoin/bitcoin#33932: ci: Use latest Xcode that the minimum macOS version allows fa283d28e261 Merge bitcoin/bitcoin#33629: Cluster mempool 2e27bd9c3af9 ci: Add Windows + UCRT jobs for cross-compiling and native testing 238c1c8933b1 Merge bitcoin-core/gui#914: Revert "gui, qt: brintToFront workaround for Wayland" 8343a9ffcc75 test: add `-alertnotify` test for large work invalid chain warning c34bc01b2ff2 doc: clarify and cleanup macOS fuzzing notes fa9537cde101 ci: Use latest Xcode that the minimum macOS version allows 17cf9ff7efdb Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general 315e43e5d86c Sanity check `GetFeerateDiagram()` in CTxMemPool::check() de2e9a24c40e test: extend package rbf functional test to larger clusters 4ef4ddb504e5 doc: update policy/packages.md for new package acceptance logic 79f73ad713a8 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() a86ac1176817 Update comments for CTxMemPool class 9567eaa66da8 Invoke TxGraph::DoWork() at appropriate times 0690514d4f72 Merge bitcoin/bitcoin#33770: init: Require explicit -asmap filename b2f88b53e0ec Merge bitcoin/bitcoin#33286: doc: update multisig tutorial to use multipath descriptors 313cdd2bfb71 Merge bitcoin/bitcoin#33915: test: Retry download in get_previous_releases.py bd130db994e2 ci: Rename items specific to Windows + MSVCRT 0672e727bf1d Revert "gui, qt: brintToFront workaround for Wayland" 17072f70051d Merge bitcoin/bitcoin#33912: clang-format: Set PackConstructorInitializers: CurrentLine fa7ea497c3ef ci: Run GUI unit tests in cross-Windows task fa0fee44a89c ci: Remove redundant busybox option fa102ec69fae doc: Shorten ci name fa7e222a2326 clang-format: Set Bitcoin Core IncludeCategories 222222378048 doc: Remove bash -c wrapper 6b2d17b13220 Merge bitcoin/bitcoin#33888: ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG ac71df43383a Merge bitcoin/bitcoin#33870: refactor: remove incorrect lifetimebounds 6cdb51c14eba Merge bitcoin/bitcoin#33887: doc: Improve CI docs on env and qemu-user-static 50cbde3295b4 depends: Update Qt download link 29c37651c74b Merge bitcoin/bitcoin#33880: test: Fix race condition in IPC interface block progation test 32368cd3e9f3 Merge bitcoin/bitcoin#33905: ci: Consistenly only cache on the default branch e55c49f85143 Merge bitcoin/bitcoin#33851: depends: update xcb-util packages to latest versions a07bd8415df4 Merge bitcoin/bitcoin#33824: ci: Enable experimental kernel stuff in most CI tasks via `dev-mode` c0bfe72f6e1f Change Parse descriptor argument to string_view f541b92cf2bb depends: expat 2.7.3 fad06f3bb436 test: retry download in get_previous_releases.py 2ebf4356e63d depends: libxcb 1.17.0 ba7ac870a32a depends: xcb_proto 1.17.0 fad0c76d0a10 clang-format: Set PackConstructorInitializers: CurrentLine 42d0692f9131 depends: libxcb-util-cursor 0.1.6 25b85919ab62 depends: libxcb 1.15 d129384ca97f depends: libxcb-util-wm 0.4.2 0b857ae9e555 depends: libxcb-util-renderutil 0.3.10 35e50488b25a depends: libxcb-util-keysyms 0.4.1 74b68ad28ba2 depends: libxcb-util-image 0.4.1 5bc0dde85d74 depends: libxcb-util 0.4.1 8d07292c286f depends: libXau 1.0.12 1af46cff9478 Merge bitcoin/bitcoin#33896: clang-format: Set InsertNewlineAtEOF: true 27ac11ea0a27 Merge bitcoin/bitcoin#33867: kernel: handle null or empty directories in implementation 2578e6fc0f4a test: Fix race condition in IPC interface block propagation test 288b8c30be42 doc: Drop (default: none) from -i2psam description 509dc91db143 Merge bitcoin/bitcoin#33026: test, refactor: Embedded ASMap [1/3]: Selected minor preparatory work b126f981943d Merge bitcoin-core/gui#910: Added test coverage for qt gui#901 console history filter 7d7b829c36b7 Merge bitcoin-core/gui#908: Remove HD seed reference from blank wallet tooltip 8558902e576e depends: Add patch for Windows11Style plugin 53b72372da91 Merge bitcoin/bitcoin#31734: miniscript: account for all `StringType` variants in `Miniscriptdescriptor::ToString()` a7f9bbe4c5e7 Merge bitcoin/bitcoin#32821: rpc: Handle -named argument parsing where '=' character is used 55555db055b5 doc: Add missing --platform=linux to docker build command fa0ce4c1486b ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG faa0973de296 ci: [refactor] Rename CIRRUS_PR env var to LINT_CI_IS_PR fa411f938e47 ci: Consistenly only cache on the default branch 6c5c44f77405 test: add functional test for new cluster mempool RPCs 72f60c877e00 doc: Update mempool_replacements.md to reflect feerate diagram checks 21693f031a53 Expose cluster information via rpc 72e74e0d4228 fuzz: try to add more code coverage for mempool fuzzing f107417490ab bench: add more mempool benchmarks 7976eb1ae77a Avoid violating mempool policy limits in tests 84de685cf7ee Stop tracking parents/children outside of txgraph 88672e205ba1 Rewrite GatherClusters to use the txgraph implementation 1ca4f01090cf Fix miniminer_tests to work with cluster limits 1902111e0f20 Eliminate CheckPackageLimits, which no longer does anything 3a646ec46264 Rework RBF and TRUC validation 19b8479868e5 Make getting parents/children a function of the mempool, not a mempool entry 5560913e51af Rework truc_policy to use descendants, not children a4458d6c4062 Use txgraph to calculate descendants c8b6f70d6492 Use txgraph to calculate ancestors 241a3e666b59 Simplify ancestor calculation functions b9cec7f0a1e0 Make removeConflicts private 0402e6c78080 Remove unused limits from CalculateMemPoolAncestors 08be765ac26a Remove mempool logic designed to maintain ancestor/descendant state fc4e3e6bc122 Remove unused members from CTxMemPoolEntry ff3b398d124b mempool: eliminate accessors to mempool entry ancestor/descendant cached state b9a2039f5122 Eliminate use of cached ancestor data in miniminer_tests and truc_policy ba09fc9774d5 mempool: Remove unused function CalculateDescendantMaximum 8e49477e86b3 wallet: Replace max descendant count with cluster_count e031085fd464 Eliminate Single-Conflict RBF Carve Out cf3ab8e1d0a2 Stop enforcing descendant size/count limits 89ae38f48965 test: remove rbf carveout test from mempool_limit.py c0bd04d18fdf Calculate descendant information for mempool RPC output on-the-fly bdcefb8a8b06 Use mempool/txgraph to determine if a tx has descendants 69e1eaa6ed22 Add test case for cluster size limits to TRUC logic 9cda64b86c59 Stop enforcing ancestor size/count limits 1f93227a84a5 Remove dependency on cached ancestor data in mini-miner 9fbe0a4ac26c rpc: Calculate ancestor data from scratch for mempool rpc calls 7961496dda2e Reimplement GetTransactionAncestry() to not rely on cached data feceaa42e8eb Remove CTxMemPool::GetSortedDepthAndScore 21b5cea588a7 Use cluster linearization for transaction relay sort order 6445aa7d9755 Remove the ancestor and descendant indices from the mempool 216e69372903 Implement new RBF logic for cluster mempool ff8f115dec6e policy: Remove CPFP carveout rule c3f1afc934e6 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits 47ab32fdb158 Select transactions for blocks based on chunk feerate dec138d1ddc7 fuzz: remove comparison between mini_miner block construction and miner 6c2bceb200aa bench: rewrite ComplexMemPool to not create oversized clusters 1ad4590f6385 Limit mempool size based on chunk feerate b11c89cab210 Rework miner_tests to not require large cluster limit 95a8297d481e Check cluster limits when using -walletrejectlongchains 95762e675959 Do not allow mempool clusters to exceed configured limits edb3e7cdf636 [test] rework/delete feature_rbf tests requiring large clusters 435fd5671116 test: update feature_rbf.py replacement test 34e32985e811 Add new (unused) limits for cluster size/count 838d7e355366 Add transactions to txgraph, but without cluster dependencies 552eb90071fd doc: CI - Describe qemu-user-static usage 2afbbddee550 doc: CI - Clarify how important `env -i` is and why 2444488f6ad3 Merge bitcoin/bitcoin#33894: net: Remove unused `local_socket_bytes` variable in `CConnman::GetAddresses()` fa1bf6818f09 clang-format: Set InsertNewlineAtEOF: true 115d298a9fa3 Merge bitcoin/bitcoin#33872: init: completely remove `-maxorphantx` option a90f3922ff7d Merge bitcoin/bitcoin#32419: psbt: clarify PSBT, PSBTInput, PSBTOutput unserialization flows 4d893c0f4605 net: Remove unused `local_socket_bytes` variable in `CConnman::GetAddresses()` fa1dacaebe5d ci: Move lint exec snippet to stand-alone py file ead849c9f177 Merge bitcoin/bitcoin#33886: test: Remove tests violating hardened std::span c03081fdb467 Merge bitcoin/bitcoin#33776: ci: Lint follow-ups fadb4f63cb0f test: Remove tests violating hardened std::span 6e2155816058 Merge bitcoin/bitcoin#33869: refactor: Avoid -W*-whitespace in git archive c8715aca95d0 Merge bitcoin/bitcoin#33247: build: Remove CMAKE_SKIP_BUILD_RPATH and SKIP_BUILD_RPATH settings ee5de407e369 Merge bitcoin/bitcoin#33537: guix: build `bitcoin-qt` with static libxcb & utils 024a7873500e Merge bitcoin/bitcoin#33876: doc: Update NetBSD Build Guide c66e98875439 Merge bitcoin/bitcoin#33865: cmake: Specify Windows plugin path in `test_bitcoin-qt` property c29eaeeaf937 doc: Update NetBSD Build Guide 7f318e1dd049 test: Add better coverage for Autofile size() e221b2524659 Merge bitcoin/bitcoin#33860: depends: drop Qt patches b7af960eb82f refactor: Add AutoFile::size ec0f75862e67 refactor: Modernize logging in util/asmap.cpp 606a251e0a31 tests: add unit test vectors for asmap interpreter 6657bcbdb4d0 kernel: allow null data_directory 0aebdac95da9 init: completely remove `-maxorphantx` option 99d012ec80a4 refactor: return reference instead of pointer f743e6c5dd38 refactor: add missing LIFETIMEBOUND annotation for parameter fa95353902b7 ci: Run macos tasks in a git archive, not git checkout 141117f5e8b4 refactor: remove incorrect LIFETIMEBOUND annotations fae3618fd6c8 ci: Annotate all check runs with the pull request number faf05d637d67 ci: Retry lint image building once after failure 96963b888e5a depends: static libxcb ad06843fab06 depends: avoid qdbusviewer in Qt build 6848ed56dc5f depends: apply Qt patches to fix static libxcb use dfde31f2ec1f Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h` 5f1b016bebd2 depends: static libxcb-util-image 98a2fbbe70b8 depends: static libxkbcommon 1412baf77295 depends: static libxcb-util-wm a4009dadf466 depends: static libxcb-keysyms bcfb8679b3ba depends: static libxcb-render-util faf99ae37963 refactor: Avoid -W*-whitespace in git archive 2594d5a189e5 build: Remove CMAKE_SKIP_BUILD_RPATH and SKIP_BUILD_RPATH settings 310e4979b36c qt: Added test coverage for qt gui#901 console history filter 0dd8d5c237e2 cmake: Specify Windows plugin path in `test_bitcoin-qt` property b0a38871546d scripted-diff: fix leftover references to `policy/fees.h` 48d4b936e09f Merge bitcoin/bitcoin#33511: init: Fix Ctrl-C shutdown hangs during wait calls 3c3c6adb7260 Merge bitcoin/bitcoin#33745: mining: check witness commitment in submitBlock e652b69b8da4 Merge bitcoin/bitcoin#33003: test: add option to skip large re-org test in feature_block 3e9aca6f1b52 depends: drop qtbase-moc-ignore-gcc-macro.patch qt patch fac4f6de28e7 ci: Rewrite lint task Bash snippet to Python fa0d37a57985 ci: Rewrite Bash to check inputs to Python 0da5a82700e9 depends: drop unused qt patch fae83611b8ef ci: [refactor] Use --preset=dev-mode in mac_native task fadb67b4b4e1 ci: [refactor] Base nowallet task on --preset=dev-mode 6666980e8653 ci: Enable bitcoin-chainstate and test_bitcoin-qt in win64 task 096924d39d64 kernel: add btck_block_tree_entry_equals faff7b231246 ci: Enable experimental kernel stuff in i686 task fa1632eecf58 ci: Enable experimental kernel stuff in mac-cross tasks fad10ff7c923 ci: Enable experimental kernel stuff in armhf task fa9d67c13d0d ci: Enable experimental kernel stuff in Alpine task fab3fb83026e ci: Enable experimental kernel stuff in s390x task fa7da8a646ed ci: Enable experimental kernel stuff in valgrind task fa9c2973d60b ci: Enable experimental kernel stuff in TSan task fad30d439502 ci: Enable experimental kernel stuff in MSan task d5ed9cb3eb52 Add accessor for sigops-adjusted weight 1bf3b513966e Add sigops adjusted weight calculator c18c68a950d3 Create a txgraph inside CTxMemPool 29a94d5b2f26 Make CTxMemPoolEntry derive from TxGraph::Ref 92b0079fe386 Allow moving CTxMemPoolEntry objects, disallow copying f6ec3519a330 init: Require explicit -asmap filename 6eaa00fe2020 test: clarify submitBlock() mutates the template 862bd432837e mining: ensure witness commitment check in submitBlock 00d1b6ef4b12 doc: clarify UpdateUncommittedBlockStructures 929f69d0ff29 qt: Remove HD seed reference from blank wallet tooltip 1db74914706f depends: sqlite 3.50.4 286f3e49c84c guix: sqlite wants tcl de7c3587cd45 doc: Update add checksum instructions in tutorial 6c73e4744837 mempool: Store iterators into mapTx in mapNextTx 51430680ecb7 Allow moving an Epoch::Marker 2a46e94a1600 doc: Update multisig-tutorial.md to use multipath descriptors c25a5e670b27 init: Signal m_tip_block_cv on Ctrl-C f53dbbc5057b test: Add functional tests for named argument parsing 694f04e2bd34 rpc: Handle -named argument parsing where '=' character is used 6a29f79006a9 test: Test SIGTERM handling during waitforblockheight call 8810642b571e test: add option to skip large re-org test in feature_block d31158d3646f psbt: clarify PSBT, PSBTInput, PSBTOutput unserialization flows 28a4fcb03c0f test: check listdescriptors do not return a mix of hardened derivation marker 975783cb79e9 descriptor: account for all StringType in MiniscriptDescriptor::ToStringHelper() git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 6356041e58d1ba86695e2e7c219c68ee5abe583f
Only include what we really need. Skip 100s of mb of manpages, swiftmodules, modulemaps. Note that System/Library is only needed for the Qt build. Github-Pull: bitcoin#32009 Rebased-From: a33d034
Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output. For now, or until a better solution exists, disable compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit). See: https://docs.python.org/3/library/gzip.html#gzip.compress Co-authored-by: stickies-v <[email protected]> Github-Pull: bitcoin#32009 Rebased-From: c1213a3
This puts it in scope for the Python linters. Github-Pull: bitcoin#32009 Rebased-From: 3e01b5d
|
Backported to 30.x in #33997. |
d59ebac doc: update manual pages for v30.1rc1 (fanquake) b83d4f7 build: bump version to v30.1rc1 (fanquake) b26c93a doc: update release notes for v30.1rc1 (fanquake) 18f3ada Remove unreliable seed from chainparams.cpp, and the associated README (SatsAndSports) 187e3b8 [test] wallet send 3 generation TRUC (glozow) ab58b2c [wallet] never try to spend from unconfirmed TRUC that already has ancestors (glozow) d872277 contrib: fix manpage generation (fanquake) 2256f89 contrib: rename gen-sdk to gen-sdk.py (fanquake) 5408e85 macdeploy: disable compression in macOS gen-sdk script (fanquake) 338570d contrib: more selectively pick files for macOS SDK (fanquake) Pull request description: Backports: * #32009 * #33528 * #33723 * #33996 Prior `30.x` backports in #33609. ACKs for top commit: hebasto: re-ACK d59ebac. marcofleon: ACK d59ebac Tree-SHA512: 41656dc7c2e7a023325b288ff38cf95877a3c223ed5c6defc1c0939b725fb5423455583881f283aff14821bc4fea5a4d4d0ac458cab6450eb9b1bc7730fc027c
938d7aacab Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage 597b8be223 Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling d155fc12a0 Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate 07135290c1 rest: allow reading partial block data from storage 4e2af1c065 blockstorage: allow reading partial block data from storage f2fd1aa21c blockstorage: return an error code from `ReadRawBlock()` 5be20c380d Merge bitcoin/bitcoin#34033: scripted-diff: Unify error and warning log formatting b31f786695 Merge bitcoin/bitcoin#34045: test: Log IP of download server in get_previous_releases.py b26762bdcb Merge bitcoin/bitcoin#33805: merkle: migrate `path` arg to reference and drop unused args 0f6d8a347a Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs c2975f26d6 Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation cdaf25f9c3 test: Log IP of download server in get_previous_releases.py c1f0a89d9c Merge bitcoin/bitcoin#34040: test: Detect truncated download in get_previous_releases.py fa75480c84 test: Detect truncated download in get_previous_releases.py 56ce78d5f6 Merge bitcoin/bitcoin#34031: net: Remove "tor" as a network specification 500862b2d4 Merge bitcoin/bitcoin#33423: qa: Improvements to debug_assert_log + busy_wait_for_debug_log 5f5c1ea019 net: Cache -capturemessages setting cca113f5b0 Merge bitcoin/bitcoin#34008: log: don't rate-limit "new peer" with -debug=net 2c44c41984 Merge bitcoin/bitcoin#33553: validation: Improve warnings in case of chain corruption 6eb5ba5691 refactor: extract shared `SipHash` state into `SipHashState` 118d22ddb4 optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs` 9ca52a4cbe optimization: migrate `SipHashUint256` to `PresaltedSipHasher` ec11b9fede optimization: introduce `PresaltedSipHasher` for repeated hashing d23d49ee3f Merge bitcoin/bitcoin#31823: tests: Add witness commitment if we have a witness transaction in `FullBlockTest.update_block()` 20330548cf refactor: extract `SipHash` C0-C3 constants to class scope 9f9eb7fbc0 test: rename k1/k2 to k0/k1 in `SipHash` consistency tests 29ed608dc7 Merge bitcoin/bitcoin#33961: script: Add a separate ScriptError for empty pubkeys encountered in Tapscript d2a199bca7 Merge bitcoin/bitcoin#33909: doc, ci: Make the max number of commits tested explicit dbc8928069 Merge bitcoin/bitcoin#33993: init: point out -stopatheight may be imprecise d4d184eda9 log: don't rate-limit "new peer" with -debug=net e7ac5a133c doc: add release note for 34031 c4c70a256e netbase: Remove "tor" as a network specification fa89f60e31 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning() fa6c7a1954 scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug() d5c8199b79 Merge bitcoin/bitcoin#34006: Add util::Expected (std::expected) 77248e8496 Merge bitcoin/bitcoin#33771: refactor: C++20 operators 36073d56db Merge bitcoin/bitcoin#33952: depends: update freetype and document remaining `bitcoin-qt` runtime libs f09ae5f96f Merge bitcoin/bitcoin#33950: guix: reduce allowed exported symbols cea443e246 net: Pass time to InactivityChecks fuctions 89dc82295e Merge bitcoin/bitcoin#29641: scripted-diff: Use LogInfo over LogPrintf eb19a2dac5 Merge bitcoin/bitcoin#34017: fuzz: Add a test case for `ParseByteUnits()` faa23738fc refactor: Enable clang-tidy bugprone-unused-return-value fa114be27b Add util::Expected (std::expected) e68517208b Merge bitcoin/bitcoin#33995: depends: Propagate native C compiler to `sqlite` package 091cae6fdf Merge bitcoin/bitcoin#33939: contrib: Count entry differences in asmap-tool diff summary 57b888ce0e fuzz: Add a test case for `ParseByteUnits()` b8e66b901d Merge bitcoin/bitcoin#33858: test: add unit test coverage for the empty leaves path in MerkleComputation 0c9ab0f8f8 Merge bitcoin/bitcoin#33956: net: fix use-after-free with v2->v1 reconnection logic fa4395dffd refactor: Remove unused LogPrintf fa05181d90 scripted-diff: LogPrintf -> LogInfo 9890058b37 Merge bitcoin/bitcoin#33723: chainparams: remove dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us 9e02f78089 Merge bitcoin/bitcoin#33774: cmake: Move IPC tests to `ipc/test` ad452a1e65 Merge bitcoin/bitcoin#33528: wallet: don't consider unconfirmed TRUC coins with ancestors ff06e2468a init: point out -stopatheight may be imprecise 9a29b2d331 Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 69e66efe45 Merge bitcoin/bitcoin#32882: index: remove unnecessary locator cleaning in BaseIndex::Init() 6581ac5d9f Merge bitcoin/bitcoin#33996: contrib: fix manpage generation 39ca015259 Merge bitcoin/bitcoin#33140: test: Avoid shutdown race in NetworkThread e9536faaee contrib: fix manpage generation bcf794d5f3 Merge bitcoin/bitcoin#30455: test: assumeutxo: add missing tests in wallet_assumeutxo.py af0e6a65c9 Merge bitcoin/bitcoin#33702: contrib: Remove brittle, confusing and redundant UTF8 encoding from Python IO 4b47113698 validation: Reword CheckForkWarningConditions and call it also during IBD and at startup 2f51951d03 p2p: Add warning message when receiving headers for blocks cached as invalid 4c784b25c4 Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight 710031ebef Revert "guix: sqlite wants tcl" 4cf5ea6c3d depends: Propagate native C compiler to `sqlite` package ce771726f3 Merge bitcoin/bitcoin#33960: log: Use more severe log level (warn/err) where appropriate cb7d5bfe4a test, assumeutxo: loading a wallet (backup) on a pruned node 7a365244f8 test, refactor snapshot import and background validation e0ba6bbed9 Merge bitcoin/bitcoin#33591: Cluster mempool followups b8d279a81c doc: add comment to explain correctness of GatherClusters() aba7500a30 Fix parameter name in getmempoolcluster rpc 6c1325a091 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology bc2eb931da Require mempool lock to be held when invoking TRUC checks 957ae23241 Improve comments for getTransactionAncestry to reference cluster counts instead of descendants d97d6199ce Fix comment to reference cluster limits, not chain limits a1b341ef98 Sanity check feerate diagram in CTxMemPool::check() 23d6f457c4 rpc: improve getmempoolcluster output d2dcd37aac Avoid using mapTx.modify() to update modified fees d84ffc24d2 doc: add release notes snippet for cluster mempool b0417ba944 doc: Add design notes for cluster mempool and explain new mempool limits 804329400a fuzz: gate mempool entry based on weight 6356041e58 Merge bitcoin/bitcoin#33972: cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 7d7cb1bb48 Merge bitcoin/bitcoin#33971: cmake: Set `WITH_ZMQ` to `ON` in Windows presets 2d88966e43 miner: replace "package" with "chunk" 6f3e8eb300 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler b5f245f6f2 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB 1dac54d506 Use cluster size limit instead of ancestor size limit in txpackage unit test 04f65488ca Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits 634291a7dc Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits fc18ef1f3f Remove ancestor and descendant vsize limits from MemPoolLimits ed8e819121 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect 80d8df2d47 Invoke removeUnchecked() directly in removeForBlock() 9292570f4c Rewrite GetChildren without sets 3e39ea8c30 Rewrite removeForReorg to avoid using sets a3c31dfd71 scripted-diff: rename AddToMempool -> TryAddToMempool a5a7905d83 Simplify removeRecursive 01d8520038 Remove unused argument to RemoveStaged fe1815d48f cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB` 49c6728535 cmake: Set `WITH_ZMQ` to `ON` in Windows presets ec8eb013a9 doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md` 48496caa12 ci: Remove redundant `DEP_OPTS` from “Windows-cross UCRT” job f6acbef108 Merge bitcoin/bitcoin#33764: ci: Add Windows + UCRT jobs for cross-compiling and native testing b5a7a685bb ci: Make the max number of commits tested explicit 9d5021a05b script: add SCRIPT_ERR_TAPSCRIPT_EMPTY_PUBKEY 7b90b4f5bb guix: reduce allowed exported symbols 41e657aacf guix: add bitcoin-qt runtime libs doc in symbol-check ef4ce19a15 depends: freetype 2.11.1 808f1d972b Merge bitcoin/bitcoin#32009: contrib: turn off compression of macOS SDK to fix determinism (across distros) 4de26b111f Merge bitcoin/bitcoin#33514: ci: clear out space on CentOS, depends, gui GHA job fa45a1503e log: Use LogWarning for non-critical logs fa0018d011 log: Use LogError for fatal errors 22229de728 doc: Fix typo in init log 38c8474d0d Merge bitcoin/bitcoin#33914: Change Parse descriptor argument to string_view 4b25b274de Merge bitcoin/bitcoin#33951: test: check for output to stdout in `TestShell` test 167df7a98c net: fix use-after-free with v2->v1 reconnection logic 52230a7f69 test: check for output to stdout in `TestShell` test 85d058dc53 Merge bitcoin/bitcoin#33946: interfaces: remove redundant mempool lock in ChainImpl::isInMempool() fd4ce55121 contrib: Count entry differences in asmap-tool diff summary e07e57368e ci: clear out space on centos job 79d6e874e1 Merge bitcoin/bitcoin#32587: test: Fix reorg patterns in tests to use proper fork-based approach e249ea7da6 Merge bitcoin/bitcoin#33945: depends: latest config.guess & config.sub 3e01b5d0e7 contrib: rename gen-sdk to gen-sdk.py c1213a35ab macdeploy: disable compression in macOS gen-sdk script a33d034545 contrib: more selectively pick files for macOS SDK fad6118586 test: Fix "typo" in written invalid content fab085c15f contrib: Use text=True in subprocess over manual encoding handling fa71c15f86 scripted-diff: Bump copyright headers after encoding changes fae612424b contrib: Remove confusing and redundant encoding from IO fa7d72bd1b lint: Drop check to enforce encoding to be specified in Python scripts faf39d8539 test: Clarify that Python UTF-8 mode is the default today for most systems fa83e3a81d lint: Do not allow locale dependent shell scripts 70d9e8f0a1 fix: reorg behaviour in mempool tests to match real one 540ed333f6 Move the create_empty_fork method to the test framework's blocktools.py module to enable reuse across multiple tests. 2909655fba fix: remove redundant mempool lock in ChainImpl::isInMempool() d5ed4ba9d8 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin 3e4355314b depends: latest config.sub 04eb84fe3f depends: latest config.guess b30262dcaa Merge bitcoin/bitcoin#33903: ci: Remove redundant busybox option 1a5f1eb080 Merge bitcoin/bitcoin#33921: doc: clarify and cleanup macOS fuzzing notes 72cb8cef97 Merge bitcoin/bitcoin#33862: txgraph: drop move assignment operator bc64013e6f Remove unused variable (cacheMap) in mempool ade0397f59 txgraph: drop move assignment operator 5336bcd578 Merge bitcoin/bitcoin#33855: kernel: add btck_block_tree_entry_equals 4f65a1c5db Merge bitcoin/bitcoin#33917: clang-format: Set Bitcoin Core IncludeCategories 902717b66d Merge bitcoin/bitcoin#33918: depends: Update Qt download link 68ab2b65bf Merge bitcoin/bitcoin#33919: ci: Run GUI unit tests in cross-Windows task 7e129b644e Merge bitcoin/bitcoin#33893: test: add `-alertnotify` test for large work invalid chain warning 5fe753b56f Merge bitcoin/bitcoin#32655: depends: sqlite 3.50.4; switch to autosetup ff8c2f3749 Merge bitcoin/bitcoin#33932: ci: Use latest Xcode that the minimum macOS version allows fa283d28e2 Merge bitcoin/bitcoin#33629: Cluster mempool 2e27bd9c3a ci: Add Windows + UCRT jobs for cross-compiling and native testing 238c1c8933 Merge bitcoin-core/gui#914: Revert "gui, qt: brintToFront workaround for Wayland" 8343a9ffcc test: add `-alertnotify` test for large work invalid chain warning c34bc01b2f doc: clarify and cleanup macOS fuzzing notes fa9537cde1 ci: Use latest Xcode that the minimum macOS version allows 17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general 315e43e5d8 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() de2e9a24c4 test: extend package rbf functional test to larger clusters 4ef4ddb504 doc: update policy/packages.md for new package acceptance logic 79f73ad713 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() a86ac11768 Update comments for CTxMemPool class 9567eaa66d Invoke TxGraph::DoWork() at appropriate times bd130db994 ci: Rename items specific to Windows + MSVCRT 0672e727bf Revert "gui, qt: brintToFront workaround for Wayland" fa7ea497c3 ci: Run GUI unit tests in cross-Windows task fa0fee44a8 ci: Remove redundant busybox option fa102ec69f doc: Shorten ci name fa7e222a23 clang-format: Set Bitcoin Core IncludeCategories 2222223780 doc: Remove bash -c wrapper 50cbde3295 depends: Update Qt download link c0bfe72f6e Change Parse descriptor argument to string_view 8558902e57 depends: Add patch for Windows11Style plugin 6c5c44f774 test: add functional test for new cluster mempool RPCs 72f60c877e doc: Update mempool_replacements.md to reflect feerate diagram checks 21693f031a Expose cluster information via rpc 72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing f107417490 bench: add more mempool benchmarks 7976eb1ae7 Avoid violating mempool policy limits in tests 84de685cf7 Stop tracking parents/children outside of txgraph 88672e205b Rewrite GatherClusters to use the txgraph implementation 1ca4f01090 Fix miniminer_tests to work with cluster limits 1902111e0f Eliminate CheckPackageLimits, which no longer does anything 3a646ec462 Rework RBF and TRUC validation 19b8479868 Make getting parents/children a function of the mempool, not a mempool entry 5560913e51 Rework truc_policy to use descendants, not children a4458d6c40 Use txgraph to calculate descendants c8b6f70d64 Use txgraph to calculate ancestors 241a3e666b Simplify ancestor calculation functions b9cec7f0a1 Make removeConflicts private 0402e6c780 Remove unused limits from CalculateMemPoolAncestors 08be765ac2 Remove mempool logic designed to maintain ancestor/descendant state fc4e3e6bc1 Remove unused members from CTxMemPoolEntry ff3b398d12 mempool: eliminate accessors to mempool entry ancestor/descendant cached state b9a2039f51 Eliminate use of cached ancestor data in miniminer_tests and truc_policy ba09fc9774 mempool: Remove unused function CalculateDescendantMaximum 8e49477e86 wallet: Replace max descendant count with cluster_count e031085fd4 Eliminate Single-Conflict RBF Carve Out cf3ab8e1d0 Stop enforcing descendant size/count limits 89ae38f489 test: remove rbf carveout test from mempool_limit.py c0bd04d18f Calculate descendant information for mempool RPC output on-the-fly bdcefb8a8b Use mempool/txgraph to determine if a tx has descendants 69e1eaa6ed Add test case for cluster size limits to TRUC logic 9cda64b86c Stop enforcing ancestor size/count limits 1f93227a84 Remove dependency on cached ancestor data in mini-miner 9fbe0a4ac2 rpc: Calculate ancestor data from scratch for mempool rpc calls 7961496dda Reimplement GetTransactionAncestry() to not rely on cached data feceaa42e8 Remove CTxMemPool::GetSortedDepthAndScore 21b5cea588 Use cluster linearization for transaction relay sort order 6445aa7d97 Remove the ancestor and descendant indices from the mempool 216e693729 Implement new RBF logic for cluster mempool ff8f115dec policy: Remove CPFP carveout rule c3f1afc934 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits 47ab32fdb1 Select transactions for blocks based on chunk feerate dec138d1dd fuzz: remove comparison between mini_miner block construction and miner 6c2bceb200 bench: rewrite ComplexMemPool to not create oversized clusters 1ad4590f63 Limit mempool size based on chunk feerate b11c89cab2 Rework miner_tests to not require large cluster limit 95a8297d48 Check cluster limits when using -walletrejectlongchains 95762e6759 Do not allow mempool clusters to exceed configured limits edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters 435fd56711 test: update feature_rbf.py replacement test 34e32985e8 Add new (unused) limits for cluster size/count 838d7e3553 Add transactions to txgraph, but without cluster dependencies a7c96f874d tests: Add witness commitment if we have a witness transaction in FullBlockTest.update_block() 096924d39d kernel: add btck_block_tree_entry_equals ffcae82a68 test: exercise TransactionMerklePath with empty block; targets the MerkleComputation empty-leaves path that was only reached by fuzz tests d5ed9cb3eb Add accessor for sigops-adjusted weight 1bf3b51396 Add sigops adjusted weight calculator c18c68a950 Create a txgraph inside CTxMemPool 29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref 92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying 24ed820d4f merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` 63d640fa6a merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` be270551df merkle: migrate `path` arg of `MerkleComputation` to a reference 866bbb98fd cmake, test: Improve locality of `bitcoin_ipc_test` library description ae2e438b25 cmake: Move IPC tests to `ipc/test` 48840bfc2d refactor: Prefer `<=>` over multiple relational operators 5a0f49bd26 refactor: Remove all `operator!=` definitions 0ac969cddf validation: don't reallocate cache for short-lived CCoinsViewCache c8f5e446dc coins: reduce lookups in dbcache layer propagation 1db7491470 depends: sqlite 3.50.4 286f3e49c8 guix: sqlite wants tcl b0c706795c Remove unreliable seed from chainparams.cpp, and the associated README 6c73e47448 mempool: Store iterators into mapTx in mapNextTx 51430680ec Allow moving an Epoch::Marker dcd42d6d8f [test] wallet send 3 generation TRUC e753fadfd0 [wallet] never try to spend from unconfirmed TRUC that already has ancestors fa6db79302 test: Avoid shutdown race in NetworkThread a1f7623020 qa: Only complain about expected messages that were not found 1e54125e2e refactor(qa): Avoid unnecessary string operations a9021101dc qa: Replace always-escaped regexps with "X in Y" 5c16e4631c doc: Remove no longer correct comment facd01e6ff refactor: remove redundant locator cleanup in BaseIndex::Init() c1e554d3e5 refactor: consolidate 3 separate locks into one block 41479ed1d2 test: add test for periodic flush inside ActivateBestChain 84820561dc validation: periodically flush dbcache during reindex-chainstate git-subtree-dir: depend/bitcoin git-subtree-split: 938d7aacabd0bb3784bb3e529b1ed06bb2891864
This includes three changes. The first is to more selectively pick files for inclusion into our macOS SDK tarball (skip manpages, binaries etc), which is nice because it redues the size of the tarball (from ~80mb to 20mb), and makes the size increase that happens with the next commit, less-bad.
The second change removes compression of the tarball. Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output.
For now, or until a better solution exists, remove compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit).
See: https://docs.python.org/3/library/gzip.html#gzip.compress
The third renames
gen-sdktogen-sdk.py, so that it will be linted, along with the rest of our Python files.Fixes #31873. We could probably also put this into 30.x.