Skip to content

Conversation

@fanquake
Copy link
Member

@fanquake fanquake commented Mar 6, 2025

This includes three changes. The first is to more selectively pick files for inclusion into our macOS SDK tarball (skip manpages, binaries etc), which is nice because it redues the size of the tarball (from ~80mb to 20mb), and makes the size increase that happens with the next commit, less-bad.

The second change removes compression of the tarball. Starting with Python 3.11, Pythons gzip might delegate to zlib. Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib implementation might differ, resulting in different output.

For now, or until a better solution exists, remove compression. This results in the SDK increasing in size to ~157mb. Which is not unreasonable, to regain determinism (and would be significantly worse without the previous commit).

See: https://docs.python.org/3/library/gzip.html#gzip.compress

The third renames gen-sdk to gen-sdk.py, so that it will be linted, along with the rest of our Python files.

Fixes #31873. We could probably also put this into 30.x.

@DrahtBot
Copy link
Contributor

DrahtBot commented Mar 6, 2025

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32009.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK stickies-v, davidgumberg
Concept ACK hebasto, willcl-ark, laanwj

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

@achow101
Copy link
Member

achow101 commented Mar 6, 2025

Is determinism of that tarball necessary? IIRC it doesn't actually affect the build.

@fanquake
Copy link
Member Author

fanquake commented Mar 6, 2025

Is determinism of that tarball necessary?

I don't see why we wouldn't want it. If we don't care about it, we should remove: "The sha256sum should be c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d." So that issues like #31873 aren't opened.

@davidgumberg
Copy link
Contributor

Concept ACK, while it might not be strictly necessary for this tarball to be deterministic, I think matching hashes of tarballs is the simplest way to verify that you've followed the same procedure as others to build the macOS sdk and have gotten the same result.

150MB seems to me a small price to pay to fix #31873 and reduce surface area for nondeterminism in generating the macOS sdk.

@hebasto
Copy link
Member

hebasto commented Mar 7, 2025

Concept ACK for the same reasoning as in #32009 (comment).

@willcl-ark
Copy link
Member

Concept ACK.

I don't think a few hundred MB more disk space required for (power-)users doing guix builds will be a great concern to anyone, and I agree with @davidgumberg that having as many steps in the guix workflow as possible be reproducible, is probably best.

@fanquake fanquake changed the title RFC contrib: turn off compression of macOS SDK to fix determinism (across distros) contrib: turn off compression of macOS SDK to fix determinism (across distros) Mar 7, 2025
@laanwj
Copy link
Member

laanwj commented Mar 8, 2025

Concept ACK, imo determinism is good because it makes it possible to check if mistakes have been made in the process at earlier steps.

One comment i was about to give is "why use a .tar.gz instead of .tar, that's the only way to be sure to remove dependency on zlib", but that's a much more spread out code change. This is most likely fine.

edit: Could do that when we have to upgrade the MacOS SDK anyway, but definitely not for 29.0

@Sjors
Copy link
Member

Sjors commented Mar 10, 2025

I'm not too worried about the size increase. A typical guix build eats 30GB on my Ubuntu VM (3 GB for the outputs).

it makes it possible to check if mistakes have been made in the process at earlier steps.

Agreed.

It seems simple enough to backport, especially because we're not bumping the version.

I'll see if I can reproduce the new hashes.

@Sjors
Copy link
Member

Sjors commented Mar 10, 2025

One slight problem is that Apple no longer makes XCode 15.0 available for download. They do have Xcode 15.0.1: https://download.developer.apple.com/Developer_Tools/Xcode_15.0.1/Xcode_15.0.1.xip

I still have the original xip on one of my machines, I'll upload it somewhere for testing. But we should probably bump it.

Here you go: https://download.sprovoost.nl/download.php?id=21&token=ee9af1cfea8fa495cc842a407249126d

(sha256 checksum is in contrib/macdeploy/README.md)

Update: XCode 15 is still there, see below.


I was able to reproduce the hash for the first commit 6998e933f935a379c3ad55c2fb16eca9b854f40b, but not for the second commit 20778eb0235df70397fc285f9e3b72270bd4aaf4. I get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 instead on macOS 15.3.1, Python 3.10.14. Ditto on my Ubuntu 24.04 VM inside the same machine.


It looks like Apple Silicon has hardware acceleration for zip, maybe that's a factor? (and would also explain why extract_xcode.py is super fast on my VM running on the M4 mac, while super slow on the AMD Ryzen 7950x). (probably not, because the first commit did produce identical results)


I also get 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82 for Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz on the AMD Ubuntu machine.

@davidgumberg
Copy link
Contributor

@Sjors
Copy link
Member

Sjors commented Mar 11, 2025

@davidgumberg ah indeed, I see it as well. You have to click on the download link in order to for the download URL to work at all, so that's probably why it didn't work for me.

So that just leaves the hash mismatch to figure out.

@willcl-ark
Copy link
Member

At commit 20778eb0235df70397fc285f9e3b72270bd4aaf4 I get the following guix output:

env HOSTS="x86_64-apple-darwin arm64-apple-darwin" ./contrib/guix/guix-build

<snip>

$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
30aa32906f879c5347be27bd9cb1a65b3d839acd72805fe3185276f788971fdb  guix-build-20778eb0235d/output/arm64-apple-darwin/SHA256SUMS.part
0763a089bcc5d1d22191414f556bf461cefa8ea8a18971b171cf6fc5570b8790  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
18cd712ac4cbd63122483a083a9f29d7c184ffaead45078fb27b3d6381b691a6  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
c3fdc3b50077f2a4b772d4c1cf50e230f358542844e800e77194bf85dee9eb28  guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.zip
b97da95ff464b8bdcb9ce31dc783f947dd24d93eb6bfdfea71c3070d51991058  guix-build-20778eb0235d/output/dist-archive/bitcoin-20778eb0235d.tar.gz
e180a9ef657ea3f0a6e42c4babe7b38a8e585902fbac301a1f16bccb0c53a18b  guix-build-20778eb0235d/output/x86_64-apple-darwin/SHA256SUMS.part
5b032a2b6dd2dad11f3884194bcb85aea37237246162dfad27099e6c5175071d  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
39d23ec95f34cd7d8bbdb6228e55f7f8a0f48a3997c3dcb146a2ff3744f37629  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz
76e36fa1af2f788dea631a7fcc93451fdf794a40dded0e3c7c12f74ab136aefb  guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.zip

$ eza -al guix-build-20778eb0235d/output/*/*apple*.tar.gz
.rw-r--r-- 52M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-codesigning.tar.gz
.rw-r--r-- 35M will 19 Mar 13:02 guix-build-20778eb0235d/output/arm64-apple-darwin/bitcoin-20778eb0235d-arm64-apple-darwin-unsigned.tar.gz
.rw-r--r-- 56M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-codesigning.tar.gz
.rw-r--r-- 38M will 19 Mar 12:55 guix-build-20778eb0235d/output/x86_64-apple-darwin/bitcoin-20778eb0235d-x86_64-apple-darwin-unsigned.tar.gz


@davidgumberg
Copy link
Contributor

davidgumberg commented Mar 22, 2025

I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

pkgdiff again reports the mismatching archives as identical. But, now that the archives are uncompressed, the binary diff output is a little more revealing, here's the start of the diff:

git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

@@ -2050,7 +2050,7 @@
 00008010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00008020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00008030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-00008040: 0000 0000 0000 0000 00a0 815f 7e58 636f  ..........._~Xco
+00008040: 0000 0000 0000 0000 0000 d0ff 2f58 636f  ............/Xco
 00008050: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
 00008060: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
 00008070: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
@@ -4124,30 +4124,30 @@
 000101b0: 6129 2041 5050 4c45 5f41 5243 4849 5645  a) APPLE_ARCHIVE
 000101c0: 5f53 5749 4654 5f50 5249 5641 5445 3b0a  _SWIFT_PRIVATE;.
 000101d0: 0a23 6966 6465 6620 5f5f 6370 6c75 7370  .#ifdef __cplusp
-000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 608e  lus.}.#endif..`.
-000101f0: 9f71 0000 0000 0000 0000 0000 0000 0000  .q..............
+000101e0: 6c75 730a 7d0a 2365 6e64 6966 0a00 0000  lus.}.#endif....
+000101f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00010200: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00010210: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00010220: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00010230: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-00010240: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-00010250: 0000 5863 6f64 652d 3135 2e30 2d31 3541  ..Xcode-15.0-15A
-00010260: 3234 3064 2d65 7874 7261 6374 6564 2d53  240d-extracted-S
-00010270: 444b 2d77 6974 682d 6c69 6263 7878 2d68  DK-with-libcxx-h
-00010280: 6561 6465 7273 2f75 7372 2f69 6e63 6c75  eaders/usr/inclu
-00010290: 6465 2f41 7070 6c65 4172 6368 6976 652f  de/AppleArchive/
-000102a0: 4141 456e 7472 7958 4154 426c 6f62 2e68  AAEntryXATBlob.h
-000102b0: 0000 0000 0000 3030 3030 3634 3400 3030  ......0000644.00
-000102c0: 3030 3030 3000 3030 3030 3030 3000 3030  00000.0000000.00
-000102d0: 3030 3030 3135 3437 3400 3030 3030 3030  000015474.000000
-000102e0: 3030 3030 3000 3032 3631 3631 0020 3000  00000.026161. 0.
+00010240: 0000 0000 0000 0000 0000 0000 0058 636f  .............Xco
+00010250: 6465 2d31 352e 302d 3135 4132 3430 642d  de-15.0-15A240d-
+00010260: 6578 7472 6163 7465 642d 5344 4b2d 7769  extracted-SDK-wi
+00010270: 7468 2d6c 6962 6378 782d 6865 6164 6572  th-libcxx-header
+00010280: 732f 7573 722f 696e 636c 7564 652f 4170  s/usr/include/Ap
+00010290: 706c 6541 7263 6869 7665 2f41 4145 6e74  pleArchive/AAEnt
+000102a0: 7279 5841 5442 6c6f 622e 6800 0000 0000  ryXATBlob.h.....
+000102b0: 0030 3030 3036 3434 0030 3030 3030 3030  .0000644.0000000
+000102c0: 0030 3030 3030 3030 0030 3030 3030 3031  .0000000.0000001
+000102d0: 3534 3734 0030 3030 3030 3030 3030 3030  5474.00000000000
+000102e0: 0030 3236 3136 3100 2030 0000 0000 0000  .026161. 0......

Testing methodology

Modified from the repro instructions in #31873 to be faster, most importantly reusing the result of the lengthy extraction step, and just downloading the relevant branches once.

Setup phase:

# assumes you have acquired `Xcode_15.xip` based on the instructions in
# contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
XCODE=~/xcode # feel free to change this

cd $XCODE
sha256sum Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch

git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.

To verify that the source material for gen-sdk is good, we can generate the
sdk using a known good setup, ubuntu 24.04:

Verifying the setup
docker pull ubuntu:24.04
docker run -it \
  -v $XCODE:/xcode \
  ubuntu:24.04 \
  /bin/bash

Inside the container:

export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
sha256sum /xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
apt update
apt install python3 -y

xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

We should get the same hash here as what contrib/macdeploy/README.md in
master promises.


To test this branch's version of gen-sdk on a variety of platforms:

1. Container setup

XCODE=~/xcode # where the files from the setup above went.
# specify the container platform
PLATFORM=fedora:40 

docker pull $PLATFORM && \
  docker run -it \
    -v $XCODE:/xcode \
    $PLATFORM \
    /bin/bash

2. In the container

Compare the output of the final sha256sum to what contrib/macdeploy/README.md in this branch promises.

Debian/Ubuntu
export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
apt update > /dev/null
apt install python3 -y > /dev/null
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

Fedora/CentOS

dnf install python -y --quiet # python3 on rocky8.9
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

Arch

pacman -Sy
pacman --noconfirm -S python
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

Results Tables

Success

image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)"
fedora:39 3.12.7 1.2.13
fedora:40 3.12.9 1.3.1-zlib-ng
fedora:41 3.13.2 1.3.1-zlib-ng
ubuntu:24.04 3.12.3 1.3
ubuntu:24.10 3.12.7 1.3.1
archlinux:latest 3.13.2 1.3.1

Failed to reproduce:

image python --version python -c "import zlib; print(zlib.ZLIB_VERSION)" hash
debian:bookworm 3.11.2 1.2.13 8e085768391abfceae619a89ab151d148afe09f4867f1b4c4ce9c5693b92ec82
rockylinux:8.9 3.6.8 1.2.11 e779914636e6a3a417bf2a19dbce6f0bf8fab10b16717df769d107a5aad6aa2e
rockylinux:9.3 3.9.18 1.2.11 07b12c2a489c241bbc8c853fe78f2e92faf8ff51631311d142aeb8c7e20e7268

I downgraded to python 3.11.3 on fedora 41 with zlib 1.3, and got a bad hash, upgraded to python 3.12.0 with zlib-ng 1.3 on debian:bookworm and got a good hash.

custom zlib pyenv setup

Also see #31873 (comment)

dnf install -y gcc git make openssl-devel xz-devel git
curl -L https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz | tar xzvf -
cd zlib-1.3 && ./configure && make -j $(nproc) && make install && cd ..

curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

pyenv install 3.11.3
pyenv global 3.11.3

# verify zlib version
python -c "import zlib; print(zlib.ZLIB_VERSION)"
# 1.3

/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# get something bad

pyenv install 3.12.0
pyenv global 3.12.0
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

Don't have super strong evidence for this, but I suspect that this might be a second source of nondeterministic behavior that was already present before, as in my previous testing of master when I reported this issue, when I was trying different combinations of python and zlib, I also saw a similar pattern where when changing to random python, some of the hashes were randomly mismatched, and the rest were either of two hashes, one right and one wrong. I avoided sharing the random hashes in my table in #31873, and all of the values I share are for python 3.12.x

@maflcko
Copy link
Member

maflcko commented Mar 22, 2025

I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

A wild guess: Could this be related to python/cpython#95385?

@laanwj
Copy link
Member

laanwj commented Mar 26, 2025

git diff <(xxd rocky9.3/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz) <(xxd fedora41/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz)

This seems to suggest the divergence is in the tarring, not the gzipping. Do you still see a difference if you gunzip the .gzs and compare the tars only?

@luke-jr
Copy link
Member

luke-jr commented Apr 9, 2025

Maybe keep the filename the same, so people can optionally compress it on their own (or even use the current file) without breaking the build?

@fanquake fanquake force-pushed the macos_sdk_select branch from 20778eb to fc81a75 Compare June 2, 2025 14:42
@fanquake
Copy link
Member Author

fanquake commented Jun 2, 2025

Rebased and added a change to skip more files to the first commit, which reduces the final uncompressed size from ~230mb to ~157mb.

@Sjors
Copy link
Member

Sjors commented Jun 3, 2025

nit: can drop "RFC:" from the second commit?

I was able to reproduce the hash for the first commit 6998e93, but not for the second commit 20778eb.

Tested this again against a2bf3b03b19a0e668bf86eea7de3e3a592b4b23f and fc81a759413230daf3e05b448b357e62915bb3ba on my M4 machine running macOS 15.5 with Python 3.10.14 (via PyEnv). Again the hash matches in the first commit, but with second commit I get:

fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

If I use Python 3.11.11 instead I get the same hash as with 3.10.14.

@davidgumberg wrote:

I don't have an explanation for this yet, but I am able to reproduce the provided hash on systems with python versions >= 3.12.0, with various zlib and zlib-ng versions, but python < 3.12.0 I get varying hashes.

With Python 3.12.9 I indeed get ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6.

We could bump the version in .python-version and/or just mention a minimum Python version for this tool? The 3.10 security fix maintenance ends before our v30 maintenance window, and they're already not fixing (non-security) bugs: https://devguide.python.org/versions/

@fanquake fanquake force-pushed the macos_sdk_select branch from fc81a75 to 0a9b63b Compare June 3, 2025 10:09
@fanquake
Copy link
Member Author

fanquake commented Jun 3, 2025

nit: can drop "RFC:" from the second commit?

Dropped.

@maflcko
Copy link
Member

maflcko commented Jun 3, 2025

We could bump the version in .python-version and/or just mention a minimum Python version for this tool?

Wholesale bumping to 3.12 seems a bit early. Maybe just add an option to this tool --force-non-determinstic-old-python to enable 3.10 and 3.11 and otherwise use 3.12+ by default?

Copy link
Contributor

@stickies-v stickies-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 71b0f47, Concept ACK on disabling compression to increase determinism.

71b0f47 seems like a strict win, no need to include what we don't use. I think conceptually 0a9b63b also makes sense: the increased size of the tarball should not be a blocked for increased determinism. However, on my machines (arm64 macOS 15.6 and x86_64 Debian Bookworm ) determinism seems to be reduced without compression across Python versions (as also reported by other reviewers). I'm not sure whether increased determinism across distros is more important than determinism across (supported) Python versions?

(note: below results are from macOS, the hashes are the same for Debian but the filesizes vary a bit)

With selective picking and no compression (0a9b63b) different sha256sum hashes are produced by python 3.{10,11}, 3.{12, 13} and 3.{14}:

py10: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
py11: fbf3a5d369eab69702ae6ca4e7099c4ae6c9d09c5be47e7eef0df8bd0a004cfa (161mb)
py12: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
py13: ae224e4eccbf705ea35000732f332bacf4155d8d9e725aa794fe457cb68f35d6 (161mb)
py14: ea4684c568bbf63ca9fa4d1a9116751ba5b870728c4e9f72862f1b0d453ef05d (161mb)

With selective picking and compression (71b0f47) the results are consistent:

py10: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py11: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py12: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py13: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)
py14: 5aa41897b7f00abdaf1ece242dde3eb96a395746c09638b3a59720694712387d (21mb)

On master (e872a56) the results are consistent:

py10: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py11: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py12: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py13: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)
py14: c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d (81mb)

@achow101
Copy link
Member

One comment i was about to give is "why use a .tar.gz instead of .tar, that's the only way to be sure to remove dependency on zlib", but that's a much more spread out code change.

I don't think that's true. Our guix build documentation states that the tarball needs to be extracted into depends/SDK (or some directory that SDK_PATH points to), and depends/hosts/darwin.mk uses the path to the extracted directory, not the tarball itself. Other than in the docs, I don't think the tarball itself is ever handled by our build scripts. So it seems to me that dropping the gzip altogether, and just going with a .tar would resolve this problem with minimal issues.

@stickies-v
Copy link
Contributor

stickies-v commented Nov 18, 2025

ACK e7c22c8eb446727df00d1cb2ce554be9324c7fbd once the new .tar SDK is uploaded

Copy link
Contributor

@l0rinc l0rinc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't yet have experience with this part of the code, left a few nits.
I arrived here while investigating an upgrade to Xcode 16.3 in #29415 (comment) and noticed a couple of potential issues around the argparse usage and the -o handling.

I have tested the following changes locally:

diff --git a/contrib/macdeploy/gen-sdk.py b/contrib/macdeploy/gen-sdk.py
index 426d82e46c..3bf9154887 100755
--- a/contrib/macdeploy/gen-sdk.py
+++ b/contrib/macdeploy/gen-sdk.py
@@ -20,12 +20,12 @@ def run():
     parser = argparse.ArgumentParser(
         description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
 
-    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
-    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdkt', required=False)
+    parser.add_argument('xcode_app', metavar='XCODEAPP')
+    parser.add_argument("-o", metavar='OUTSDKTAR', dest='out_sdkt', required=False)
 
     args = parser.parse_args()
 
-    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
+    xcode_app = pathlib.Path(args.xcode_app).resolve()
     assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
 
     xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
@@ -46,10 +46,10 @@ def run():
     out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
 
     if args.out_sdkt:
-        out_sdkt_path = pathlib.Path(args.out_sdkt_path)
+        out_sdkt_path = pathlib.Path(args.out_sdkt)
     else:
         # Construct our own out_sdkt if not specified on the command line
-        out_sdkt_path = pathlib.Path("./{}.tar".format(out_name))
+        out_sdkt_path = pathlib.Path(f"./{out_name}.tar")
 
     def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
         """Add all files in dir_to_add to tarfp, but prepend alt_base_dir to the files'

This removes the unnecessary nargs=1 (so the positional is a scalar string instead of a 1-element list), fixes the -o code path to actually use args.out_sdkt, and updates the metavar to reflect that we now produce a .tar instead of .tar.gz.


Ran

python3 contrib/macdeploy/gen-sdk.py /Applications/Xcode.app -o /tmp/Xcode-sdk-test.tar

it shows:

Found Xcode (version: 26.1.1, build id: 17B100)
Found MacOSX SDK (version: 26.1, build id: 25B74)
Creating output .tar file...
Adding MacOSX SDK 26.1 files...
Done! Find the resulting tarball at:
/private/tmp/Xcode-sdk-test.tar

fanquake and others added 3 commits November 26, 2025 11:05
Only include what we really need. Skip 100s of mb of manpages,
swiftmodules, modulemaps.
Note that System/Library is only needed for the Qt build.
Starting with Python 3.11, Pythons gzip might delegate to zlib.
Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib
implementation might differ, resulting in different output.

For now, or until a better solution exists, disable compression. This
results in the SDK increasing in size to ~157mb. Which is not
unreasonable, to regain determinism (and would be significantly worse
without the previous commit).

See: https://docs.python.org/3/library/gzip.html#gzip.compress

Co-authored-by: stickies-v <[email protected]>
This puts it in scope for the Python linters.
@stickies-v
Copy link
Contributor

ACK 3e01b5d modulo the new .tar SDK being uploaded

@fanquake fanquake closed this Nov 26, 2025
@fanquake fanquake reopened this Nov 26, 2025
@fanquake
Copy link
Member Author

New tarball is now available. Open/close to kick the CI.

@davidgumberg
Copy link
Contributor

davidgumberg commented Nov 27, 2025

Tested ACK 3e01b5d

This fixes #31873.

sha256 os:ver python zlib
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 archlinux:latest 3.13.7 1.3.1
19e29377936086e7c8079c1047f6536e9bd53a46e180a3f49441ef38aaed900d rockylinux:8 3.6.8 1.2.11
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 rockylinux:9 3.9.21 1.2.11
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:38 3.11.8 1.2.13
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:39 3.12.7 1.2.13
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:40 3.12.8 1.3.1.zlib-ng
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:41 3.13.9 1.3.1.zlib-ng
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:42 3.13.9 1.3.1.zlib-ng
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 fedora:43 3.14.0 1.3.1.zlib-ng
8d0d3dce5da18e8eb27d02e1921c5feef971c0a7f94b8229dfe4da27f3a5b401 ubuntu:20.04 3.8.10 1.2.11
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:22.04 3.10.12 1.2.11
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:24.04 3.12.3 1.3
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.04 3.13.3 1.3.1
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 ubuntu:25.10 3.13.7 1.3.1
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:11 3.9.2 1.2.11
95b00dc41fa090747dc0a7907a5031a2fcb2d7f95c9584ba6bccdb99b6e3d498 debian:12 3.11.2 1.2.13

Versions of python <3.9.0 are still nondeterministic but this is fine, since we have a minimum python >= 3.10.

I bisected to this commit that fixed the non-determinism: python/cpython@674935b, prior to which python was incorrectly writing the device id of the target file to the tar header, which obviously varies between systems.

Using the same methodology of testing this branch, I tested master and it exhibited inconsistent hashes depending on which version of zlib python was built with, as discussed here and in #31873.

master hashes

sha256 os:ver python zlib
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d archlinux:latest 3.13.7 1.3.1
be232dd9c11726b56bebd4bdc85620a6eaf77dda46f1d45de1ebfdd36ecc5d57 rockylinux:8 3.6.8 1.2.11
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d rockylinux:9 3.9.21 1.2.11
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:38 3.11.8 1.2.13
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d fedora:39 3.12.7 1.2.13
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:40 3.12.8 1.3.1.zlib-ng
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:41 3.13.9 1.3.1.zlib-ng
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:42 3.13.9 1.3.1.zlib-ng
5b1a05d3e79fd14f5c8f6d3565762c89a522c7f5e7efbed4353d878410f2d765 fedora:43 3.14.0 1.3.1.zlib-ng
96152d637d375e94c061ab0d8441805788755eea07600ed654eeea602a33f763 ubuntu:20.04 3.8.10 1.2.11
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:22.04 3.10.12 1.2.11
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:24.04 3.12.3 1.3
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.04 3.13.3 1.3.1
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d ubuntu:25.10 3.13.7 1.3.1
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:11 3.9.2 1.2.11
c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d debian:12 3.11.2 1.2.13

Testing methodology

Modified from the repro instructions I posted above.

Setup

# assumes you have acquired `Xcode_15.xip` based on the instructions in
# contrib/macdeploy/README.md and placed it at ~/xcode/Xcode_15.xip
XCODE=~/xcode # feel free to change this

cd $XCODE
sha256sum Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e  /home/user/xcode/Xcode_15.xip
git clone --depth 1 https://github.com/bitcoin/bitcoin.git bitcoin-master
cd bitcoin-master && git fetch --depth 1 origin pull/32009/head:32009 && git worktree add ../bitcoin-32009 32009 && cd ../ # fetch the pull branch

git clone --depth 1 https://github.com/bitcoin-core/apple-sdk-tools.git
python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i # single threaded, slow, we want to reuse this.

To verify that the source material for gen-sdk is good, we can generate the sdk using a known good setup on master, ubuntu 24.04:

Verifying the setup
docker pull ubuntu:24.04
docker run -it \
  -v $XCODE:/xcode \
  ubuntu:24.04 \
  /bin/bash

Inside the container:

export DEBIAN_FRONTEND=noninteractive # prevents apt from halting to interact
sha256sum /xcode/Xcode_15.xip
# 4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e
apt update
apt install python3 -y

xcode/bitcoin-master/contrib/macdeploy/gen-sdk xcode/Xcode.app/  # we are reusing the extracted result from above
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz
# c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d  Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz

We should get the same hash here as what contrib/macdeploy/README.md in
master promises.


To test this branch's version of gen-sdk on a variety of platforms:

1. Container setup

XCODE=~/xcode # where the files from the setup above went.

# Where stuff is located in the container.
GEN_SDK="/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py"
EXTRACTED="/xcode/Xcode.app"
OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar"
# If you want to test master 
#GEN_SDK="/xcode/bitcoin-master/contrib/macdeploy/gen-sdk"
#OUT="Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz"

# The only difference between platforms is how python is installed.
declare -A INSTALL_PYTHON=(
    [rhel]="dnf install -y python3 >/dev/null 2>&1"
    [deb]="export DEBIAN_FRONTEND=noninteractive && \
           apt-get update >/dev/null 2>&1 && \
           apt-get install -y python3 >/dev/null 2>&1"
    [arch]="pacman -Sy --noconfirm >/dev/null && \
            pacman -S --noconfirm python >/dev/null"
)

tartest() {
    docker pull "$1" -q
    docker run --rm -i \
        -v "$XCODE:/xcode" \
        "$1" /bin/bash -c "
            ${INSTALL_PYTHON[$2]}
            $GEN_SDK $EXTRACTED
            PY_VER=\$(python3 --version | awk '{print \$2}')
            ZLIB_VER=\$(python3 -c 'import zlib; print(zlib.ZLIB_VERSION)')
            HASH=\$(sha256sum $OUT | awk '{print \$1}')
            echo \"|\$HASH|$1|\$PY_VER|\$ZLIB_VER|\"
        "
}


# e.g. Fedora 42
tartest fedora:42 rhel

# e.g. Ubuntu 25.04
tartest ubuntu:25.04 deb

# e.g. Arch
tartest archlinux:latest arch

2. Loops to check multiple versions

declare -A versions=(
    [fedora]="$(echo {38..43})"
    [rockylinux]="8 9"
    [ubuntu]="20.04 22.04 24.04 25.04 25.10"
    [debian]="11 12"
    [archlinux]="latest"
)

declare -A family=(
    [fedora]="rhel"
    [rockylinux]="rhel"
    [ubuntu]="deb"
    [debian]="deb"
    [archlinux]="arch"
)

echo "|sha256                                                          | os:ver | python | zlib |"
echo "|----------------------------------------------------------------|--------|--------|------|"
for os in "${!versions[@]}"; do
    for ver in ${versions[$os]}; do
        tartest "${os}:${ver}" ${family[$os]} | tail -n1
    done
done

3. Checking specific python versions

XCODE=~/xcode # where the files from the setup above went.
# specify the container platform
PLATFORM=fedora:40 

docker pull $PLATFORM && \
  docker run -it \
    -v $XCODE:/xcode \
    $PLATFORM \
    /bin/bash

In the container:

dnf install -y gcc git make openssl-devel patch xz-devel zlib-devel

curl https://pyenv.run | bash
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

Now you can install and test with any python release e.g.

pyenv install 3.9.0 && pyenv global 3.9.0
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar

We can use a pyenv build file to build a specific commit hash, e.g. to verify the commit that fixes the issue:

echo 'install_package "badcommit" "https://github.com/python/cpython/archive/9017e0bd5e124ae6d2ed94b9e9cacb2e86270980.tar.gz" standard' > badcommit
pyenv install badcommit && pyenv global badcommit
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar
# Nondeterministic hash!

echo 'install_package "goodcommit" "https://github.com/python/cpython/archive/674935b8caf33e47c78f1b8e197b1b77a04992d2.tar.gz" standard' > goodcommit
pyenv install goodcommit && pyenv global goodcommit
/xcode/bitcoin-32009/contrib/macdeploy/gen-sdk.py /xcode/Xcode.app/
sha256sum Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar

@fanquake
Copy link
Member Author

Guix Build (x86_64):

cc72adc0e567f33605050a98cd7cd08db2b0c7da1aa11dd5f73b9077653b5e00  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/SHA256SUMS.part
15d23cda22ba4afceddb733820893e4ef3a5307717002403cdc5e66d3561bafb  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu-debug.tar.gz
2d0e852259c802ea3c2411a38b93148dcaa2415b5d5d34d1f6362df39108e718  guix-build-3e01b5d0e7be/output/aarch64-linux-gnu/bitcoin-3e01b5d0e7be-aarch64-linux-gnu.tar.gz
b9c4572af1a6312575f01414b0bb1530250c3c52da5dc53638134c399ca1048b  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/SHA256SUMS.part
ee2d60c4b260107b36933e66d7eb8c2a1b69f323661348ac461a66056d2522d4  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf-debug.tar.gz
900d4bc2e57448087e3bdda96873d98eeceb9ac05d48ca33843eee3c2a60e2bc  guix-build-3e01b5d0e7be/output/arm-linux-gnueabihf/bitcoin-3e01b5d0e7be-arm-linux-gnueabihf.tar.gz
324e3036941d34dae95579ba7f4ca006514d2a7a99431deb8333c61d61e2a527  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/SHA256SUMS.part
fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
8088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  guix-build-3e01b5d0e7be/output/arm64-apple-darwin/bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
1b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  guix-build-3e01b5d0e7be/output/dist-archive/bitcoin-3e01b5d0e7be.tar.gz
e5758de2f2acc35d2ef953b80b1e8bf618e1e157f325b5d9bfa94f74e7be9c86  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/SHA256SUMS.part
5800c2135a6148bba763cb68f23708ef7fc62c850dac507a544381cccba49fa9  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu-debug.tar.gz
eae287e264b5e0ae88e0aee92a520e13aa00341266239a5ec257577f4c1a2d5c  guix-build-3e01b5d0e7be/output/powerpc64-linux-gnu/bitcoin-3e01b5d0e7be-powerpc64-linux-gnu.tar.gz
eafdaacbc9540f9e9706db281ec9f6db859e80dcb8e0e79efdc7e536799029d7  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/SHA256SUMS.part
fba989ba1b4890e1161e98b05d6ebf53cd19aab5f5be44a0081cd9eed1499138  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu-debug.tar.gz
c0e4f20087abb4ad0a58b06a7718913dd851ff46e092f95384618ad74cebff58  guix-build-3e01b5d0e7be/output/riscv64-linux-gnu/bitcoin-3e01b5d0e7be-riscv64-linux-gnu.tar.gz
78a684a96bc9c0d901cc77601263277cf142cb20659b99451247f671b11afafc  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/SHA256SUMS.part
04713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
3ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  guix-build-3e01b5d0e7be/output/x86_64-apple-darwin/bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip
d95954c5818d7d8036c5f8353063739e2c1fe65dc40eb1224e00a3dc3f4b56b4  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/SHA256SUMS.part
d8cf149a5d762da733f0efdcd67d09fbde8db601d1cd4d22ee57e87a2dcf6ce2  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu-debug.tar.gz
ffc1fb0b72f22ae574628e15dbfcd52bc37d5c37bdfd1665ecc320cca4830eae  guix-build-3e01b5d0e7be/output/x86_64-linux-gnu/bitcoin-3e01b5d0e7be-x86_64-linux-gnu.tar.gz
ba812f2cd3b757c0096c6070e5bf072442a74a5614b1dfc92a8af39c6afbffdf  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/SHA256SUMS.part
4dac4c0f9bc0109e2beca6ec3c14d650538d1a3271bae92bb84a4d05612665ff  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-codesigning.tar.gz
06f08c02fc90e461ce837f4b7cc4d0a32ada0bc3a1ebec3eae4507b00e4ca8af  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-debug.zip
ef1ce4596f1d98f684604c58c8e207e2cb30858d90232ea7e3978e03c7545a85  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-setup-unsigned.exe
301f9ee12bb95e109b0f278e3ffaace61c5c218effdf6dc475dbb358bf4116fc  guix-build-3e01b5d0e7be/output/x86_64-w64-mingw32/bitcoin-3e01b5d0e7be-win64-unsigned.zip

@fanquake fanquake merged commit 808f1d9 into bitcoin:master Nov 27, 2025
44 of 46 checks passed
@fanquake fanquake deleted the macos_sdk_select branch November 27, 2025 15:04
@pinheadmz
Copy link
Member

post-merge ACK 3e01b5d
instructions are clear, built mac outputs on x86 debian, my usual guix builder:

fe24816c78e64f0ac463eebd8dda589d6e64630527f67c10f97f4118380097dd  bitcoin-3e01b5d0e7be-arm64-apple-darwin-codesigning.tar.gz
ddd416028f6858282d485f0017d8d777d3cee0bcd207f0d8e2542191425602cb  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.tar.gz
8088cd6ef218acc83928bd415b88bbeb082447890e7e497ee0b28e149b0bfc59  bitcoin-3e01b5d0e7be-arm64-apple-darwin-unsigned.zip
1b254bdac0893af4c012b6f093ee7be5b6f5662ceb114841b16d625c1586d415  bitcoin-3e01b5d0e7be.tar.gz
04713a87ee3c418a35103f843c2d725f237a44aebc2fe28b6fbc3f5b73424f18  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-codesigning.tar.gz
3ce4ea97fe2da7f41f0886f933ac0712528e90279386783af64b2ef0394fa2f3  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.tar.gz
ddfdbfa347386326fd274069009ad9ed6c5e8837d7595027d6e095193e251176  bitcoin-3e01b5d0e7be-x86_64-apple-darwin-unsigned.zip

alexanderwiederin added a commit to alexanderwiederin/rust-bitcoinkernel that referenced this pull request Dec 1, 2025
…356041e58d1

6356041e58d1 Merge bitcoin/bitcoin#33972: cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB`
7d7cb1bb48f1 Merge bitcoin/bitcoin#33971: cmake: Set `WITH_ZMQ` to `ON` in Windows presets
fe1815d48f0c cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB`
49c672853503 cmake: Set `WITH_ZMQ` to `ON` in Windows presets
f6acbef1084e Merge bitcoin/bitcoin#33764: ci: Add Windows + UCRT jobs for cross-compiling and native testing
808f1d972be3 Merge bitcoin/bitcoin#32009: contrib: turn off compression of macOS SDK to fix determinism (across distros)
4de26b111f4d Merge bitcoin/bitcoin#33514: ci: clear out space on CentOS, depends, gui GHA job
38c8474d0d77 Merge bitcoin/bitcoin#33914: Change Parse descriptor argument to string_view
4b25b274de66 Merge bitcoin/bitcoin#33951: test: check for output to stdout in `TestShell` test
52230a7f697f test: check for output to stdout in `TestShell` test
85d058dc537e Merge bitcoin/bitcoin#33946: interfaces: remove redundant mempool lock in ChainImpl::isInMempool()
e07e57368e9f ci: clear out space on centos job
79d6e874e1da Merge bitcoin/bitcoin#32587: test: Fix reorg patterns in tests to use proper fork-based approach
e249ea7da6c2 Merge bitcoin/bitcoin#33945: depends: latest config.guess & config.sub
3e01b5d0e7be contrib: rename gen-sdk to gen-sdk.py
c1213a35abed macdeploy: disable compression in macOS gen-sdk script
a33d03454508 contrib: more selectively pick files for macOS SDK
70d9e8f0a15d fix: reorg behaviour in mempool tests to match real one
540ed333f6c8 Move the create_empty_fork method to the test framework's blocktools.py module to enable reuse across multiple tests.
2909655fba91 fix: remove redundant mempool lock in ChainImpl::isInMempool()
d5ed4ba9d862 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin
3e4355314b1a depends: latest config.sub
04eb84fe3f73 depends: latest config.guess
b30262dcaa28 Merge bitcoin/bitcoin#33903: ci: Remove redundant busybox option
1a5f1eb08067 Merge bitcoin/bitcoin#33921: doc: clarify and cleanup macOS fuzzing notes
72cb8cef9778 Merge bitcoin/bitcoin#33862: txgraph: drop move assignment operator
ade0397f59f2 txgraph: drop move assignment operator
5336bcd57849 Merge bitcoin/bitcoin#33855: kernel: add btck_block_tree_entry_equals
4f65a1c5db84 Merge bitcoin/bitcoin#33917: clang-format: Set Bitcoin Core IncludeCategories
902717b66dc1 Merge bitcoin/bitcoin#33918: depends: Update Qt download link
68ab2b65bfac Merge bitcoin/bitcoin#33919: ci: Run GUI unit tests in cross-Windows task
7e129b644ec9 Merge bitcoin/bitcoin#33893: test: add `-alertnotify` test for large work invalid chain warning
5fe753b56f45 Merge bitcoin/bitcoin#32655: depends: sqlite 3.50.4; switch to autosetup
ff8c2f37497f Merge bitcoin/bitcoin#33932: ci: Use latest Xcode that the minimum macOS version allows
fa283d28e261 Merge bitcoin/bitcoin#33629: Cluster mempool
2e27bd9c3af9 ci: Add Windows + UCRT jobs for cross-compiling and native testing
238c1c8933b1 Merge bitcoin-core/gui#914: Revert "gui, qt: brintToFront workaround for Wayland"
8343a9ffcc75 test: add `-alertnotify` test for large work invalid chain warning
c34bc01b2ff2 doc: clarify and cleanup macOS fuzzing notes
fa9537cde101 ci: Use latest Xcode that the minimum macOS version allows
17cf9ff7efdb Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general
315e43e5d86c Sanity check `GetFeerateDiagram()` in CTxMemPool::check()
de2e9a24c40e test: extend package rbf functional test to larger clusters
4ef4ddb504e5 doc: update policy/packages.md for new package acceptance logic
79f73ad713a8 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology()
a86ac1176817 Update comments for CTxMemPool class
9567eaa66da8 Invoke TxGraph::DoWork() at appropriate times
0690514d4f72 Merge bitcoin/bitcoin#33770: init: Require explicit -asmap filename
b2f88b53e0ec Merge bitcoin/bitcoin#33286: doc: update multisig tutorial to use multipath descriptors
313cdd2bfb71 Merge bitcoin/bitcoin#33915: test: Retry download in get_previous_releases.py
bd130db994e2 ci: Rename items specific to Windows + MSVCRT
0672e727bf1d Revert "gui, qt: brintToFront workaround for Wayland"
17072f70051d Merge bitcoin/bitcoin#33912: clang-format: Set PackConstructorInitializers: CurrentLine
fa7ea497c3ef ci: Run GUI unit tests in cross-Windows task
fa0fee44a89c ci: Remove redundant busybox option
fa102ec69fae doc: Shorten ci name
fa7e222a2326 clang-format: Set Bitcoin Core IncludeCategories
222222378048 doc: Remove bash -c wrapper
6b2d17b13220 Merge bitcoin/bitcoin#33888: ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG
ac71df43383a Merge bitcoin/bitcoin#33870: refactor: remove incorrect lifetimebounds
6cdb51c14eba Merge bitcoin/bitcoin#33887: doc: Improve CI docs on env and qemu-user-static
50cbde3295b4 depends: Update Qt download link
29c37651c74b Merge bitcoin/bitcoin#33880: test: Fix race condition in IPC interface block progation test
32368cd3e9f3 Merge bitcoin/bitcoin#33905: ci: Consistenly only cache on the default branch
e55c49f85143 Merge bitcoin/bitcoin#33851: depends: update xcb-util packages to latest versions
a07bd8415df4 Merge bitcoin/bitcoin#33824: ci: Enable experimental kernel stuff in most CI tasks via `dev-mode`
c0bfe72f6e1f Change Parse descriptor argument to string_view
f541b92cf2bb depends: expat 2.7.3
fad06f3bb436 test: retry download in get_previous_releases.py
2ebf4356e63d depends: libxcb 1.17.0
ba7ac870a32a depends: xcb_proto 1.17.0
fad0c76d0a10 clang-format: Set PackConstructorInitializers: CurrentLine
42d0692f9131 depends: libxcb-util-cursor 0.1.6
25b85919ab62 depends: libxcb 1.15
d129384ca97f depends: libxcb-util-wm 0.4.2
0b857ae9e555 depends: libxcb-util-renderutil 0.3.10
35e50488b25a depends: libxcb-util-keysyms 0.4.1
74b68ad28ba2 depends: libxcb-util-image 0.4.1
5bc0dde85d74 depends: libxcb-util 0.4.1
8d07292c286f depends: libXau 1.0.12
1af46cff9478 Merge bitcoin/bitcoin#33896: clang-format: Set InsertNewlineAtEOF: true
27ac11ea0a27 Merge bitcoin/bitcoin#33867: kernel: handle null or empty directories in implementation
2578e6fc0f4a test: Fix race condition in IPC interface block propagation test
288b8c30be42 doc: Drop (default: none) from -i2psam description
509dc91db143 Merge bitcoin/bitcoin#33026: test, refactor: Embedded ASMap [1/3]: Selected minor preparatory work
b126f981943d Merge bitcoin-core/gui#910: Added test coverage for qt gui#901 console history filter
7d7b829c36b7 Merge bitcoin-core/gui#908: Remove HD seed reference from blank wallet tooltip
8558902e576e depends: Add patch for Windows11Style plugin
53b72372da91 Merge bitcoin/bitcoin#31734: miniscript: account for all `StringType` variants in `Miniscriptdescriptor::ToString()`
a7f9bbe4c5e7 Merge bitcoin/bitcoin#32821: rpc: Handle -named argument parsing where '=' character is used
55555db055b5 doc: Add missing --platform=linux to docker build command
fa0ce4c1486b ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG
faa0973de296 ci: [refactor] Rename CIRRUS_PR env var to LINT_CI_IS_PR
fa411f938e47 ci: Consistenly only cache on the default branch
6c5c44f77405 test: add functional test for new cluster mempool RPCs
72f60c877e00 doc: Update mempool_replacements.md to reflect feerate diagram checks
21693f031a53 Expose cluster information via rpc
72e74e0d4228 fuzz: try to add more code coverage for mempool fuzzing
f107417490ab bench: add more mempool benchmarks
7976eb1ae77a Avoid violating mempool policy limits in tests
84de685cf7ee Stop tracking parents/children outside of txgraph
88672e205ba1 Rewrite GatherClusters to use the txgraph implementation
1ca4f01090cf Fix miniminer_tests to work with cluster limits
1902111e0f20 Eliminate CheckPackageLimits, which no longer does anything
3a646ec46264 Rework RBF and TRUC validation
19b8479868e5 Make getting parents/children a function of the mempool, not a mempool entry
5560913e51af Rework truc_policy to use descendants, not children
a4458d6c4062 Use txgraph to calculate descendants
c8b6f70d6492 Use txgraph to calculate ancestors
241a3e666b59 Simplify ancestor calculation functions
b9cec7f0a1e0 Make removeConflicts private
0402e6c78080 Remove unused limits from CalculateMemPoolAncestors
08be765ac26a Remove mempool logic designed to maintain ancestor/descendant state
fc4e3e6bc122 Remove unused members from CTxMemPoolEntry
ff3b398d124b mempool: eliminate accessors to mempool entry ancestor/descendant cached state
b9a2039f5122 Eliminate use of cached ancestor data in miniminer_tests and truc_policy
ba09fc9774d5 mempool: Remove unused function CalculateDescendantMaximum
8e49477e86b3 wallet: Replace max descendant count with cluster_count
e031085fd464 Eliminate Single-Conflict RBF Carve Out
cf3ab8e1d0a2 Stop enforcing descendant size/count limits
89ae38f48965 test: remove rbf carveout test from mempool_limit.py
c0bd04d18fdf Calculate descendant information for mempool RPC output on-the-fly
bdcefb8a8b06 Use mempool/txgraph to determine if a tx has descendants
69e1eaa6ed22 Add test case for cluster size limits to TRUC logic
9cda64b86c59 Stop enforcing ancestor size/count limits
1f93227a84a5 Remove dependency on cached ancestor data in mini-miner
9fbe0a4ac26c rpc: Calculate ancestor data from scratch for mempool rpc calls
7961496dda2e Reimplement GetTransactionAncestry() to not rely on cached data
feceaa42e8eb Remove CTxMemPool::GetSortedDepthAndScore
21b5cea588a7 Use cluster linearization for transaction relay sort order
6445aa7d9755 Remove the ancestor and descendant indices from the mempool
216e69372903 Implement new RBF logic for cluster mempool
ff8f115dec6e policy: Remove CPFP carveout rule
c3f1afc934e6 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits
47ab32fdb158 Select transactions for blocks based on chunk feerate
dec138d1ddc7 fuzz: remove comparison between mini_miner block construction and miner
6c2bceb200aa bench: rewrite ComplexMemPool to not create oversized clusters
1ad4590f6385 Limit mempool size based on chunk feerate
b11c89cab210 Rework miner_tests to not require large cluster limit
95a8297d481e Check cluster limits when using -walletrejectlongchains
95762e675959 Do not allow mempool clusters to exceed configured limits
edb3e7cdf636 [test] rework/delete feature_rbf tests requiring large clusters
435fd5671116 test: update feature_rbf.py replacement test
34e32985e811 Add new (unused) limits for cluster size/count
838d7e355366 Add transactions to txgraph, but without cluster dependencies
552eb90071fd doc: CI - Describe qemu-user-static usage
2afbbddee550 doc: CI - Clarify how important `env -i` is and why
2444488f6ad3 Merge bitcoin/bitcoin#33894: net: Remove unused `local_socket_bytes` variable in `CConnman::GetAddresses()`
fa1bf6818f09 clang-format: Set InsertNewlineAtEOF: true
115d298a9fa3 Merge bitcoin/bitcoin#33872: init: completely remove `-maxorphantx` option
a90f3922ff7d Merge bitcoin/bitcoin#32419: psbt: clarify PSBT, PSBTInput, PSBTOutput unserialization flows
4d893c0f4605 net: Remove unused `local_socket_bytes` variable in `CConnman::GetAddresses()`
fa1dacaebe5d ci: Move lint exec snippet to stand-alone py file
ead849c9f177 Merge bitcoin/bitcoin#33886: test: Remove tests violating hardened std::span
c03081fdb467 Merge bitcoin/bitcoin#33776: ci: Lint follow-ups
fadb4f63cb0f test: Remove tests violating hardened std::span
6e2155816058 Merge bitcoin/bitcoin#33869: refactor: Avoid -W*-whitespace in git archive
c8715aca95d0 Merge bitcoin/bitcoin#33247: build: Remove CMAKE_SKIP_BUILD_RPATH and SKIP_BUILD_RPATH settings
ee5de407e369 Merge bitcoin/bitcoin#33537: guix: build `bitcoin-qt` with static libxcb & utils
024a7873500e Merge bitcoin/bitcoin#33876: doc: Update NetBSD Build Guide
c66e98875439 Merge bitcoin/bitcoin#33865: cmake: Specify Windows plugin path in `test_bitcoin-qt` property
c29eaeeaf937 doc: Update NetBSD Build Guide
7f318e1dd049 test: Add better coverage for Autofile size()
e221b2524659 Merge bitcoin/bitcoin#33860: depends: drop Qt patches
b7af960eb82f refactor: Add AutoFile::size
ec0f75862e67 refactor: Modernize logging in util/asmap.cpp
606a251e0a31 tests: add unit test vectors for asmap interpreter
6657bcbdb4d0 kernel: allow null data_directory
0aebdac95da9 init: completely remove `-maxorphantx` option
99d012ec80a4 refactor: return reference instead of pointer
f743e6c5dd38 refactor: add missing LIFETIMEBOUND annotation for parameter
fa95353902b7 ci: Run macos tasks in a git archive, not git checkout
141117f5e8b4 refactor: remove incorrect LIFETIMEBOUND annotations
fae3618fd6c8 ci: Annotate all check runs with the pull request number
faf05d637d67 ci: Retry lint image building once after failure
96963b888e5a depends: static libxcb
ad06843fab06 depends: avoid qdbusviewer in Qt build
6848ed56dc5f depends: apply Qt patches to fix static libxcb use
dfde31f2ec1f Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to `policy/fees.h`
5f1b016bebd2 depends: static libxcb-util-image
98a2fbbe70b8 depends: static libxkbcommon
1412baf77295 depends: static libxcb-util-wm
a4009dadf466 depends: static libxcb-keysyms
bcfb8679b3ba depends: static libxcb-render-util
faf99ae37963 refactor: Avoid -W*-whitespace in git archive
2594d5a189e5 build: Remove CMAKE_SKIP_BUILD_RPATH and SKIP_BUILD_RPATH settings
310e4979b36c qt: Added test coverage for qt gui#901 console history filter
0dd8d5c237e2 cmake: Specify Windows plugin path in `test_bitcoin-qt` property
b0a38871546d scripted-diff: fix leftover references to `policy/fees.h`
48d4b936e09f Merge bitcoin/bitcoin#33511: init: Fix Ctrl-C shutdown hangs during wait calls
3c3c6adb7260 Merge bitcoin/bitcoin#33745: mining: check witness commitment in submitBlock
e652b69b8da4 Merge bitcoin/bitcoin#33003: test: add option to skip large re-org test in feature_block
3e9aca6f1b52 depends: drop qtbase-moc-ignore-gcc-macro.patch qt patch
fac4f6de28e7 ci: Rewrite lint task Bash snippet to Python
fa0d37a57985 ci: Rewrite Bash to check inputs to Python
0da5a82700e9 depends: drop unused qt patch
fae83611b8ef ci: [refactor] Use --preset=dev-mode in mac_native task
fadb67b4b4e1 ci: [refactor] Base nowallet task on --preset=dev-mode
6666980e8653 ci: Enable bitcoin-chainstate and test_bitcoin-qt in win64 task
096924d39d64 kernel: add btck_block_tree_entry_equals
faff7b231246 ci: Enable experimental kernel stuff in i686 task
fa1632eecf58 ci: Enable experimental kernel stuff in mac-cross tasks
fad10ff7c923 ci: Enable experimental kernel stuff in armhf task
fa9d67c13d0d ci: Enable experimental kernel stuff in Alpine task
fab3fb83026e ci: Enable experimental kernel stuff in s390x task
fa7da8a646ed ci: Enable experimental kernel stuff in valgrind task
fa9c2973d60b ci: Enable experimental kernel stuff in TSan task
fad30d439502 ci: Enable experimental kernel stuff in MSan task
d5ed9cb3eb52 Add accessor for sigops-adjusted weight
1bf3b513966e Add sigops adjusted weight calculator
c18c68a950d3 Create a txgraph inside CTxMemPool
29a94d5b2f26 Make CTxMemPoolEntry derive from TxGraph::Ref
92b0079fe386 Allow moving CTxMemPoolEntry objects, disallow copying
f6ec3519a330 init: Require explicit -asmap filename
6eaa00fe2020 test: clarify submitBlock() mutates the template
862bd432837e mining: ensure witness commitment check in submitBlock
00d1b6ef4b12 doc: clarify UpdateUncommittedBlockStructures
929f69d0ff29 qt: Remove HD seed reference from blank wallet tooltip
1db74914706f depends: sqlite 3.50.4
286f3e49c84c guix: sqlite wants tcl
de7c3587cd45 doc: Update add checksum instructions in tutorial
6c73e4744837 mempool: Store iterators into mapTx in mapNextTx
51430680ecb7 Allow moving an Epoch::Marker
2a46e94a1600 doc: Update multisig-tutorial.md to use multipath descriptors
c25a5e670b27 init: Signal m_tip_block_cv on Ctrl-C
f53dbbc5057b test: Add functional tests for named argument parsing
694f04e2bd34 rpc: Handle -named argument parsing where '=' character is used
6a29f79006a9 test: Test SIGTERM handling during waitforblockheight call
8810642b571e test: add option to skip large re-org test in feature_block
d31158d3646f psbt: clarify PSBT, PSBTInput, PSBTOutput unserialization flows
28a4fcb03c0f test: check listdescriptors do not return a mix of hardened derivation marker
975783cb79e9 descriptor: account for all StringType in MiniscriptDescriptor::ToStringHelper()

git-subtree-dir: libbitcoinkernel-sys/bitcoin
git-subtree-split: 6356041e58d1ba86695e2e7c219c68ee5abe583f
fanquake added a commit to fanquake/bitcoin that referenced this pull request Dec 3, 2025
Only include what we really need. Skip 100s of mb of manpages,
swiftmodules, modulemaps.
Note that System/Library is only needed for the Qt build.

Github-Pull: bitcoin#32009
Rebased-From: a33d034
fanquake added a commit to fanquake/bitcoin that referenced this pull request Dec 3, 2025
Starting with Python 3.11, Pythons gzip might delegate to zlib.
Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib
implementation might differ, resulting in different output.

For now, or until a better solution exists, disable compression. This
results in the SDK increasing in size to ~157mb. Which is not
unreasonable, to regain determinism (and would be significantly worse
without the previous commit).

See: https://docs.python.org/3/library/gzip.html#gzip.compress

Co-authored-by: stickies-v <[email protected]>

Github-Pull: bitcoin#32009
Rebased-From: c1213a3
fanquake added a commit to fanquake/bitcoin that referenced this pull request Dec 3, 2025
This puts it in scope for the Python linters.

Github-Pull: bitcoin#32009
Rebased-From: 3e01b5d
@fanquake
Copy link
Member Author

fanquake commented Dec 3, 2025

Backported to 30.x in #33997.

fanquake added a commit that referenced this pull request Dec 5, 2025
d59ebac doc: update manual pages for v30.1rc1 (fanquake)
b83d4f7 build: bump version to v30.1rc1 (fanquake)
b26c93a doc: update release notes for v30.1rc1 (fanquake)
18f3ada Remove unreliable seed from chainparams.cpp, and the associated README (SatsAndSports)
187e3b8 [test] wallet send 3 generation TRUC (glozow)
ab58b2c [wallet] never try to spend from unconfirmed TRUC that already has ancestors (glozow)
d872277 contrib: fix manpage generation (fanquake)
2256f89 contrib: rename gen-sdk to gen-sdk.py (fanquake)
5408e85 macdeploy: disable compression in macOS gen-sdk script (fanquake)
338570d contrib: more selectively pick files for macOS SDK (fanquake)

Pull request description:

  Backports:
  * #32009
  * #33528
  * #33723
  * #33996

  Prior `30.x` backports in #33609.

ACKs for top commit:
  hebasto:
    re-ACK d59ebac.
  marcofleon:
    ACK d59ebac

Tree-SHA512: 41656dc7c2e7a023325b288ff38cf95877a3c223ed5c6defc1c0939b725fb5423455583881f283aff14821bc4fea5a4d4d0ac458cab6450eb9b1bc7730fc027c
stringintech added a commit to stringintech/go-bitcoinkernel that referenced this pull request Dec 12, 2025
938d7aacab Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage
597b8be223 Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling
d155fc12a0 Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate
07135290c1 rest: allow reading partial block data from storage
4e2af1c065 blockstorage: allow reading partial block data from storage
f2fd1aa21c blockstorage: return an error code from `ReadRawBlock()`
5be20c380d Merge bitcoin/bitcoin#34033: scripted-diff: Unify error and warning log formatting
b31f786695 Merge bitcoin/bitcoin#34045: test: Log IP of download server in get_previous_releases.py
b26762bdcb Merge bitcoin/bitcoin#33805: merkle: migrate `path` arg to reference and drop unused args
0f6d8a347a Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs
c2975f26d6 Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation
cdaf25f9c3 test: Log IP of download server in get_previous_releases.py
c1f0a89d9c Merge bitcoin/bitcoin#34040: test: Detect truncated download in get_previous_releases.py
fa75480c84 test: Detect truncated download in get_previous_releases.py
56ce78d5f6 Merge bitcoin/bitcoin#34031: net: Remove "tor" as a network specification
500862b2d4 Merge bitcoin/bitcoin#33423: qa: Improvements to debug_assert_log + busy_wait_for_debug_log
5f5c1ea019 net: Cache -capturemessages setting
cca113f5b0 Merge bitcoin/bitcoin#34008: log: don't rate-limit "new peer" with -debug=net
2c44c41984 Merge bitcoin/bitcoin#33553: validation: Improve warnings in case of chain corruption
6eb5ba5691 refactor: extract shared `SipHash` state into `SipHashState`
118d22ddb4 optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs`
9ca52a4cbe optimization: migrate `SipHashUint256` to `PresaltedSipHasher`
ec11b9fede optimization: introduce `PresaltedSipHasher` for repeated hashing
d23d49ee3f Merge bitcoin/bitcoin#31823: tests: Add witness commitment if we have a witness transaction in `FullBlockTest.update_block()`
20330548cf refactor: extract `SipHash` C0-C3 constants to class scope
9f9eb7fbc0 test: rename k1/k2 to k0/k1 in `SipHash` consistency tests
29ed608dc7 Merge bitcoin/bitcoin#33961: script: Add a separate ScriptError for empty pubkeys encountered in Tapscript
d2a199bca7 Merge bitcoin/bitcoin#33909: doc, ci: Make the max number of commits tested explicit
dbc8928069 Merge bitcoin/bitcoin#33993: init: point out -stopatheight may be imprecise
d4d184eda9 log: don't rate-limit "new peer" with -debug=net
e7ac5a133c doc: add release note for 34031
c4c70a256e netbase: Remove "tor" as a network specification
fa89f60e31 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning()
fa6c7a1954 scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug()
d5c8199b79 Merge bitcoin/bitcoin#34006: Add util::Expected (std::expected)
77248e8496 Merge bitcoin/bitcoin#33771: refactor: C++20 operators
36073d56db Merge bitcoin/bitcoin#33952: depends: update freetype and document remaining `bitcoin-qt` runtime libs
f09ae5f96f Merge bitcoin/bitcoin#33950: guix: reduce allowed exported symbols
cea443e246 net: Pass time to InactivityChecks fuctions
89dc82295e Merge bitcoin/bitcoin#29641: scripted-diff: Use LogInfo over LogPrintf
eb19a2dac5 Merge bitcoin/bitcoin#34017: fuzz: Add a test case for `ParseByteUnits()`
faa23738fc refactor: Enable clang-tidy bugprone-unused-return-value
fa114be27b Add util::Expected (std::expected)
e68517208b Merge bitcoin/bitcoin#33995: depends: Propagate native C compiler to `sqlite` package
091cae6fdf Merge bitcoin/bitcoin#33939: contrib: Count entry differences in asmap-tool diff summary
57b888ce0e fuzz: Add a test case for `ParseByteUnits()`
b8e66b901d Merge bitcoin/bitcoin#33858: test: add unit test coverage for the empty leaves path in MerkleComputation
0c9ab0f8f8 Merge bitcoin/bitcoin#33956: net: fix use-after-free with v2->v1 reconnection logic
fa4395dffd refactor: Remove unused LogPrintf
fa05181d90 scripted-diff: LogPrintf -> LogInfo
9890058b37 Merge bitcoin/bitcoin#33723: chainparams: remove dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us
9e02f78089 Merge bitcoin/bitcoin#33774: cmake: Move IPC tests to `ipc/test`
ad452a1e65 Merge bitcoin/bitcoin#33528: wallet: don't consider unconfirmed TRUC coins with ancestors
ff06e2468a init: point out -stopatheight may be imprecise
9a29b2d331 Merge bitcoin/bitcoin#33857: doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md`
69e66efe45 Merge bitcoin/bitcoin#32882: index: remove unnecessary locator cleaning in BaseIndex::Init()
6581ac5d9f Merge bitcoin/bitcoin#33996: contrib: fix manpage generation
39ca015259 Merge bitcoin/bitcoin#33140: test: Avoid shutdown race in NetworkThread
e9536faaee contrib: fix manpage generation
bcf794d5f3 Merge bitcoin/bitcoin#30455: test: assumeutxo: add missing tests in wallet_assumeutxo.py
af0e6a65c9 Merge bitcoin/bitcoin#33702: contrib: Remove brittle, confusing and redundant UTF8 encoding from Python IO
4b47113698 validation: Reword CheckForkWarningConditions and call it also during IBD and at startup
2f51951d03 p2p: Add warning message when receiving headers for blocks cached as invalid
4c784b25c4 Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight
710031ebef Revert "guix: sqlite wants tcl"
4cf5ea6c3d depends: Propagate native C compiler to `sqlite` package
ce771726f3 Merge bitcoin/bitcoin#33960: log: Use more severe log level (warn/err) where appropriate
cb7d5bfe4a test, assumeutxo: loading a wallet (backup) on a pruned node
7a365244f8 test, refactor snapshot import and background validation
e0ba6bbed9 Merge bitcoin/bitcoin#33591: Cluster mempool followups
b8d279a81c doc: add comment to explain correctness of GatherClusters()
aba7500a30 Fix parameter name in getmempoolcluster rpc
6c1325a091 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology
bc2eb931da Require mempool lock to be held when invoking TRUC checks
957ae23241 Improve comments for getTransactionAncestry to reference cluster counts instead of descendants
d97d6199ce Fix comment to reference cluster limits, not chain limits
a1b341ef98 Sanity check feerate diagram in CTxMemPool::check()
23d6f457c4 rpc: improve getmempoolcluster output
d2dcd37aac Avoid using mapTx.modify() to update modified fees
d84ffc24d2 doc: add release notes snippet for cluster mempool
b0417ba944 doc: Add design notes for cluster mempool and explain new mempool limits
804329400a fuzz: gate mempool entry based on weight
6356041e58 Merge bitcoin/bitcoin#33972: cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB`
7d7cb1bb48 Merge bitcoin/bitcoin#33971: cmake: Set `WITH_ZMQ` to `ON` in Windows presets
2d88966e43 miner: replace "package" with "chunk"
6f3e8eb300 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler
b5f245f6f2 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB
1dac54d506 Use cluster size limit instead of ancestor size limit in txpackage unit test
04f65488ca Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits
634291a7dc Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits
fc18ef1f3f Remove ancestor and descendant vsize limits from MemPoolLimits
ed8e819121 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect
80d8df2d47 Invoke removeUnchecked() directly in removeForBlock()
9292570f4c Rewrite GetChildren without sets
3e39ea8c30 Rewrite removeForReorg to avoid using sets
a3c31dfd71 scripted-diff: rename AddToMempool -> TryAddToMempool
a5a7905d83 Simplify removeRecursive
01d8520038 Remove unused argument to RemoveStaged
fe1815d48f cmake: Make `BUILD_KERNEL_TEST` depend on `BUILD_KERNEL_LIB`
49c6728535 cmake: Set `WITH_ZMQ` to `ON` in Windows presets
ec8eb013a9 doc: Add `x86_64-w64-mingw32ucrt` triplet to `depends/README.md`
48496caa12 ci: Remove redundant `DEP_OPTS` from “Windows-cross UCRT” job
f6acbef108 Merge bitcoin/bitcoin#33764: ci: Add Windows + UCRT jobs for cross-compiling and native testing
b5a7a685bb ci: Make the max number of commits tested explicit
9d5021a05b script: add SCRIPT_ERR_TAPSCRIPT_EMPTY_PUBKEY
7b90b4f5bb guix: reduce allowed exported symbols
41e657aacf guix: add bitcoin-qt runtime libs doc in symbol-check
ef4ce19a15 depends: freetype 2.11.1
808f1d972b Merge bitcoin/bitcoin#32009: contrib: turn off compression of macOS SDK to fix determinism (across distros)
4de26b111f Merge bitcoin/bitcoin#33514: ci: clear out space on CentOS, depends, gui GHA job
fa45a1503e log: Use LogWarning for non-critical logs
fa0018d011 log: Use LogError for fatal errors
22229de728 doc: Fix typo in init log
38c8474d0d Merge bitcoin/bitcoin#33914: Change Parse descriptor argument to string_view
4b25b274de Merge bitcoin/bitcoin#33951: test: check for output to stdout in `TestShell` test
167df7a98c net: fix use-after-free with v2->v1 reconnection logic
52230a7f69 test: check for output to stdout in `TestShell` test
85d058dc53 Merge bitcoin/bitcoin#33946: interfaces: remove redundant mempool lock in ChainImpl::isInMempool()
fd4ce55121 contrib: Count entry differences in asmap-tool diff summary
e07e57368e ci: clear out space on centos job
79d6e874e1 Merge bitcoin/bitcoin#32587: test: Fix reorg patterns in tests to use proper fork-based approach
e249ea7da6 Merge bitcoin/bitcoin#33945: depends: latest config.guess & config.sub
3e01b5d0e7 contrib: rename gen-sdk to gen-sdk.py
c1213a35ab macdeploy: disable compression in macOS gen-sdk script
a33d034545 contrib: more selectively pick files for macOS SDK
fad6118586 test: Fix "typo" in written invalid content
fab085c15f contrib: Use text=True in subprocess over manual encoding handling
fa71c15f86 scripted-diff: Bump copyright headers after encoding changes
fae612424b contrib: Remove confusing and redundant encoding from IO
fa7d72bd1b lint: Drop check to enforce encoding to be specified in Python scripts
faf39d8539 test: Clarify that Python UTF-8 mode is the default today for most systems
fa83e3a81d lint: Do not allow locale dependent shell scripts
70d9e8f0a1 fix: reorg behaviour in mempool tests to match real one
540ed333f6 Move the create_empty_fork method to the test framework's blocktools.py module to enable reuse across multiple tests.
2909655fba fix: remove redundant mempool lock in ChainImpl::isInMempool()
d5ed4ba9d8 Merge bitcoin/bitcoin#33906: depends: Add patch for Windows11Style plugin
3e4355314b depends: latest config.sub
04eb84fe3f depends: latest config.guess
b30262dcaa Merge bitcoin/bitcoin#33903: ci: Remove redundant busybox option
1a5f1eb080 Merge bitcoin/bitcoin#33921: doc: clarify and cleanup macOS fuzzing notes
72cb8cef97 Merge bitcoin/bitcoin#33862: txgraph: drop move assignment operator
bc64013e6f Remove unused variable (cacheMap) in mempool
ade0397f59 txgraph: drop move assignment operator
5336bcd578 Merge bitcoin/bitcoin#33855: kernel: add btck_block_tree_entry_equals
4f65a1c5db Merge bitcoin/bitcoin#33917: clang-format: Set Bitcoin Core IncludeCategories
902717b66d Merge bitcoin/bitcoin#33918: depends: Update Qt download link
68ab2b65bf Merge bitcoin/bitcoin#33919: ci: Run GUI unit tests in cross-Windows task
7e129b644e Merge bitcoin/bitcoin#33893: test: add `-alertnotify` test for large work invalid chain warning
5fe753b56f Merge bitcoin/bitcoin#32655: depends: sqlite 3.50.4; switch to autosetup
ff8c2f3749 Merge bitcoin/bitcoin#33932: ci: Use latest Xcode that the minimum macOS version allows
fa283d28e2 Merge bitcoin/bitcoin#33629: Cluster mempool
2e27bd9c3a ci: Add Windows + UCRT jobs for cross-compiling and native testing
238c1c8933 Merge bitcoin-core/gui#914: Revert "gui, qt: brintToFront workaround for Wayland"
8343a9ffcc test: add `-alertnotify` test for large work invalid chain warning
c34bc01b2f doc: clarify and cleanup macOS fuzzing notes
fa9537cde1 ci: Use latest Xcode that the minimum macOS version allows
17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general
315e43e5d8 Sanity check `GetFeerateDiagram()` in CTxMemPool::check()
de2e9a24c4 test: extend package rbf functional test to larger clusters
4ef4ddb504 doc: update policy/packages.md for new package acceptance logic
79f73ad713 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology()
a86ac11768 Update comments for CTxMemPool class
9567eaa66d Invoke TxGraph::DoWork() at appropriate times
bd130db994 ci: Rename items specific to Windows + MSVCRT
0672e727bf Revert "gui, qt: brintToFront workaround for Wayland"
fa7ea497c3 ci: Run GUI unit tests in cross-Windows task
fa0fee44a8 ci: Remove redundant busybox option
fa102ec69f doc: Shorten ci name
fa7e222a23 clang-format: Set Bitcoin Core IncludeCategories
2222223780 doc: Remove bash -c wrapper
50cbde3295 depends: Update Qt download link
c0bfe72f6e Change Parse descriptor argument to string_view
8558902e57 depends: Add patch for Windows11Style plugin
6c5c44f774 test: add functional test for new cluster mempool RPCs
72f60c877e doc: Update mempool_replacements.md to reflect feerate diagram checks
21693f031a Expose cluster information via rpc
72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing
f107417490 bench: add more mempool benchmarks
7976eb1ae7 Avoid violating mempool policy limits in tests
84de685cf7 Stop tracking parents/children outside of txgraph
88672e205b Rewrite GatherClusters to use the txgraph implementation
1ca4f01090 Fix miniminer_tests to work with cluster limits
1902111e0f Eliminate CheckPackageLimits, which no longer does anything
3a646ec462 Rework RBF and TRUC validation
19b8479868 Make getting parents/children a function of the mempool, not a mempool entry
5560913e51 Rework truc_policy to use descendants, not children
a4458d6c40 Use txgraph to calculate descendants
c8b6f70d64 Use txgraph to calculate ancestors
241a3e666b Simplify ancestor calculation functions
b9cec7f0a1 Make removeConflicts private
0402e6c780 Remove unused limits from CalculateMemPoolAncestors
08be765ac2 Remove mempool logic designed to maintain ancestor/descendant state
fc4e3e6bc1 Remove unused members from CTxMemPoolEntry
ff3b398d12 mempool: eliminate accessors to mempool entry ancestor/descendant cached state
b9a2039f51 Eliminate use of cached ancestor data in miniminer_tests and truc_policy
ba09fc9774 mempool: Remove unused function CalculateDescendantMaximum
8e49477e86 wallet: Replace max descendant count with cluster_count
e031085fd4 Eliminate Single-Conflict RBF Carve Out
cf3ab8e1d0 Stop enforcing descendant size/count limits
89ae38f489 test: remove rbf carveout test from mempool_limit.py
c0bd04d18f Calculate descendant information for mempool RPC output on-the-fly
bdcefb8a8b Use mempool/txgraph to determine if a tx has descendants
69e1eaa6ed Add test case for cluster size limits to TRUC logic
9cda64b86c Stop enforcing ancestor size/count limits
1f93227a84 Remove dependency on cached ancestor data in mini-miner
9fbe0a4ac2 rpc: Calculate ancestor data from scratch for mempool rpc calls
7961496dda Reimplement GetTransactionAncestry() to not rely on cached data
feceaa42e8 Remove CTxMemPool::GetSortedDepthAndScore
21b5cea588 Use cluster linearization for transaction relay sort order
6445aa7d97 Remove the ancestor and descendant indices from the mempool
216e693729 Implement new RBF logic for cluster mempool
ff8f115dec policy: Remove CPFP carveout rule
c3f1afc934 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits
47ab32fdb1 Select transactions for blocks based on chunk feerate
dec138d1dd fuzz: remove comparison between mini_miner block construction and miner
6c2bceb200 bench: rewrite ComplexMemPool to not create oversized clusters
1ad4590f63 Limit mempool size based on chunk feerate
b11c89cab2 Rework miner_tests to not require large cluster limit
95a8297d48 Check cluster limits when using -walletrejectlongchains
95762e6759 Do not allow mempool clusters to exceed configured limits
edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters
435fd56711 test: update feature_rbf.py replacement test
34e32985e8 Add new (unused) limits for cluster size/count
838d7e3553 Add transactions to txgraph, but without cluster dependencies
a7c96f874d tests: Add witness commitment if we have a witness transaction in FullBlockTest.update_block()
096924d39d kernel: add btck_block_tree_entry_equals
ffcae82a68 test: exercise TransactionMerklePath with empty block; targets the MerkleComputation empty-leaves path that was only reached by fuzz tests
d5ed9cb3eb Add accessor for sigops-adjusted weight
1bf3b51396 Add sigops adjusted weight calculator
c18c68a950 Create a txgraph inside CTxMemPool
29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref
92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying
24ed820d4f merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot`
63d640fa6a merkle: remove unused `proot` and `pmutated` args from `MerkleComputation`
be270551df merkle: migrate `path` arg of `MerkleComputation` to a reference
866bbb98fd cmake, test: Improve locality of `bitcoin_ipc_test` library description
ae2e438b25 cmake: Move IPC tests to `ipc/test`
48840bfc2d refactor: Prefer `<=>` over multiple relational operators
5a0f49bd26 refactor: Remove all `operator!=` definitions
0ac969cddf validation: don't reallocate cache for short-lived CCoinsViewCache
c8f5e446dc coins: reduce lookups in dbcache layer propagation
1db7491470 depends: sqlite 3.50.4
286f3e49c8 guix: sqlite wants tcl
b0c706795c Remove unreliable seed from chainparams.cpp, and the associated README
6c73e47448 mempool: Store iterators into mapTx in mapNextTx
51430680ec Allow moving an Epoch::Marker
dcd42d6d8f [test] wallet send 3 generation TRUC
e753fadfd0 [wallet] never try to spend from unconfirmed TRUC that already has ancestors
fa6db79302 test: Avoid shutdown race in NetworkThread
a1f7623020 qa: Only complain about expected messages that were not found
1e54125e2e refactor(qa): Avoid unnecessary string operations
a9021101dc qa: Replace always-escaped regexps with "X in Y"
5c16e4631c doc: Remove no longer correct comment
facd01e6ff refactor: remove redundant locator cleanup in BaseIndex::Init()
c1e554d3e5 refactor: consolidate 3 separate locks into one block
41479ed1d2 test: add test for periodic flush inside ActivateBestChain
84820561dc validation: periodically flush dbcache during reindex-chainstate

git-subtree-dir: depend/bitcoin
git-subtree-split: 938d7aacabd0bb3784bb3e529b1ed06bb2891864
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

guix: Unable to reproduce macOS SDK tarball on Fedora 40