-
Notifications
You must be signed in to change notification settings - Fork 38.7k
util: Work around libstdc++ create_directories issue #24338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Concept ACK. Mind applying this diff: #24266 (comment) ? Otherwise the blocks dir handling differs from the wallet dir handling (and root dir handling). |
36a6815 to
e6fafbd
Compare
|
|
I'm really confused by the CI output: This change doesn't actually create any symbolic links, it just tolerates them, neither does it change any Python code, so I don't understand. |
|
I've haven't been following the issue, but the code makes sense, util and dbwrapper unit tests are green for me on Debian with clang 13.0.1, but
|
e6fafbd to
01c5aef
Compare
|
I've reverted @MarcoFalke 's proposed change for now. Let's see if this passes CI. |
Interesting. For reference, the failed CI is here: https://cirrus-ci.com/build/6366611337641984 It would be good to know why it failed. Otherwise it seems that the same failure that happened on the wallet dir is going to happen on the blocks dir as well, with this patch. |
This was the first thing I checked (as it's the problem I'd been suffering from), and it worked fine. Edit: could it be a problem with making multiple directories at once? I mean, this looks if a path exists before making a subdirectory of it: It's not the same as |
|
01c5aefa032f1f7d28c132481624e2d74aa1d14b solves the |
|
Ugh, my bad. 🤦♂️ |
01c5aef to
b9dff10
Compare
|
utACK b9dff100ff2e7dfb9f216e98964fe28ef5bac3a9 The disabled function corresponds to the second |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the blocks directory is a symlink, as mentioned in #24257 (comment), the error below still occurs.
************************
EXCEPTION: NSt10filesystem7__cxx1116filesystem_errorE
filesystem error: cannot create directories: Not a directory [/home/.../.bitcoin/signet/blocks]
bitcoin in AppInit()
|
Is it possible to write a test for this with https://docs.python.org/3.8/library/os.html#os.symlink ? (Like the multiwallet test) |
Strange, it works here on Ubuntu 20.04. The Without this patch I get With this patch, it works: I have checked the same for signet. Both making the
Sure. I'll look into it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@w0xlt, @laanwj, I don't have a buggy gcc to experiment, but my guess is that if signet is a symlink and inside blocks is an ordinary directory, then is_symlink("signet/blocks") will return false. In this case our condition will be false and we will call the buggy/system create_directories(). It should return false in that case but maybe it throws instead because of the bug?
|
My bad. The symlink was created using a relative path instead of an absolute one. |
|
@w0xlt, so the link was broken, pointing to a nonexistent file/directory? |
|
Yes, the link was broken. The relative path did not exist in the |
|
Thanks for double-checking @w0xlt. |
|
Concept ACK, reviewing... |
hebasto
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approach ACK b9dff100ff2e7dfb9f216e98964fe28ef5bac3a9.
Maybe add a unit test which being applied to the master branch passes on systems that have libstdc++ already being patched?
vasild
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK b9dff100ff2e7dfb9f216e98964fe28ef5bac3a9
I managed to reproduce this and played a bit with it. This PR will fix the problem where the last component of the path is a symlink to a directory (instead of just directory).
Assuming buggy system create_directories(), then in some cases our newly introduced create_directories() still deviates from the standard - e.g. if the last component exists but is not a directory, then we would call the system function which will throw, whereas, according to the standard it should just return false. I think this is ok.
|
Added @hebasto's unit test (and using fully qualified |
b9dff10 to
d035b64
Compare
vasild
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK d035b64aa0d1b23f34698cc20df65dd613b4adf7
d035b64 to
67019cd
Compare
|
New update:
|
w0xlt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tACK 67019cd on Ubuntu 20.04
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wasn't able to make the tests fail using gcc 8.4 without the fix but am on debian testing and turns out it has libstdc++ patched. Hopefully another reviewer can see them fail without the fix (thanks for adding them).
re-ACK 67019cde228feef8c41e6c685d62a21e85a2cfd9
hebasto
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 67019cde228feef8c41e6c685d62a21e85a2cfd9
Ubuntu 20.04 is still unpatched, seems the best OS to test this on. Haven't tried it on 22.04 yet (edit: just did—also not patched yet). It makes sense for debian testing to be ahead. It would be kind of funny if by the time we release, all distros have been patched. That said, the tests are useful to keep. |
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Work around libstdc++ issue [PR101510] with create_directories where the leaf already exists as a symlink. Fixes bitcoin#24257, introduced by the switch to `std::filesystem`. It is meant to be more thorough than bitcoin#24266, which only worked around one instance of the problem. The issue was fixed upstream in https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=124eaa50e0a34f5f89572c1aa812c50979da58fc, but unfortunately we'll have to carry a fix for it for a while. This introduces a function `fs::create_directories` which wraps `std::filesystem::create_directories`. This allows easiliy reverting the workaround when it is no longer necessary.
67019cd to
b223c3c
Compare
hebasto
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK b223c3c
👍 re-ACK b223c3c per Nice touch adding the splat to |
|
Guix build: bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
075732c9a4af5d0c3cc09bc7bf8c15ef41c62d0984c4556c2756fb1d684ddc11 guix-build-b223c3c21e89/output/aarch64-linux-gnu/SHA256SUMS.part
fd13a2477eb6400aa00200e46df3dd3734e54f19f156807ba8e0fe2d1659dafb guix-build-b223c3c21e89/output/aarch64-linux-gnu/bitcoin-b223c3c21e89-aarch64-linux-gnu-debug.tar.gz
8c1c6cd750fdf1cc6f3990a1d15a03076fcdb3fef76916359d76b11370f11e73 guix-build-b223c3c21e89/output/aarch64-linux-gnu/bitcoin-b223c3c21e89-aarch64-linux-gnu.tar.gz
0d957e31ab5645703be7fc62d09c1096677c25b395f97bed11ae7bf1b4da0a44 guix-build-b223c3c21e89/output/arm-linux-gnueabihf/SHA256SUMS.part
44fdc0ddfc86eab2a747f4087bbcf79b532b5d7dd5a77b1ef33b49f5c65a95b9 guix-build-b223c3c21e89/output/arm-linux-gnueabihf/bitcoin-b223c3c21e89-arm-linux-gnueabihf-debug.tar.gz
456b5b91302568e59ee8d5329d065efffcbd7fd76a064cd9c67e7f404dc7249d guix-build-b223c3c21e89/output/arm-linux-gnueabihf/bitcoin-b223c3c21e89-arm-linux-gnueabihf.tar.gz
3a0c3420152e95c88ac67ecf114e88a43af2b82b17cf5778b66645b4273bf995 guix-build-b223c3c21e89/output/arm64-apple-darwin/SHA256SUMS.part
c2f03f7866bdb1455a277dd2c2539facbf0a64ed256ab226e61422dec60098eb guix-build-b223c3c21e89/output/arm64-apple-darwin/bitcoin-b223c3c21e89-arm64-apple-darwin.tar.gz
a2093272f16783e68a207fe7ece4f7159dea079c29315f72d4a995201542b514 guix-build-b223c3c21e89/output/arm64-apple-darwin/bitcoin-b223c3c21e89-osx-unsigned.dmg
7c98dcd13c45af6e4039a0cde255818552981c5690dedb764b89bc84c9b09d97 guix-build-b223c3c21e89/output/arm64-apple-darwin/bitcoin-b223c3c21e89-osx-unsigned.tar.gz
c5e1be7e7fd68fc9b0cc87829c94c12dd00386905961b012f778cd1a2eaca1bc guix-build-b223c3c21e89/output/dist-archive/bitcoin-b223c3c21e89.tar.gz
b6044165141e6dbbf8a2deac093b46bc7216a8f8027a41d551eb6adeb6b35caa guix-build-b223c3c21e89/output/powerpc64-linux-gnu/SHA256SUMS.part
db025c0407b73aa6ba385d508d3cb9fc5483271fa92bc0cc09065f10230c5d3d guix-build-b223c3c21e89/output/powerpc64-linux-gnu/bitcoin-b223c3c21e89-powerpc64-linux-gnu-debug.tar.gz
ca4285de33997bd7e6cd3a9ce41038221c488d86c6dcd082c2479d8c18cd0c09 guix-build-b223c3c21e89/output/powerpc64-linux-gnu/bitcoin-b223c3c21e89-powerpc64-linux-gnu.tar.gz
e12d1e7864f79a37639156751b3b03c4e2f9bf9ca53ac9ae279aeabaa1f692ba guix-build-b223c3c21e89/output/powerpc64le-linux-gnu/SHA256SUMS.part
94b7da0b74207974acb5a3d614397c04acb56a07ae1a97d6e782760ff24271e2 guix-build-b223c3c21e89/output/powerpc64le-linux-gnu/bitcoin-b223c3c21e89-powerpc64le-linux-gnu-debug.tar.gz
7e64a5d5a7f586244684a4c7b275640fb09b82b9591c48b0440b7e5ca16b9781 guix-build-b223c3c21e89/output/powerpc64le-linux-gnu/bitcoin-b223c3c21e89-powerpc64le-linux-gnu.tar.gz
38277de2c4cc5c539796a92e4c97f4dec4eb8a39aee32af97a21fca02e0a3fa7 guix-build-b223c3c21e89/output/riscv64-linux-gnu/SHA256SUMS.part
d89076a6d7c7d693aea0a3ca2ac49e76f2c72ae61261ba1623b34b2d03f4937f guix-build-b223c3c21e89/output/riscv64-linux-gnu/bitcoin-b223c3c21e89-riscv64-linux-gnu-debug.tar.gz
57741dadfbe3c8b744e3e700a1b8c314d442d99ef3aff281efe4428e5938e8f0 guix-build-b223c3c21e89/output/riscv64-linux-gnu/bitcoin-b223c3c21e89-riscv64-linux-gnu.tar.gz
762aa6260628afbaf5cee1b60a4fcecc7de5453b1b526bbba222c85b33e1ac7f guix-build-b223c3c21e89/output/x86_64-apple-darwin/SHA256SUMS.part
21a5d675de49c2cc2c94406e1d5b5565c0c04d8c7f21b7d8fde111dd68ffcb9f guix-build-b223c3c21e89/output/x86_64-apple-darwin/bitcoin-b223c3c21e89-osx-unsigned.dmg
1caf7a2115fc0aac3216cd06cdb4eb3c715299f6e707eb6ebe0cd62acffa2621 guix-build-b223c3c21e89/output/x86_64-apple-darwin/bitcoin-b223c3c21e89-osx-unsigned.tar.gz
81b0cc3cbc308ebc6b778fdfc2f38c550e273883c47f909d12b7e096d3d950ca guix-build-b223c3c21e89/output/x86_64-apple-darwin/bitcoin-b223c3c21e89-osx64.tar.gz
0a3b41c634a9952c0084fd43f757971e9b55dd099b850b4585621be263ed01cf guix-build-b223c3c21e89/output/x86_64-linux-gnu/SHA256SUMS.part
c0d461b1006201f2ae202019f06cb4709a2627f0e28a8cd9ebafed6ee8bd8231 guix-build-b223c3c21e89/output/x86_64-linux-gnu/bitcoin-b223c3c21e89-x86_64-linux-gnu-debug.tar.gz
31bcf08af8012bfbff3e417c784ee0d546f916d24495143cff581a23e0bb9690 guix-build-b223c3c21e89/output/x86_64-linux-gnu/bitcoin-b223c3c21e89-x86_64-linux-gnu.tar.gz
c9bb21542cdb876ef9e7dd4b4d18dda2955d56a003b5e06ff171f8480d52bb15 guix-build-b223c3c21e89/output/x86_64-w64-mingw32/SHA256SUMS.part
ccf12d1929669f1edbe80496e7cd48b96eb336de596c46b666802c784aec9808 guix-build-b223c3c21e89/output/x86_64-w64-mingw32/bitcoin-b223c3c21e89-win-unsigned.tar.gz
4f91e09b2ba0b130e101ce3e8616463d8f4a72098c9adc1ac46fa7aa5bf92188 guix-build-b223c3c21e89/output/x86_64-w64-mingw32/bitcoin-b223c3c21e89-win64-debug.zip
6d40204e81732857cf28a32262e9ae4170714627efc81ebe7ddad7b50faa45b1 guix-build-b223c3c21e89/output/x86_64-w64-mingw32/bitcoin-b223c3c21e89-win64-setup-unsigned.exe
9c6881f8734ffa8b0765aa791b10c28e8ce0f983ca35b3ba2ddf2577173cdd58 guix-build-b223c3c21e89/output/x86_64-w64-mingw32/bitcoin-b223c3c21e89-win64.zip |
w0xlt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK b223c3c
vasild
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK b223c3c
…ssue b223c3c test: Add functional test for symlinked blocks directory (laanwj) ddb75c2 test: Add fs_tests/create_directories unit test (Hennadii Stepanov) 1f46b6e util: Work around libstdc++ create_directories issue (laanwj) Pull request description: Work around libstdc++ issue [PR101510] with create_directories where the leaf already exists as a symlink. Fixes bitcoin#24257, introduced by the switch to `std::filesystem`. It is meant to be more thorough than bitcoin#24266, which worked around one instance of the problem. The issue was [fixed upstream](https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=124eaa50e0a34f5f89572c1aa812c50979da58fc), but unfortunately we'll have to carry a fix for it for a while. This introduces a function `fs::create_directories` which wraps `std::filesystem::create_directories`. This allows easiliy reverting the workaround when it is no longer necessary. ACKs for top commit: jonatack: re-ACK b223c3c per `git range-diff df08250 67019cd b223c3c` hebasto: re-ACK b223c3c w0xlt: re-ACK b223c3c vasild: ACK b223c3c Tree-SHA512: 028321717c8b10d16185c3711b35da6b05fb7aa31cee1c8c7e754e92bf5a0b02719a3785cd0f6f8bf052b3bd759f644af212320672baabc9e44e0b93ba464abc
Work around libstdc++ issue [PR101510] with create_directories where the leaf already exists as a symlink. Fixes #24257, introduced by the switch to
std::filesystem. It is meant to be more thorough than #24266, which worked around one instance of the problem.The issue was fixed upstream, but unfortunately we'll have to carry a fix for it for a while.
This introduces a function
fs::create_directorieswhich wrapsstd::filesystem::create_directories. This allows easiliy reverting theworkaround when it is no longer necessary.