Overview
A sparse file copy ➡️ within a sparse disk image formatted with APFS, ➡️ stored on an APFS volume on physical media gets corrupted when the right conditions are met. The OpenWRT build system is guaranteed to trigger this every time we run:
make tools/compile -j 16
make toolchain/gcc/initial/{clean,compile} -j 16.
Then make toolchain/{clean,compile} fails because we don't have a valid compiler.

Details

• First the build system compiles toolchain/gcc/initial, I can confirm that it works (in the build folder).
• These binaries contain segments that are allocated and segments that are not (they read as zeroes).
• Next it gets installed by coreutils/install, which performs a sparse copy since version 9.
• The installed copy does not work.
• So either some allocated segments are not reported by APFS as allocated and hence they are skipped.
• Or the code in coretils which identifies them has issues.
• We need to make coreutils and apple aware.

Note
A sparse copy is more efficient, since it only copies data that is allocated.
With APFS there is also copy on write. If you copy file A of size 1 GB to B, you still use 1 GB for both.
That's because both share the same disk area for the parts that have not changed since the copy.

I'm curious what conditions trigger this?

@httpstorm i would accept this patch as a workaround BUT we really need a way to tell coreutils that there is this kind of problem... To me it really seems a bug in apple virtual disk... a bug in compacting the virtual disk and something is dropped for sparse files?

A good idea would be create a repro env so coreutils guys can quickly repro this. (a good argument to gain some traction may be that we recently switched to new version on openwrt and this started to get broken and is caused by the new sparse feature)

@Ansuel
I agree. Any idea how to reach the Apple developers?
I mailed the patch and details to the coreutils team.
Let's wait for a week and see if they are willing to accept it as a workaround.
If not, we can add it to OpenWRT.

I also created a pull request, but it seems they don't like using Github.
Is there an easy way to contribute or report bugs to them?
coreutils/coreutils#68

Creating a repro environment
It would be very helpful if we can invite someone from gcc to create PoC code.
Their build environment triggers it every time.

steps to reproduce

Create a sparse disk image with APFS

• open Disk Utility
• press Command-N to create a new virtual disk
• Image Format: sparse disk image
• save as: ~/vhd/test
• Name: test
• Size: 50 GB
• Format: APFS (Case-sensitive)
• Encryption: none
• Partitions: Single partition - GUID Partition Map
• Save
• the image is mounted under /Volumes/test

clone and build coreutils

cd /Volumes/test
# patched with sparse support disabled on macOS
git clone https://github.com/httpstorm/coreutils.git
mv coreutils coreutils-no-sparse
cd coreutils-no-sparse
git checkout macos_disable-sparse-copy
git submodule foreach git pull origin master
./bootstrap
./configure
make -j 16
cd ..

# original code
git clone https://github.com/coreutils/coreutils.git
cd coreutils
git submodule foreach git pull origin master
./bootstrap
./configure
make -j 16

download OpenWRT and build gcc

cd /Volumes/test
git clone https://github.com/openwrt/openwrt.git
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a
make defconfig
make menuconfig
# Target System (Marvell EBU Armada)
# Target Profile (Linksys WRT3200ACM)
# Save, Exit
make tools/compile -j 16
make toolchain/gcc/initial/compile -j 16

proof of concept

cd /Volumes/test/openwrt
cd build_dir/toolchain-arm_cortex-a9+vfpv3-d16_gcc-12.2.0_musl_eabi/gcc-12.2.0-initial/gcc

/Volumes/test/coreutils/src/cp cc1 cc1-ori
./cc1-ori --help
-bash: ./cc1-ori: cannot execute binary file: Exec format error

/Volumes/test/coreutils-no-sparse/src/cp cc1 cc1-fix
./cc1-fix --help
# works correctly

sha1sum cc1
# fcf5d9458f54d5588efc5a76a89388925ca04eda  cc1
sha1sum cc1-fix 
# fcf5d9458f54d5588efc5a76a89388925ca04eda  cc1-fix
sha1sum cc1-ori
# 7b447132f56f3b549ef62a4780945e82d26f7d2c  cc1-ori

Good news, I got a replay on the coretils mailing list.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61386

We'll try to test for a better solution and fall-back to accepting the patch.

@httpstorm nice! then lets wait

We implemented a solution: for copies on the same filesystem, fclonefileat is used, which clones a file CoW. The clone is instant even for large files and uses the same disk sectors so no additional space is allocated. For copies from one filesystem to another, we fall-back to regular copy. On macOS sparse copy is disabled, since the API is broken. I will post an update when a new coreutils release is available.

edit: This link explains why sparse copies were corrupted.
https://lists.gnu.org/archive/html/bug-gnulib/2018-09/msg00054.html

@httpstorm is the patch got merged in coreutils repo? If it's the case then lets just backport the fix to the current release... It's totally ok to backport merged patch... not ok to add workaround without approval from the related repo.

@Ansuel No, it's not merged yet, and I expect two patches: the one in this pull request, and another to enable CoW. But let's wait until they confirm everything is merged. I posted earlier to keep you updated. I just tested the suggested patch and confirmed everything is ok. You can find our conversation and patch details on the mailing list. Link in a previous comment.

I found the underlaying issue: one way of creating sparse files is to map a file to memory view using mmap. Then we write to the memory view. Finally we munmap the view and close the file. This results in a valid file, however because we did not explicitly call msync, the filesystem is not fully updated. It will get updated eventually. Specifically the list of valid pages cached by the filesystem is old. As a result lseek(SEEK_DATA) does not return all of the valid pages, and install/cp create corrupted copies when sparse support is enabled (default). More info and PoC code here:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61386#158

I am still working with the coreutil team. We wanted to make sure we know the reason for the issue, and that compliance is maintained.

(me being very picky) please wrap the link with the commit tag Link:

Link: https:// .....
Signed-off-by: .....

has anyone looked into whether coreutils can be configured this way instead of patching?

has anyone looked into whether coreutils can be configured this way instead of patching?

@mpratt14 I'm still working with the coreutils team, you can check our discussion:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61386

They recently committed a few patches to Savannah, that address the issue, however that broke some of the tests. They added more patches yesterday but now the tests hang, so I just sent them a new report. Hopefully they will address it soon. The fix is way more sophisticated than my PR, so I will close it when everything settles, and we'll update to a new version of coreutils. I will keep you updated.
Apple is also aware of the issue and have reproduced it.

coreutils-9.2 should be released in about 1.5 weeks. I'll make a PR when it is ready.

@httpstorm news with this?

@Ansuel
The fix is already on Savannah master, but coreutils is still on v9.1. v9.2 has not been released yet. They were working on split and tee in the past days. I asked for an update.
edit: I just received a replay from coreutils

Best guess is 9.2 release is 5 days away.

@httpstorm btw if we notice problem with 9.2 backporting the patch is a way (just for reference)

Coreutils 9.2 is now released

@pixelb Thank you very much for all your work, Pádraig! I will send you the test results later. So far everything works well. I also installed and tested coreutils-9.2 from brew.
@Ansuel I am preparing to build OpenWRT in a fresh directory for the following targets: WRT3200ACM, x64 and 1043nd. I will come back in a few hours with the results and create a PR.

@pixelb @Ansuel
I need help. Updating to coreutils-9.2 is easy, and it compiles correctly. However the fix in gnulib also modifies m4 and there is no new release. The latest version is m4-1.4.19 from 2021-05-28 17:55. I would assume that we can clone gnulib, checkout f17d397771164c1b0f77fea8fb0abdc99cf4a3e1 or master dd723a3ed53cc3b969c6abdf7b0fb6ea8339079a, and build m4 from there. However I'm not sure how to configure tools/m4/Makefile to achieve this. We can also follow the steps for building coreutils without relying on tools/m4. The build process automatically downloads and builds gnulib and m4:

git clone git://git.savannah.gnu.org/coreutils.git
cd coreutils
./bootstrap
./configure
make -j 16

Once we update gnulib/m4, the following patch will no longer be needed:
001-m4.patch

I created a draft PR to update to coreutils-9.2. We can continue the discussion here: #12233

This was fixed by updating coreutils to version 9.2, see #12233