Skip to content

cp,mv,install: Disable sparse copy on macOS#68

Closed
httpstorm wants to merge 1 commit intocoreutils:masterfrom
httpstorm:macos_disable-sparse-copy
Closed

cp,mv,install: Disable sparse copy on macOS#68
httpstorm wants to merge 1 commit intocoreutils:masterfrom
httpstorm:macos_disable-sparse-copy

Conversation

@httpstorm
Copy link

@httpstorm httpstorm commented Feb 9, 2023 •
edited
Loading

Due to a bug in macOS, sparse copies are corrupted on virtual disks formatted with APFS. HFS is not affected. Affected are coreutils install, and gcp when compiled with SEEK_HOLE, as well as macOS Finder.

While reading the entire file returns valid data, scanning for allocated segments may return holes where valid data is present. In this case a sparse copy does not contain these segments and return zeroes instead. Once the virtual disk is dismounted and then mounted again, a sparse copy produces correct results.

OpenWRT recently switched to coreutils-9.1, and we need to fix the build environment.
The new sparse copy support since version 9 breaks the build on macOS:
openwrt/openwrt#11960
openwrt/openwrt#11960 (comment)

The issue is most likely within APFS, So we need to bring it to Apple's attention.
Since it is triggered by building gcc, it would be helpful if they can shine some light how their tools are build and what trigger this, in order to create PoC code.

steps to reproduce

Create a sparse disk image with APFS

  • open Disk Utility
  • press Command-N to create a new virtual disk
  • Image Format: sparse disk image
  • save as: ~/vhd/test
  • Name: test
  • Size: 50 GB
  • Format: APFS (Case-sensitive)
  • Encryption: none
  • Partitions: Single partition - GUID Partition Map
  • Save
  • the image is mounted under /Volumes/test

clone and build coreutils

cd /Volumes/test
# patched with sparse support disabled on macOS
git clone https://github.com/httpstorm/coreutils.git
mv coreutils coreutils-no-sparse
cd coreutils-no-sparse
git checkout macos_disable-sparse-copy
git submodule foreach git pull origin master
./bootstrap
make -j 16
cd ..

# original code
git clone https://github.com/coreutils/coreutils.git
cd coreutils
git submodule foreach git pull origin master
./bootstrap
make -j 16

download OpenWRT and build gcc

cd /Volumes/test
git clone https://github.com/openwrt/openwrt.git
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a
make defconfig
make menuconfig
# Target System (Marvell EBU Armada)
# Target Profile (Linksys WRT3200ACM)
# Save, Exit
make tools/compile -j 16
make toolchain/gcc/initial/compile -j 16

proof of concept

cd /Volumes/test/openwrt
cd build_dir/toolchain-arm_cortex-a9+vfpv3-d16_gcc-12.2.0_musl_eabi/gcc-12.2.0-initial/gcc

/Volumes/test/coreutils/src/cp cc1 cc1-ori
./cc1-ori --help
-bash: ./cc1-ori: cannot execute binary file: Exec format error

/Volumes/test/coreutils-no-sparse/src/cp cc1 cc1-fix
./cc1-fix --help
# works correctly

sha1sum cc1
# fcf5d9458f54d5588efc5a76a89388925ca04eda  cc1
sha1sum cc1-fix 
# fcf5d9458f54d5588efc5a76a89388925ca04eda  cc1-fix
sha1sum cc1-ori
# 7b447132f56f3b549ef62a4780945e82d26f7d2c  cc1-ori

@httpstorm
cp,mv,install: Disable sparse copy on macOS
c99e598 
Due to a bug in macOS, sparse copies are corrupted on virtual disks
formatted with APFS. HFS is not affected. Affected are coreutils
install, and gcp when compiled with SEEK_HOLE, as well as macOS Finder.

While reading the entire file returns valid data, scanning for
allocated segments may return holes where valid data is present.
In this case a sparse copy does not contain these segments and return
zeroes instead. Once the virtual disk is dismounted and then
mounted again, a sparse copy produces correct results.

This breaks OpenWRT build on macOS. Details:
openwrt/openwrt#11960

Signed-off-by: Georgi Valkov <[email protected]>
@httpstorm httpstorm mentioned this pull request Feb 9, 2023
Closed
@pixelb
Copy link
Member

pixelb commented Feb 9, 2023

Thanks for the patch.
Moving this to mailing list:
https://bugs.gnu.org/61386

@pixelb pixelb closed this Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@httpstorm @pixelb