I will amend the commit with the bug id in the coreutils bugtracker once the bug is opened

I'm happy to run a darwin build tonight to check if the issue is resolved.

On chat I understood that @r-burns did that (bisecting coreutils and building jx), but confirmation from elsewhere would be nice. Especially as Hydra doesn't progress fast on this and it can't even cancel whole evaluations at this moment :-/

@happysalada are you using zfs on darwin by any chance?
We know zfs has issues with SEEK_HOLE at least.
If not zfs what file system is the copy being done from. Thanks

@happysalada are you using zfs on darwin by any chance? We know zfs has issues with SEEK_HOLE at least. If not zfs what file system is the copy being done from. Thanks

To my knowledge we could also reproduce (consistently even) with tmpfs and ext4. Edit: On Linux

Well, for now we need this PR as mitigation, regardless of whose bug it is or how exactly it happens.

Not using zfs, just a regular darwin systemd with apfs.
I've started that "quick" jx build (8 hours ago), I should be done tomorrow morning. Almost there.

I was wondering why my build server was having is issues:

sudo zpool status -v nixstore
  pool: nixstore
...
errors: Permanent errors have been detected in the following files:

        /nix/store/.links
        
$ ls /nix/store/.links
"/nix/store/.links": Invalid exchange (os error 52)

I've disabled auto-optimise for now. But if I want to re-enable it I would have to destroy that pool

To my knowledge we could also reproduce (consistently even) with tmpfs and ext4. Edit: On Linux

That would be very surprising. A minimal reproducer would be exceptionally useful

My guess is that coreutils is the victim of a kernel or filesystem bug here. If so, that bug really needs fixing as other programs use lseek+SEEK_DATA too. See my comment in GNU coreutils Bug#51433.

Ok, jx and alacritty build fine on this PR, all is good with darwin!
Happy to merge whenever (unless you still want more feedback).
I would say let's merge now and get the staging-next cycle merged as well.

To my knowledge we could also reproduce (consistently even) with tmpfs and ext4. Edit: On Linux

That would be very surprising. A minimal reproducer would be exceptionally useful

After further thought and investigation with the help of others I strongly believe this to be a misattribution. Identifying the root cause in the case of peertube was difficult because of the unclear error message and strange reproducibility. I believe we downloaded dependencies from peertube from the so-called binary cache (which likely has zfs-backed builders) one of which was esbuild the actual broken derivation. So this issue is pretty surely not actually on ext4/tmpfs. I'm sorry for causing confusion.

@mohe2015 Thanks for all the info and testing. We'll focus on handling ZFS in upstream coreutils

Apparently this PR broke one test on coreutils-prefixed + aarch64-linux.

FAIL: tests/misc/cut-huge-range
===============================

FAIL tests/misc/cut-huge-range.sh (exit status: 1)

I reproduced the issue on the community box and also retried the two unprefixed builds and x86_64-linux builds, and all seems reproducible with failure only in this single combination. So far I have no idea why it happens and why --program-prefix=g apparently makes a difference here.

Without looking at the code I assume that test would also need to be skipped like the other test.

Currently testing around this issue but it takes a lot of time because my aarch64 hardware isn't the fastest

Can look at the code right now but maybe it's reproducible even on a slow HDD on x86

It built for me on aarch64-linux. So it's probably just a flakey test. Should I disable the test or should we just restart the build on Hydra, @vcunat ?

It seemed reproducible to me. BTW, I only have https://github.com/nix-community/aarch64-build-box#want-access

Either way, the coreutils-prefixed derivation doesn't seem important, so I don't consider it such a hot issue.