Add copy-on-write support for Linux BTRFS filesystem #952
Conversation
|
This looks good. However, as someone that doesn't use linux much and has 0 experience with the BTRFS, I'll have to rely on you to support this. Based on our previous interactions, I don't think that'll be a problem, but I wanted to make that clear :) This doesn't work when I build the linux binaries: Probably because I'm building cross platform binaries without cgo. Should be easy to fix with another file with a build flag like What will this do on a linux system without btrfs? Will that syscall just return an error, causing |
|
I write wrong build condition: It's also means: you build without cgo
Btrfs is supported on linux a long time (for example Ubuntu 11.04 supports btrfs filesystem). In situation with unknown syscall it simply return error and failback to block-by-block copy. Failback to block-by-block copy also will be in any situations when copy-on-write is inpossible (files on different filesystems, files on ext4, too old btrfs implementation without copy-on-write and etc). In benefits btrfs users have:
All other user have penalty by one redundant syscall per file. I checked this code locally on Ubuntu 14.04 ext4 and btrfs. Personally, I'm most upset me, I do not know how to implement a similar function under Windows :( |
|
Cool, thanks for fixing the build flag. This should make it into a new release next week. I want to make sure my linux and windows vm tests all pass first. Thanks! |
|
This fails in the centos 5 docker build. $ ./docker/run_dockers.bsh centos_5
centos_5: Pulling from andyneff/git-lfs_dockers
Digest: sha256:e75cba3fb67d5b2a3490ea800bd50d15543e7529358367f1def9bf2e68cc18d3
Status: Image is up to date for andyneff/git-lfs_dockers:centos_5
Compiling LFS in docker image centos_5
...
+ ./script/bootstrap
lfs/util_linux.go
# github.com/github/git-lfs/lfs
/usr/bin/ld: unrecognized option '--build-id=none'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit statusIt builds successfully in all of the other docker files: $ ./docker/run_dockers.bsh centos_{6,7} debian_{7,8}It looks like we can pass custom build flags. Would it be worth trying to pass flags like Though, I'm not even sure that would fix that |
|
If I remember correctly, golang doesn't like having a build ids in the debug symbols. This is basically a checksum of a portion of the binary data (Not all of it so that if you compile the same code twice you get the same checksum). This is important in debugging in Fedora operating systems because this is how debugging symbols are matched with binary files after being turned into RPMs, where the debug symbols and binary files are split into two. In the end, I gave up and just disabled the debug symbol handling in the RPMs. I'm guessing something in the Since go doesn't really support Centos 5... This might be a good excuse to drop the Centos 5 building... ;) |
|
About this build issue About CentOS 5
|
|
Is cgo not needed for this BTRFS feature? |
|
I use cgo for calculate correct BTRFS_IOC_CLONE value (I'm not sure that _IOW macro is platform-independent). |
|
I'm in favor of disabling CGO on centos 5. If it's worth just dropping all support, that works too :) Once that happens, we can merge this PR. |
Disable CGO for Centos 5 to assist #952
|
We just ended up disabling CGO in centos 5: We'll drop centos 5 support once it's no longer under "maintenance support". Thanks again for the patch! |
Add copy-on-write support for Linux BTRFS filesystem
We added initial support for the deduplication of Git LFS object files between the Git working tree and our internal Git LFS storage directories in commit ea16fd5 of PR git-lfs#952, using the copy-on-write reflink functionality provided by the Linux Btrfs filesystem. Since we didn't implement equivalent capabilities for any other operating systems, we defined two versions of a new CloneFile() function in our "lfs" package, one for Linux, and one for all other types of systems. Later, in commit 1ef2d43 of PR git-lfs#1265, we moved the CopyWithCallback() function from our "lfs" package into our "tools" package, so it would be more widely available for use in our code base. In that same PR, for instance, the function was first used outside of the "lfs" package, in the download() method of what is now the basicDownloadAdapter in our "tq" package. Because the only caller of the CloneFile() function at the time was the CopyWithCallback() function, both versions of the function were copied to the "tools" package in commit 1ef2d43. However, while the Linux version was completely moved to the "tools" package by renaming the lfs/util_linux.go source file, the generic version was only copied to a new tools/util_generic.go source file, leaving the original lfs/util_generic.go in place. As the lfs/util_generic.go source file and the CloneFile() function it defines within the "lfs" package are no longer used, we can simply delete them now.
In commit 6006acc in PR git-lfs#976 we introduced a conditional build clause to the SPEC file use we to build RPM Linux packages, in an attempt to avoid compiling with cgo enabled on Red Hat Enterprise Linux (RHEL) 5 and CentOS 5 platforms. This change was made after the use of the "C" pseudo- module was introduced into the Git LFS client code in commit ea16fd5 of PR git-lfs#952, as part of our initial support for copy-on-write file creation. In practice, though, this conditional build clause was ineffective, according to the notes in PR git-lfs#976. Instead, one of the Dockerfiles used to build RPM packages for the RHEL/CentOS 5 platform was updated in commit git-lfs/build-dockers@39cec36 of PR git-lfs/build-dockers#1 to set the CGO_ENABLED environment variable with a value of "0". The conditional build clause in our SPEC file was never removed afterwards, however. Later, the Dockerfile in which the CGO_ENABLED environment variable was set was removed in commit git-lfs/build-dockers@d4c2fe6, and the primary Dockerfile used for building packages for the RHEL/CentOS 5 platform was then dropped in commit git-lfs/build-dockers@64a3a9f of PR #3. This followed the removal of the references to the RHEL/CentOS 5 platform from the scripts in our "dockers" directory in commit b560b85 of PR git-lfs#1298. As there is no value to retaining the conditional build clause for the RHEL/CentOS 5 platform in the SPEC file for our RPM packages, we simply remove it now.
3 participants
Use btrfs copy-on-write feature for copy lfs files from .git/lfs/objects to working copy (
git lfs checkout).Copy-on-write operation is look like hardlink, but change in one file is not visible in another file.
This command works like:
cp --reflink=auto src.txt dst.txtFor this change I use https://github.com/wertarbyte/coreutils/blob/master/src/copy.c as reference.