Support for file URLs #3748
Merged
Support for file URLs #3748
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PastelMobileSuit
suggested changes
Jul 31, 2019
Contributor
PastelMobileSuit
left a comment
PastelMobileSuit
left a comment
There was a problem hiding this comment.
Looks good! ✨ I just had a couple of minor suggestions
| grep "locks/verify$" pushcustom.log && false | ||
|
|
||
| grep "xfer: started custom adapter process" pushcustom.log | ||
| grep "Uploading LFS objects: 100% (12/12)" pushcustom.log |
Contributor
There was a problem hiding this comment.
Would it be worth explicitly checking here that the files ended up at the correct destination?
Member
Author
There was a problem hiding this comment.
I can do that. They won't have been able to be downloaded if they didn't, but I think it's fine to be more explicit.
| config *config.Configuration | ||
| } | ||
|
|
||
| func fileUrlFromRemote(cfg *config.Configuration, name string, direction string) (*url.URL, error) { |
Contributor
There was a problem hiding this comment.
I think it would be worth adding some more detailed comments here on what each of the functions are doing. Since this is at least partially intended as an example, that'll help make it easier for people to use it as such
Member
Author
There was a problem hiding this comment.
That sounds like a good idea.
bk2204
force-pushed
the
standalone-file
branch
2 times, most recently
from
363ec3e to
de04fe5
Compare
PastelMobileSuit
approved these changes
Aug 1, 2019
Contributor
PastelMobileSuit
left a comment
PastelMobileSuit
left a comment
There was a problem hiding this comment.
Looks great! ✨ I found one typo that needs to be fixed, once that's worked out and the failing CI job is passing I'm happy for this to be merged
lfshttp/standalone/standalone.go
Outdated
| return oid, "", lfs.LinkOrCopy(h.remoteConfig, path, dest) | ||
| } | ||
|
|
||
| // download performs the upload action for the given OID and size. It returns |
bk2204
force-pushed
the
standalone-file
branch
3 times, most recently
from
0a0940c to
b4feea2
Compare
We don't currently handle file URLs, but we don't really want to tell people they're an invalid remote type, since we'll handle them later on. If the user doesn't have a custom transfer agent, we'll warn them that this will only work with such an agent at that point.
When we use the smudge call (as opposed to filter-process), we want to be able to look up a custom transfer agent based on the URL we're fetching from. In order to do so, pass the remote we're using in order to find the right URL and therefore the right custom transfer agent.
There are several cases in which we can pass in Unix-style paths in the Git Bash environment, which is MINGW64-based, including in file URLs. To make such paths work, let's accept such URLs with the Unix-style paths and then translate them to Windows-style paths automatically. Note that passing Windows-style paths to "cygpath -W", which is what we use in this case, works fine, so there's no regression in the Git Bash case.
One commonly requested feature for Git LFS is support for local files. Currently, we tell users that they must use a standalone transfer agent, which is true, but nobody has provided one yet. Since writing a simple transfer agent is not very difficult, let's provide one ourselves. Introduce a basic standalone transfer agent, git lfs standalone-file, that handles uploads and downloads. Add a default configuration required for it to work, while still allowing users to override this configuration if they have a preferred implementation that is more featureful. We provide this as a transfer agent instead of built-in because it avoids the complexity of adding a different code path to the main codebase, but also serves as a demonstration of how to write a standalone transfer agent for others who might want to do so, much like Git demonstrates remote helpers using its HTTP helper.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Jul 27, 2023
The three test functions in the tq/transfer_test.go source file are all named with the prefix "test" rather than "Test", and as a result, do not actually execute. This oversight dates from the original introduction of these tests in the "transfer" package in commit 10623f5 of PR git-lfs#1265. (The package was later renamed to the current "tq" package in commit 891db97 of PR git-lfs#1780.) We therefore change the test function names to begin with "Test", and resolve several test regressions which have accumulated since the tests were first added. First, the TestBasicAdapterExists() function calls the GetDownloadAdapterNames() and GetUploadAdapterNames() methods of the Manifest structure, and these now return the names of three transfer adapter implementations rather than just the original "basic" one, so we allow for all three names to appear in any order. (The "lfs-standalone-file" adapter was added in commit bb05cf5 of PR git-lfs#3748, and the "ssh" adapter was added in commit 594f8e3 of PR git-lfs#4446.) Second, the TestAdapterRegAndOverride() function expects the NewDownloadAdapter() and NewUploadAdapter() methods of the Manifest structure to return nil if the provided name argument does not match that of any registered transfer adapter. However, this has not been the behaviour of those methods since commit c5c2a75 of PR git-lfs#1279, shortly after the tests were first introduced in PR git-lfs#1265. In that commit, the NewAdapterOrDefault() method was added, and the NewDownloadAdapter() and NewUploadAdapter() revised to call it, so they return the default "basic" adapter if the requested name does not match a registered adapter. We therefore revise and expand the test to account for this behaviour, and also make sure to directly test the simpler NewAdapter() method, which retains the originally intended behaviour and returns nil if it does not find a matching adapter for the provided name argument.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Jul 27, 2023
The three test functions in the tq/transfer_test.go source file are all named with the prefix "test" rather than "Test", and as a result, do not actually execute. This oversight dates from the original introduction of these tests in the "transfer" package in commit 10623f5 of PR git-lfs#1265. (The package was later renamed to the current "tq" package in commit 891db97 of PR git-lfs#1780.) We therefore change the test function names to begin with "Test", and resolve several test regressions which have accumulated since the tests were first added. First, the TestBasicAdapterExists() function calls the GetDownloadAdapterNames() and GetUploadAdapterNames() methods of the Manifest structure, and these now return the names of three transfer adapter implementations rather than just the original "basic" one, so we allow for all three names to appear in any order. (The "lfs-standalone-file" adapter was added in commit bb05cf5 of PR git-lfs#3748, and the "ssh" adapter was added in commit 594f8e3 of PR git-lfs#4446.) Second, the TestAdapterRegAndOverride() function expects the NewDownloadAdapter() and NewUploadAdapter() methods of the Manifest structure to return nil if the provided name argument does not match that of any registered transfer adapter. However, this has not been the behaviour of those methods since commit c5c2a75 of PR git-lfs#1279, shortly after the tests were first introduced in PR git-lfs#1265. In that commit, the NewAdapterOrDefault() method was added, and the NewDownloadAdapter() and NewUploadAdapter() revised to call it, so they return the default "basic" adapter if the requested name does not match a registered adapter. We therefore revise and expand the test to account for this behaviour, and also make sure to directly test the simpler NewAdapter() method, which retains the originally intended behaviour and returns nil if it does not find a matching adapter for the provided name argument. Third, the TestAdapterRegButBasicOnly() function, which passes without changes, no longer fully performs the checks it was intended to make, since the NewDownloadAdapter() and NewUploadAdapter() methods now always return a non-nil value, so using a non-nil response from them to prove that the "test" adapter is found is insufficient. We therefore update the test to confirm that the returned value from these functions is a "test" adapter, as expected, and not just a "basic" one. We also replace the use of the BasicAdapterName variable with the "basic" string to align with the other tests.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Jul 27, 2023
The three test functions in the tq/transfer_test.go source file are all named with the prefix "test" rather than "Test", and as a result, do not actually execute. This oversight dates from the original introduction of these tests in the "transfer" package in commit 10623f5 of PR git-lfs#1265. (The package was later renamed to the current "tq" package in commit 891db97 of PR git-lfs#1780.) We therefore change the test function names to begin with "Test", and resolve several test regressions which have accumulated since the tests were first added. First, the TestBasicAdapterExists() function calls the GetDownloadAdapterNames() and GetUploadAdapterNames() methods of the Manifest structure, and these now return the names of three transfer adapter implementations rather than just the original "basic" one, so we allow for all three names to appear in any order. (The "lfs-standalone-file" adapter was added in commit bb05cf5 of PR git-lfs#3748, and the "ssh" adapter was added in commit 594f8e3 of PR git-lfs#4446.) Second, the TestAdapterRegAndOverride() function expects the NewDownloadAdapter() and NewUploadAdapter() methods of the Manifest structure to return nil if the provided name argument does not match that of any registered transfer adapter. However, this has not been the behaviour of those methods since commit c5c2a75 of PR git-lfs#1279, shortly after the tests were first introduced in PR git-lfs#1265. In that commit, the NewAdapterOrDefault() method was added, and the NewDownloadAdapter() and NewUploadAdapter() methods revised to call it, so they return the default "basic" adapter if the requested name does not match a registered adapter. We therefore revise and expand the test to account for this behaviour, and also make sure to directly test the underlying NewAdapter() method, which retains the originally intended behaviour and returns nil if it does not find a matching adapter for the provided name argument. Third, although the TestAdapterRegButBasicOnly() function passes without changes, it no longer fully performs the checks it was intended to make, since the NewDownloadAdapter() and NewUploadAdapter() methods now always return a non-nil value, so using a non-nil response from them to prove that the "test" adapter was found is insufficient. We therefore update the test to confirm that the returned value from these functions is a "test" adapter, as expected, and not just a "basic" one. We also replace the use of the BasicAdapterName variable with the "basic" string to align with the other tests.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Oct 16, 2023
Our Server Discovery documentation describes how we form Git LFS server API endpoint URLs based on Git remotes and other Git and Git LFS configurations. This documentation was introduced in commit 21e1695 of PR git-lfs#1641 and has only been lightly modified since then. As discusssed in issue git-lfs#5528, we append "/info/lfs" to Git remote URLs, but only after rewriting those URLs according to any "url.{name}.insteadOf" configurations, which is somewhat counter to what is implied by Git's own documentation of these settings. We may change this behaviour in a future Git LFS major release. For the present, though, we simply add a fuller description of how we convert Git remote URLs into Git LFS API server URLs to our documentation, specifically touching on the sequence in which we rewrite, convert, and append the "/info/lfs" path segments to the remote URLs. In PR git-lfs#3748 we added support for Git LFS endpoints defined by "file://" URLs, and in PR git-lfs#3918 we enhanced this by also accepting local file paths in remote URLs, so we also include comments on how we handle these cases in our new documentation. Further, although support for distinct URLs used only for upload operations has existed since PR git-lfs#949, it is not mentioned in our Server Discovery documentation, so we add notes on this feature now as well. 4469 - default remotes (lfsdefault, lfspushdefault) 5066 - autodetect, searchall
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One commonly requested feature for Git LFS is support for local files. Currently, we tell users that they must use a standalone transfer adapter, which is true, but nobody has provided one yet. Since writing a simple transfer adapter is not very difficult, let's provide one ourselves.
Introduce a basic standalone transfer adapter, git lfs standalone-file, that handles uploads and downloads. Add a default configuration required for it to work, while still allowing users to override this configuration if they have a preferred implementation that is more featureful. We provide this as a transfer adapter instead of built-in because it avoids the complexity of adding a different code path to the main codebase, but also serves as a demonstration of how to write a standalone transfer adapter for others who might want to do so, much like Git demonstrates remote helpers using its HTTP helper.
Note that this doesn't introduce support for plain paths, only file URLs, but this should be sufficient for most needs.
Fixes #3742.