-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Encrypted SSL key support #3270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
ttaylorr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great to me. I think that the changes in package creds are fine, and I appreciate the way that you broke things up.
It made it easy to review, since the majority of the change is tied up in 12fd637, which is relatively small.
I left a few nits, but as always you are more than free to ignore them. I'm marking this as approved, so you're welcome to merge this as-is, but it would be good for @PastelMobileSuit to look as well.
|
Looks like this is failing on Travis. I’ll need to spin up a test system running Ubuntu and figure out what's up there. |
1875961 to
195b353
Compare
The shell function that performed the clone for the client certificate test contained several branches, each of which ended with an exit. This caused the test to always exit early (and in the normal case, successfully) without running the rest of the test. Change this function to use "return" instead of "exit" so that the test runs to completion, and fix the single failing assertion. Ignore this test on Travis CI, just like we do for the other SSL test, for exactly the same reason. Note that this appears to be the only test helper that has this problem: the other uses of "exit" in the test helpers are all either to skip tests or to exit unsuccessfully, the latter of which is equivalent to returning unsuccessfully under set -e (which we use).
The lfstest credential helper provides credentials found by reading a file on the system. Currently, it expects that there is a colon in the URL followed by one or more slashes. If there is not (say, because we're using a file:/// or cert:/// URL), it fails to concatenate the path properly, and instead of looking in the credentials directory, looks for a different file one directory higher. This occurs because filepath.Join strips off a trailing slash if the second argument passed to it is empty. This, of course, makes the credential helper fail to work and the tests hang waiting for input. Insert an additional slash in the environment variable and handle the empty string case properly so that credentials for local paths can be looked up properly.
In the future, we'll want to run integration tests that use an encrypted private key. To make this as easy as possible, create an encrypted private key as part of the test setup. Set up credentials for this key using the credential helper. Note that in order to get Git to use the credential helper for the encrypted private key instead of prompting using OpenSSL, we need to set GIT_SSL_CERT_PASSWORD_PROTECTED, which makes Git think that the certificate is password-protected, too. Therefore, specify credentials for both the certificate (which isn't really encrypted) and the encrypted private key.
We don't use the defaultCredentialHelper variable, so remove it.
We have several persistent credential helpers which currently live in the lfsapi client. In order to make this code reusable among multiple packages, place all of these credential helpers into a context and initialize it by using a new constructor. In addition, store a URL configuration object, since we'll need that in the context in the future.
In a future commit, we're going to split out the credential functionality into its own package. In preparation for that step, turn the getCredentialHelper method into a method of CredentialHelperContext, passing all its arguments explicitly.
We're going to move credential handling into its own package in a future commit, so expose the null credential helper as a public variable so we can access it.
We're going to move credential handling into its own package in a future commit, so expose the credential cacher constructor as a public function so we can access it.
In a future commit, we're going to be accessing credential handling from multiple packages. To avoid an import loop, move credential handling into its own package. Update all the callers of the credential handling code to use a qualified name. Where there is a local variable called "creds", which would conflict with our package name, rename it "cred" instead.
When using client certificates for TLS, it's possible to specify a private key file with an encrypted private key. Previously, we silently returned a nil Certificate object in this case which promptly resulted in a panic in crypto/tls when attempting to push. Instead, let's detect that the key is encrypted and prompt for a passphrase. Git usually handles this with a prompt from OpenSSL, which we aren't using, although it can be configured to use the credential helper as well. Since there isn't a portable way to turn off the echo in order to prompt for a passphrase, even among Unix systems, let's use the credential helper route for this purpose by prompting for credentials using a cert: URL for the file holding the private key; this is the type of URL that Git uses with the credential helper for this purpose. In order to make things as intuitive as possible, tell the credential code to always include the path for cert: URLs (so we don't just prompt for "cert:///") and provide the user's current username in the username field so they don't get a useless username prompt. Provide as much helpful trace output as possible for debugging; note that credential filling success and failure already have trace logging enabled elsewhere. Note that we create our own credential helper for the client object to avoid having to pass it into the HTTP client context from the LFS API context; this should be fine, since we're going to prompt and use this value only within this context and for this purpose. Finally, since we're in a context where we can't really return an error up the chain, if for whatever reason an error occurs, ensure that we don't pass nil to crypto/tls and instead skip passing a certificate altogether. This will at least make the failure case obvious later on and provide a better user experience than a panic.
195b353 to
706beca
Compare
Due to the issue described in git-lfs#5658, the "clone ClientCert" test in our t/t-clone.sh test script does not actually run to completion, because it exits early due to an improper check of the TRAVIS variable (which is, itself, no longer used since we migrated our test suite to GitHub Actions in PR git-lfs#3808). We expect to address this issue in subsequent commit in this PR, at which point the test will run fully, exposing a number of long-standing problems with the test. One of these occurs on Windows when the test checks the use of a TLS/SSL client certificate with an encrypted client key file. This check was added in commit 706beca of PR git-lfs#3270 in 2018, but may never have been fully validated since then. Our Windows CI jobs run in the Git Bash environment provided by the Git for Windows project, which in turn is based on the MSYS2 environment. In this context, when we set the "http.<url>.sslCert" Git configuration value in our setup() shell function in t/testhelpers.sh, the command-line argument with the path to the certificate file is converted by the MSYS2 environment into a Windows-style path with a leading volume identifier. When Git later makes a request to our credential helper test program to retrieve the passphrase for the encrypted key, it passes this path in the input "path" field. Our helper program then converts this path into a filename by replacing slash characters with dashes, and looks for a matching credential record file. However, we have created this file in the setup_creds() shell function using the Unix-style path in the LFS_CLIENT_KEY_FILE_ENCRYPTED environment variable we populate in our t/testenv.sh script. As a result, our credential helper program is unable to find the record file with the passphrase for the encrypted key, and Git then attempts to prompt the user for it, which also fails because our CI jobs run in a context without a terminal prompt, and the "clone ClientCert" test fails as a consequence. In theory we might work around this problem by setting the MSYS2_ARG_CONV_EXCL environment variable with a "*" value before calling "git config" to set the "http.<url>.sslCert" configuration value, as this would prevent MSYS2 from rewriting command-line arguments which look like Unix-style paths. Unfortunately, this introduces other failures into the Windows test suite. Instead, we work around the problem by performing a simple Windows-to-Unix path translation in our credential helper test program. As we note in a code comment, a more comprehensive solution might invoke the "cygpath" utility to perform the path conversion. But as we only need to perform this translation step in a single test in our Windows CI jobs, for the deprecated "git lfs clone" command, for now we just look for paths which start with an uppercase volume identifier such as "D:" and replace them with a lowercase version which starts with a leading slash, like "/d". This is sufficient to resolve the problem in our current Windows CI jobs on GitHub Actions runners.
Due to the issue described in git-lfs#5658, the "clone ClientCert" test in our t/t-clone.sh test script does not actually run to completion, because it exits early due to an improper check of the TRAVIS variable (which is, itself, no longer used since we migrated our test suite to GitHub Actions in PR git-lfs#3808). We expect to address this issue in subsequent commit in this PR, at which point the test will run fully, exposing a number of long-standing problems with the test. One of these occurs on Windows when the test checks the use of a TLS/SSL client certificate with an encrypted client key file. This check was added in commit 706beca of PR git-lfs#3270 in 2018, but may never have been fully validated since then. Our Windows CI jobs run in the Git Bash environment provided by the Git for Windows project, which in turn is based on the MSYS2 environment. In this context, when we set the "http.<url>.sslCert" Git configuration value in our setup() shell function in t/testhelpers.sh, the command-line argument with the path to the certificate file is converted by the MSYS2 environment into a Windows-style path with a leading volume identifier. When Git later makes a request to our credential helper test program to retrieve the passphrase for the encrypted key, it passes this path in the input "path" field. Our helper program then converts this path into a filename by replacing slash characters with dashes, and looks for a matching credential record file. However, we have created this file in the setup_creds() shell function using the Unix-style path in the LFS_CLIENT_KEY_FILE_ENCRYPTED environment variable we populate in our t/testenv.sh script. As a result, our credential helper program is unable to find the record file with the passphrase for the encrypted key, and Git then attempts to prompt the user for it, which also fails because our CI jobs run in a context without a terminal prompt, and the "clone ClientCert" test fails as a consequence. In theory we might work around this problem by setting the MSYS2_ARG_CONV_EXCL environment variable with a "*" value before calling "git config" to set the "http.<url>.sslCert" configuration value, as this would prevent MSYS2 from rewriting command-line arguments which look like Unix-style paths. Unfortunately, this introduces other failures into the Windows test suite. Instead, we work around the problem by performing a simple Windows-to-Unix path translation in our credential helper test program. As we note in a code comment, a more comprehensive solution might invoke the "cygpath" utility to perform the path conversion. But as we only need to perform this translation step in a single test in our Windows CI jobs, for the deprecated "git lfs clone" command, for now we just look for paths which start with an uppercase volume identifier such as "D:" and replace them with a lowercase version which starts with a leading slash, like "/d". This is sufficient to resolve the problem in our current Windows CI jobs on GitHub Actions runners.
The "cloneSSL" test in what is now our t/t-clone.sh test script was added in commits 4c64e82 and 8f91a1b of PR git-lfs#1067, and since then has performed the same basic set of checks. A bare test repository is first cloned using a TLS/SSL connection. Example Git LFS objects are then committed to the repository, and pushed, after which a "git lfs clone" command is run with a specified, non-default directory name argument so the repository is cloned into a new working tree under that local directory. The test then checks that the Git LFS objects have also been populated into the new working tree. Finally, a regular "git clone" command is run (without a directory name argument), again with a TLS/SSL connection, which should run the Git LFS "smudge" filter and fail if that does not also succeed. This test design was later used as a template for the "clone ClientCert" test when it was added in commit daba49a of PR git-lfs#1893; like the "cloneSSL" test, it performed the same series of checks, just using a client TLS certificate and key when cloning from the remote. We then added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and at this time, the "clone ClientCert" was updated to repeat the "git lfs clone" step and checks using first an unencrypted key and then an encrypted key. However, the "git clone" step was not moved into the same loop with the "git lfs clone" command, so it is only run using the unencrypted key. For this reason, the "http.<url>.sslKey" Git configuration value is reset after the end of the loop, so it contains just the unencrypted key's path. As we expect to make several other adjustments to this test in subsequent commits in this PR, we first bring the "git clone" command into the same loop with the "git lfs clone" command, which means we no longer need to reset the "http.<url>.sslKey" configuration value, as we want to use the same value for both cloning commands. This change also ensures we perform all the same checks with both types of TLS certificate keys.
We added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and as part of that change, updated the "clone ClientCert" test in our t/t-clone.sh test script to perform "git lfs clone" operations twice, once with an unencrypted TLS certificate key and once with an encrypted one. Because our test needs to ensure that Git does not defer to OpenSSL to prompt the user for the passphrase to the encrypted key, but first checks if the configured credential helper can provide the passphrase, we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable to "1". However, we do this for both passes through the loop in our test, so we set that environment variable when testing with the unencrypted TLS key as well as with the encrypted one. We only need to set it in the latter case, though. Moreover, setting the "http.sslCertPasswordProtected" Git configuration value to "true" has the same effect as setting the environment variable to "1". To clarify the intent of the test's loops, we refactor the test now to iterate over two pairs of values, with each pair containing the TLS certificate private key file path to be used, and a "false" or "true" value to be used for the "http.sslCertPasswordProtected" Git configuration value. This means that in the case of the unencrypted key, we now avoid setting an environment variable or configuration options which does not apply to that case. We also take the opportunity to fix a small typo in the directory name argument passed to the "git lfs clone" command, and we avoid the need to delete the working tree created by the previous iteration of the loop by including our loop index value in the directory name.
We added support for encrypted TLS certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS client certificate it generates upon starting. We advise the program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
Due to the issue described in git-lfs#5658, the "clone ClientCert" test in our t/t-clone.sh test script does not actually run to completion, because it exits early due to an improper check of the TRAVIS variable (which is, itself, no longer used since we migrated our test suite to GitHub Actions in PR git-lfs#3808). We expect to address this issue in subsequent commit in this PR, at which point the test will run fully, exposing a number of long-standing problems with the test. One of these occurs on Windows when the test checks the use of a TLS/SSL client certificate with an encrypted client key file. This check was added in commit 706beca of PR git-lfs#3270 in 2018, but may never have been fully validated since then. Our Windows CI jobs run in the Git Bash environment provided by the Git for Windows project, which in turn is based on the MSYS2 environment. In this context, when we set the "http.<url>.sslCert" Git configuration value in our setup() shell function in t/testhelpers.sh, the command-line argument with the path to the certificate file is converted by the MSYS2 environment into a Windows-style path with a leading volume identifier. When Git later makes a request to our credential helper test program to retrieve the passphrase for the encrypted key, it passes this path in the input "path" field. Our helper program then converts this path into a filename by replacing slash characters with dashes, and looks for a matching credential record file. However, we have created this file in the setup_creds() shell function using the Unix-style path in the LFS_CLIENT_KEY_FILE_ENCRYPTED environment variable we populate in our t/testenv.sh script. As a result, our credential helper program is unable to find the record file with the passphrase for the encrypted key, and Git then attempts to prompt the user for it, which also fails because our CI jobs run in a context without a terminal prompt, and the "clone ClientCert" test fails as a consequence. In theory we might work around this problem by setting the MSYS2_ARG_CONV_EXCL environment variable with a "*" value before calling "git config" to set the "http.<url>.sslCert" configuration value, as this would prevent MSYS2 from rewriting command-line arguments which look like Unix-style paths. Unfortunately, this introduces other failures into the Windows test suite. Instead, we work around the problem by performing a simple Windows-to-Unix path translation in our credential helper test program. As we note in a code comment, a more comprehensive solution might invoke the "cygpath" utility to perform the path conversion. But as we only need to perform this translation step in a single test in our Windows CI jobs, for the deprecated "git lfs clone" command, for now we just look for paths which start with an uppercase volume identifier such as "D:" and replace them with a lowercase version which starts with a leading slash, like "/d". This is sufficient to resolve the problem in our current Windows CI jobs on GitHub Actions runners.
The "cloneSSL" test in what is now our t/t-clone.sh test script was added in commits 4c64e82 and 8f91a1b of PR git-lfs#1067, and since then has performed the same basic set of checks. A bare test repository is first cloned using a TLS/SSL connection. Example Git LFS objects are then committed to the repository, and pushed, after which a "git lfs clone" command is run with a specified, non-default directory name argument so the repository is cloned into a new working tree under that local directory. The test then checks that the Git LFS objects have also been populated into the new working tree. Finally, a regular "git clone" command is run (without a directory name argument), again with a TLS/SSL connection, which should run the Git LFS "smudge" filter and fail if that does not also succeed. This test design was later used as a template for the "clone ClientCert" test when it was added in commit daba49a of PR git-lfs#1893; like the "cloneSSL" test, it performed the same series of checks, just using a client TLS certificate and key when cloning from the remote. We then added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and at this time, the "clone ClientCert" was updated to repeat the "git lfs clone" step and checks using first an unencrypted key and then an encrypted key. However, the "git clone" step was not moved into the same loop with the "git lfs clone" command, so it is only run using the unencrypted key. For this reason, the "http.<url>.sslKey" Git configuration value is reset after the end of the loop, so it contains just the unencrypted key's path. As we expect to make several other adjustments to this test in subsequent commits in this PR, we first bring the "git clone" command into the same loop with the "git lfs clone" command, which means we no longer need to reset the "http.<url>.sslKey" configuration value, as we want to use the same value for both cloning commands. This change also ensures we perform all the same checks with both types of TLS certificate keys.
We added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and as part of that change, updated the "clone ClientCert" test in our t/t-clone.sh test script to perform "git lfs clone" operations twice, once with an unencrypted TLS certificate key and once with an encrypted one. Because our test needs to ensure that Git does not defer to OpenSSL to prompt the user for the passphrase to the encrypted key, but first checks if the configured credential helper can provide the passphrase, we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable to "1". However, we do this for both passes through the loop in our test, so we set that environment variable when testing with the unencrypted TLS key as well as with the encrypted one. We only need to set it in the latter case, though. Moreover, setting the "http.sslCertPasswordProtected" Git configuration value to "true" has the same effect as setting the environment variable to "1". To clarify the intent of the test's loops, we refactor the test now to iterate over two pairs of values, with each pair containing the TLS certificate private key file path to be used, and a "false" or "true" value to be used for the "http.sslCertPasswordProtected" Git configuration value. This means that in the case of the unencrypted key, we now avoid setting an environment variable or configuration options which does not apply to that case. We also take the opportunity to fix a small typo in the directory name argument passed to the "git lfs clone" command, and we avoid the need to delete the working tree created by the previous iteration of the loop by including our loop index value in the directory name.
We added support for encrypted TLS certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS client certificate it generates upon starting. We advise the program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
Due to the issue described in git-lfs#5658, the "clone ClientCert" test in our t/t-clone.sh test script does not actually run to completion, because it exits early due to an improper check of the TRAVIS variable (which is, itself, no longer used since we migrated our test suite to GitHub Actions in PR git-lfs#3808). We will address this issue in a subsequent commit in this PR, at which point the test will run fully, exposing a number of long-standing problems with the test. One of these occurs on Windows when the test checks the use of a TLS/SSL client certificate with an encrypted client key file. This check was added in commit 706beca of PR git-lfs#3270 in 2018, but may never have been fully validated since then. Our Windows CI jobs run in the Git Bash environment provided by the Git for Windows project, which in turn is based on the MSYS2 environment. In this context, when we set the "http.<url>.sslCert" Git configuration value in our setup() shell function in t/testhelpers.sh, the command-line argument with the path to the certificate file is converted by the MSYS2 environment into a Windows-style path with a leading volume identifier. When Git later makes a request to our credential helper test program to retrieve the passphrase for the encrypted key, it passes this path in the input "path" field. Our helper program then converts this path into a filename by replacing slash characters with dashes, and looks for a matching credential record file. However, we have created this file in the setup_creds() shell function using the Unix-style path in the LFS_CLIENT_KEY_FILE_ENCRYPTED environment variable we populate in our t/testenv.sh script. As a result, our credential helper program is unable to find the record file with the passphrase for the encrypted key, and Git then attempts to prompt the user for it, which also fails because our CI jobs run in a context without a terminal prompt, and the "clone ClientCert" test fails as a consequence. In theory we might work around this problem by setting the MSYS2_ARG_CONV_EXCL environment variable with a "*" value before calling "git config" to set the "http.<url>.sslCert" configuration value, as this would prevent MSYS2 from rewriting command-line arguments which look like Unix-style paths. Unfortunately, this introduces other failures into our test suite on Windows. Instead, we work around the problem by performing a simple Windows-to-Unix path translation in our credential helper test program. As we note in a code comment, a more comprehensive solution might invoke the "cygpath" utility to perform the path conversion. But as we only need to perform this translation step in a single test in our Windows CI jobs, for the deprecated "git lfs clone" command, for now we just look for paths which start with an uppercase volume identifier such as "D:" and replace them with a lowercase version which starts with a leading slash, like "/d". This is sufficient to resolve the problem in our current Windows CI jobs on GitHub Actions runners.
The "cloneSSL" test in what is now our t/t-clone.sh test script was added in commits 4c64e82 and 8f91a1b of PR git-lfs#1067, and since then has performed the same basic set of checks. A bare test repository is first cloned using a TLS/SSL connection. Example Git LFS objects are then committed to the repository, and pushed, after which a "git lfs clone" command is run with a specified, non-default directory name argument so the repository is cloned into a new working tree under that local directory. The test then checks that the Git LFS objects have also been populated into the new working tree. Finally, a regular "git clone" command is run (without a directory name argument), again with a TLS/SSL connection, which should run the Git LFS "smudge" filter and fail if that does not also succeed. This test design was later used as a template for the "clone ClientCert" test when it was added in commit daba49a of PR git-lfs#1893; like the "cloneSSL" test, it performed the same series of checks, just using a client TLS certificate and key when cloning from the remote. We then added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and at this time, the "clone ClientCert" was updated to repeat the "git lfs clone" step and checks using first an unencrypted key and then an encrypted key. However, the "git clone" step was not moved into the same loop with the "git lfs clone" command, so it is only run using the unencrypted key. For this reason, the "http.<url>.sslKey" Git configuration value is reset after the end of the loop, so it contains just the unencrypted key's path. As we expect to make several other adjustments to this test in subsequent commits in this PR, we first bring the "git clone" command into the same loop with the "git lfs clone" command, which means we no longer need to reset the "http.<url>.sslKey" configuration value, as we want to use the same value for both cloning commands. This change also ensures we perform all the same checks with both types of TLS certificate keys.
We added support for encrypted TLS certificate keys in commit 706beca of PR git-lfs#3270, and as part of that change, updated the "clone ClientCert" test in our t/t-clone.sh test script to perform "git lfs clone" operations twice, once with an unencrypted TLS certificate key and once with an encrypted one. Because our test needs to ensure that Git does not defer to OpenSSL to prompt the user for the passphrase to the encrypted key, but first checks if the configured credential helper can provide the passphrase, we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable to "1". However, we do this for both passes through the loop in our test, so we set that environment variable when testing with the unencrypted TLS key as well as with the encrypted one. We only need to set it in the latter case, though. Moreover, setting the "http.sslCertPasswordProtected" Git configuration value to "true" has the same effect as setting the environment variable to "1". To clarify the intent of the test's loops, we refactor the test now to iterate over two pairs of values, with each pair containing the TLS certificate private key file path to be used, and a "false" or "true" value to be used for the "http.sslCertPasswordProtected" Git configuration value. This means that in the case of the unencrypted key, we now avoid setting an environment variable or configuration options which does not apply to that case. We also take the opportunity to fix a small typo in the directory name argument passed to the "git lfs clone" command, and we avoid the need to delete the working tree created by the previous iteration of the loop by including our loop index value in the directory name.
We added support for encrypted TLS certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS client certificate it generates upon starting. We advise the program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
In commit b9602f3 of PR git-lfs#5616 we resolved a problem which caused flaky test results in our CI jobs by ensuring that the t/t-credential.sh test script used a unique directory for the credential record files we provide to our git-credential-lfstest helper program. This change ensured that when the "clone (HTTP server/proxy require cookies)" test in our t/t-clone.sh script creates a record file named "localhost" in the common directory specified by the CREDSDIR environment variable, specifically a copy of the "127.0.0.1" file created by the setup_creds() shell library function, this does not conflict with the "credentials from netrc" test and other related tests in the t/t-credential.sh test script, which depend on there being no credential records associated with the "localhost" hostname. However, we already made a previous attempt to resolve this problem in commits bb87944 and bcb9eb3 of PR git-lfs#3825, but that change assumes that our test scripts always run sequentially rather than in parallel. In PR git-lfs#3825 the "clone (HTTP server/proxy require cookies)" test was added, and at the end of the test, it removes the "localhost" credential record file it creates at the beginning of the test. Of course, if the test ever fails, then it leaves the "localhost" record file in place, so even if our test scripts always ran sequentially there would still be a potential conflict with the tests in the t/t-credential.sh script. We can more comprehensively ensure that these two test scripts do not conflict again in the future by using the same technique applied in commit b9602f3 to the t/t-credential.sh script in the t/t-clone.sh script as well. First, we add a call to setup_creds() at the start of the t/t-clone.sh script after setting the CREDSDIR environment variable to a path unique to the t/t-clone.sh script. This ensures that a separate copy of the default credential record file for the 127.0.0.1 hostname is available for the exclusive use of the tests in the script. Next, we move the creation of the credential record files associated with the TLS/SSL client certificate used in the "clone ClientCert" test from the generic setup_creds() function into the test itself. We would otherwise need to define the "certpath" and "keypath" variables, as their values are interpolated by the function into the names of these credential record files. The TLS/SSL client certificate and key files are generated by our lfstest-gitserver utility when it is first executed. Their locations are defined by the LFSTEST_CLIENT_* environment variables passed by the setup() function in our t/testhelpers.sh shell library to the lfstest-count-tests utility, which then runs the lfstest-gitserver program. The values of these environment variables are set from the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE* variables defined by our t/testenv.sh script. We now use these variables in the "clone ClientCert" test when we call the write_creds_file() function directly, instead of allowing the setup_creds() function to perform those calls. We could define the "certpath" and "keypath" variables (using the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE_ENCRYPTED variables) prior to calling setup_creds() at the start of the t/t-clone.sh script. However, the "clone ClientCert" test is the only one which actually makes use of these credential record files, as this is the only test which actively checks the use of an encrypted TLS/SSL client certificate, and so is the only place where we need to create these record files. Further, we can also move into the test the "git config" commands that set the "http.<url>.sslCert" and "http.<url>.sslKey" Git configuration options with the locations of the TLS/SSL client certificate files. Previously, we set these configuration options in the setup() function in our t/testhelpers.sh shell library, so they were configured for all tests in all test scripts, although only the "clone ClientCert" test makes use of them. Finally, the "clone (HTTP server/proxy require cookies)" test no longer needs to attempt to delete the credential record file for the "localhost" hostname after the test is complete, so we can simply remove that code. Note that the "clone ClientCert" test was first introduced in commit daba49a of PR git-lfs#1893, at which time the "git config" commands to set the "http.<url>.sslCert" and "http.<url>.sslKey" options were added to the setup() function, and the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE variables were defined in what was then the test/testenv.sh script. Later, in commit 52f94c2 of PR git-lfs#3270, the LFS_CLIENT_KEY_FILE_ENCRYPTED variable was added along with support in the lfstest-gitserver program to generate an encrypted certificate key file. In another commit in PR git-lfs#3270, commit 706beca, the "clone ClientCert" test was then expanded to validate use of the encrypted key file with the "git lfs clone" command. (Note too that the following test, the "clone ClientCert with homedir certs" test, appears to also depend on credential record files for the TLS/SSL client certificate files it creates in the dedicated home directory used by our tests. However, because this test does not set the "http.sslCertPasswordProtected" Git configuration option or the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable, Git does not attempt to retrieve a passphrase for the certificate files, and so the associated credential record files are never actually used. We will address this issue in a subsequent commit in this PR.)
The "cloneSSL" test in what is now our t/t-clone.sh test script was added in commits 4c64e82 and 8f91a1b of PR git-lfs#1067, and since then has performed the same basic set of checks. A bare test repository is first cloned using a TLS/SSL connection. Example Git LFS objects are then committed to the repository, and pushed, after which a "git lfs clone" command is run with a specified, non-default directory name argument so the repository is cloned into a new working tree under that local directory. The test then checks that the Git LFS objects have also been populated into the new working tree. Finally, a regular "git clone" command is run (without a directory name argument), again with a TLS/SSL connection, which should run the Git LFS "smudge" filter and fail if that does not also succeed. This test design was later used as a template for the "clone ClientCert" test when it was added in commit daba49a of PR git-lfs#1893; like the "cloneSSL" test, it performed the same series of checks, just using a client TLS/SSL certificate and key when cloning from the remote. We then added support for encrypted TLS/SSL certificate keys in commit 706beca of PR git-lfs#3270, and at this time, the "clone ClientCert" was updated to repeat the "git lfs clone" step and checks using first an unencrypted key and then an encrypted key. However, the "git clone" step was not moved into the same loop with the "git lfs clone" command, so it is only run using the unencrypted key. For this reason, the "http.<url>.sslKey" Git configuration option is reset after the end of the loop to contain the unencrypted key file's path. As we expect to make several other adjustments to this test in subsequent commits in this PR, we first bring the "git clone" command into the same loop with the "git lfs clone" command, which means we no longer need to reset the "http.<url>.sslKey" configuration option, as we want to use the same value for both cloning commands. This change also ensures we perform all the same checks with both types of TLS/SSL certificate keys.
We added support for encrypted TLS/SSL certificate keys in PR git-lfs#3270, and as part of that change, in commit 706beca we updated the "clone ClientCert" test in our t/t-clone.sh test script to perform "git lfs clone" operations twice, once with an unencrypted TLS/SSL certificate key and once with an encrypted one. Our test needs to ensure that Git and Git LFS do not generate user prompts to retrieve the passphrase to the encrypted key, as this would make impossible to run the test in our CI jobs, and so we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable to "1" at the start of the main loop in our test. However, we do this for both passes through the loop, which means we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable when testing with the unencrypted TLS/SSL certificate key as well as with the encrypted one. We only need to set it in the latter case, though. To clarify the intent of the test's loops, we refactor the test to now iterate through a list of two values, "false" and "true". When the value is "true", we set the "http.sslCertPasswordProtected" Git configuration option to "true", and reset the "http.<url>.sslKey" option with the path from the LFS_CLIENT_KEY_FILE_ENCRYPTED variable. By refactoring the test's main loop in this way we can avoid setting Git environment variables or configuration options which do not apply when neither the certificate file nor the private key file are encrypted. In a prior commit this PR we moved the "git config" commands which set the "http.<url>.sslCert" and "http.<url>.sslKey" configuration options specific to the "clone ClientCert" test out of the common setup() function in our t/testhelpers.sh test library and into the test itself. To align with those initial "git config" commands and with the command we use on the second pass through the test's loop to reset the value of the "http.<url>.sslKey" option, we prefer to set the "http.sslCertPasswordProtected" Git configuration option instead of the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable. This option has the same effect on Git's behaviour as the environment variable, so there is no change to the test conditions. We also take the opportunity to fix a small typo in the directory name argument passed to the "git lfs clone" command, and we avoid the need to delete the working tree created by the previous iteration of the loop by including our loop index value in the directory name. Finally, we add a detailed set of code comments to explain exactly why we have to create two credential record files for use by our git-credential-lfstest helper program, associated with the paths of both the TLS/SSL certificate file and the private key file, when only the latter is encrypted. This is due to the fact that Git and Git LFS each make calls to the credential helper to retrieve the private key's passphrase, but do so for different reasons, and use different "path" values in the input they send to the "git credential fill" command.
We added support for encrypted TLS/SSL certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS/SSL client certificate it generates. We advise the lfstest-gitserver program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
In commit b9602f3 of PR git-lfs#5616 we resolved a problem which caused flaky test results in our CI jobs by ensuring that the t/t-credential.sh test script used a unique directory for the credential record files we provide to our git-credential-lfstest helper program. This change ensured that when the "clone (HTTP server/proxy require cookies)" test in our t/t-clone.sh script creates a record file named "localhost" in the common directory specified by the CREDSDIR environment variable, specifically a copy of the "127.0.0.1" file created by the setup_creds() shell library function, this does not conflict with the "credentials from netrc" test and other related tests in the t/t-credential.sh test script, which depend on there being no credential records associated with the "localhost" hostname. However, we already made a previous attempt to resolve this problem in commits bb87944 and bcb9eb3 of PR git-lfs#3825, but that change assumes that our test scripts always run sequentially rather than in parallel. In PR git-lfs#3825 the "clone (HTTP server/proxy require cookies)" test was added, and at the end of the test, it removes the "localhost" credential record file it creates at the beginning of the test. Of course, if the test ever fails, then it leaves the "localhost" record file in place, so even if our test scripts always ran sequentially there would still be a potential conflict with the tests in the t/t-credential.sh script. We can more comprehensively ensure that these two test scripts do not conflict again in the future by using the same technique applied in commit b9602f3 to the t/t-credential.sh script in the t/t-clone.sh script as well. First, we add a call to setup_creds() at the start of the t/t-clone.sh script after setting the CREDSDIR environment variable to a path unique to the t/t-clone.sh script. This ensures that a separate copy of the default credential record file for the 127.0.0.1 hostname is available for the exclusive use of the tests in the script. Next, we move the creation of the credential record files associated with the TLS/SSL client certificate used in the "clone ClientCert" test from the generic setup_creds() function into the test itself. We would otherwise need to define the "certpath" and "keypath" variables, as their values are interpolated by the function into the names of these credential record files. The TLS/SSL client certificate and key files are generated by our lfstest-gitserver utility when it is first executed. Their locations are defined by the LFSTEST_CLIENT_* environment variables passed by the setup() function in our t/testhelpers.sh shell library to the lfstest-count-tests utility, which then runs the lfstest-gitserver program. The values of these environment variables are set from the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE* variables defined by our t/testenv.sh script. We now use these variables in the "clone ClientCert" test when we call the write_creds_file() function directly, instead of allowing the setup_creds() function to perform those calls. We could define the "certpath" and "keypath" variables (using the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE_ENCRYPTED variables) prior to calling setup_creds() at the start of the t/t-clone.sh script. However, the "clone ClientCert" test is the only one which actually makes use of these credential record files, as this is the only test which actively checks the use of an encrypted TLS/SSL client certificate, and so is the only place where we need to create these record files. Further, we can also move into the test the "git config" commands that set the "http.<url>.sslCert" and "http.<url>.sslKey" Git configuration options with the locations of the TLS/SSL client certificate files, and we do the same for the "create lock with server using client cert" test in the t/t-lock.sh test script. Previously, we set these configuration options in the setup() function in our t/testhelpers.sh shell library, so they were configured for all tests in all test scripts, although only the "clone ClientCert" test and the "create lock with server using client cert" test make use of them. Finally, the "clone (HTTP server/proxy require cookies)" test no longer needs to attempt to delete the credential record file for the "localhost" hostname after the test is complete, so we can simply remove that code. Note that the "clone ClientCert" test was first introduced in commit daba49a of PR git-lfs#1893, at which time the "git config" commands to set the "http.<url>.sslCert" and "http.<url>.sslKey" options were added to the setup() function, and the LFS_CLIENT_CERT_FILE and LFS_CLIENT_KEY_FILE variables were defined in what was then the test/testenv.sh script. Later, in commit 52f94c2 of PR git-lfs#3270, the LFS_CLIENT_KEY_FILE_ENCRYPTED variable was added along with support in the lfstest-gitserver program to generate an encrypted certificate key file. In another commit in PR git-lfs#3270, commit 706beca, the "clone ClientCert" test was then expanded to validate use of the encrypted key file with the "git lfs clone" command. (Note too that the following test, the "clone ClientCert with homedir certs" test, appears to also depend on credential record files for the TLS/SSL client certificate files it creates in the dedicated home directory used by our tests. However, because this test does not set the "http.sslCertPasswordProtected" Git configuration option or the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable, Git does not attempt to retrieve a passphrase for the certificate files, and so the associated credential record files are never actually used. We will address this issue in a subsequent commit in this PR.)
The "cloneSSL" test in what is now our t/t-clone.sh test script was added in commits 4c64e82 and 8f91a1b of PR git-lfs#1067, and since then has performed the same basic set of checks. A bare test repository is first cloned using a TLS/SSL connection. Example Git LFS objects are then committed to the repository, and pushed, after which a "git lfs clone" command is run with a specified, non-default directory name argument so the repository is cloned into a new working tree under that local directory. The test then checks that the Git LFS objects have also been populated into the new working tree. Finally, a regular "git clone" command is run (without a directory name argument), again with a TLS/SSL connection, which should run the Git LFS "smudge" filter and fail if that does not also succeed. This test design was later used as a template for the "clone ClientCert" test when it was added in commit daba49a of PR git-lfs#1893; like the "cloneSSL" test, it performed the same series of checks, just using a client TLS/SSL certificate and key when cloning from the remote. We then added support for encrypted TLS/SSL certificate keys in commit 706beca of PR git-lfs#3270, and at this time, the "clone ClientCert" was updated to repeat the "git lfs clone" step and checks using first an unencrypted key and then an encrypted key. However, the "git clone" step was not moved into the same loop with the "git lfs clone" command, so it is only run using the unencrypted key. For this reason, the "http.<url>.sslKey" Git configuration option is reset after the end of the loop to contain the unencrypted key file's path. As we expect to make several other adjustments to this test in subsequent commits in this PR, we first bring the "git clone" command into the same loop with the "git lfs clone" command, which means we no longer need to reset the "http.<url>.sslKey" configuration option, as we want to use the same value for both cloning commands. This change also ensures we perform all the same checks with both types of TLS/SSL certificate keys.
We added support for encrypted TLS/SSL certificate keys in PR git-lfs#3270, and as part of that change, in commit 706beca we updated the "clone ClientCert" test in our t/t-clone.sh test script to perform "git lfs clone" operations twice, once with an unencrypted TLS/SSL certificate key and once with an encrypted one. Our test needs to ensure that Git and Git LFS do not generate user prompts to retrieve the passphrase to the encrypted key, as this would make impossible to run the test in our CI jobs, and so we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable to "1" at the start of the main loop in our test. However, we do this for both passes through the loop, which means we set the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable when testing with the unencrypted TLS/SSL certificate key as well as with the encrypted one. We only need to set it in the latter case, though. To clarify the intent of the test's loops, we refactor the test to now iterate through a list of two values, "false" and "true". When the value is "true", we set the "http.sslCertPasswordProtected" Git configuration option to "true", and reset the "http.<url>.sslKey" option with the path from the LFS_CLIENT_KEY_FILE_ENCRYPTED variable. By refactoring the test's main loop in this way we can avoid setting Git environment variables or configuration options which do not apply when neither the certificate file nor the private key file are encrypted. In a prior commit this PR we moved the "git config" commands which set the "http.<url>.sslCert" and "http.<url>.sslKey" configuration options specific to the "clone ClientCert" test out of the common setup() function in our t/testhelpers.sh test library and into the test itself. To align with those initial "git config" commands and with the command we use on the second pass through the test's loop to reset the value of the "http.<url>.sslKey" option, we prefer to set the "http.sslCertPasswordProtected" Git configuration option instead of the GIT_SSL_CERT_PASSWORD_PROTECTED environment variable. This option has the same effect on Git's behaviour as the environment variable, so there is no change to the test conditions. We also take the opportunity to fix a small typo in the directory name argument passed to the "git lfs clone" command, and we avoid the need to delete the working tree created by the previous iteration of the loop by including our loop index value in the directory name. Finally, we add a detailed set of code comments to explain exactly why we have to create two credential record files for use by our git-credential-lfstest helper program, associated with the paths of both the TLS/SSL certificate file and the private key file, when only the latter is encrypted. This is due to the fact that Git and Git LFS each make calls to the credential helper to retrieve the private key's passphrase, but do so for different reasons, and use different "path" values in the input they send to the "git credential fill" command.
We added support for encrypted TLS/SSL certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS/SSL client certificate it generates. We advise the lfstest-gitserver program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
We added support for encrypted TLS/SSL certificate keys in PR git-lfs#3270, and as part of that change, in commit 52f94c2, we revised our lfstest-gitserver test utility program to create an encrypted private key file of the TLS/SSL client certificate it generates. We advise the lfstest-gitserver program where to write the encrypted key file using an LFSTEST_CLIENT_KEY_ENCRYPTED environment variable, which we populate in turn from an LFS_CLIENT_KEY_FILE_ENCRYPTED variable defined in our t/testenv.sh shell script, both of which were added in the same commit. However, the comment which describes the LFS_CLIENT_KEY_FILE_ENCRYPTED variable is identical to the comment for the LFS_CLIENT_KEY_FILE variable, so we update the one for the LFS_CLIENT_KEY_FILE_ENCRYPTED variable to clarify its intent.
The Creds structure in our "creds" package was first introduced in commit e8de7a3, where it was part of a different package in an early pre-alpha version of the Git LFS client. A single method was defined for the structure in that commit, the Buffer() method, which formatted lines of key/value pairs to be sent to the git-credential(1) command. In commit 3a271b1 of PR #1839 this method was converted into the current bufferCreds() function, a change made to avoid exporting the method out of the "lfsapi" package, where the structure was defined at that time. We next moved the Creds structure (which was then a simple map of strings to strings) and its one function into the current "creds" package in commit c752dcd of PR #3270. Later, in commits 5d5f90e of PR #5381 and 6783078 of PR #5803 we revised the Creds structure so its map contains arrays of strings, and added the IsMultistage() method, so we again have a single method defined for this structure. In subsequent commits in this PR we will refactor the bufferCreds() function to accept and return additional values. As a first step, in order to keep our code as consistent as possible, we convert it back into a method, as it was originally defined, except with the lowercase name of buffer() so it is not exported outside of the "creds" package. This retains the intent of commit 3a271b1 from PR #1839, but uses a naming pattern that is more idiomatic for the Go language. As well, we add a test function named TestCredsBufferFormat() to the accompanying creds/creds_test.go source file. This test function simply verifies the behaviour of the former bufferCreds() function that we have now renamed to the buffer() method. To permit comparison of the contents of the method's output buffer with the lines of key/value pairs expected by the test function, we also add an assertCredsLinesMatch() helper function which splits and sorts the lines in the buffer and compares them with the sorted set of expected output lines.
Some users would like to use encrypted X.509 keys when using client certificate authentication. This series introduces code that uses the credential helper for this case. Upstream Git by default has OpenSSL (or another crypto library) prompt for this data on
/dev/tty, but also has support for the credential helper. Because we don't want to have to write unportable TTY-related prompting code to turn off the echo, we always use the credential helper.The first patch fixes a bug that caused the client cert integration test to not run completely and makes it so that it passes. The next two patches prepare the testsuite to handle encrypted private keys and local URLs.
The next several patches prepare the credential helper code to move into its own package (
creds), since we need to access it from both thelfsapiandlfshttppackages and can't have an import loop. @PastelMobileSuit intentionally separated this code out of thelfshttppackage and into thelfsapipackage. I regret having to reintroduce it into thelfshttppackage, but I think it's okay for the purpose for which we're doing it; if others think this is a bad idea, I'm open to hearing suggestions for alternatives. Credential caching is certainly a requirement in this code path.The final patch implements the actual encrypted TLS private key handling. It should be relatively boring. The changes to
t/t-clone.share best viewed withgit diff -w, since most of the change is to indent code.Because of the complexity of this change and the potential interaction with recent and inflight work, I'm requesting two reviewers.
This should fix #2602.