-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Use constant format strings and fix HTTP error messages #5998
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 28, 2025
In commit f972064 of PR git-lfs#5066 we introduced the "lfs.remote.searchAll" configuration option and updated the "smudge" Git LFS filter logic so that when the option is enabled, the filter searches all Git remotes for Git LFS objects that are not present locally rather than only attempting to fetch the objects from the default Git LFS remote URL. The downloadFileFallBack() method of the GitFilter structure in the "lfs" package was added to implement this logic. One of the error conditions it must handle may occur if no Git remotes are defined. In this case, we append (i.e., wrap) an initial error message with a second generic error message indicating which file and Git LFS object ID could not be downloaded. While we allow for the generic error message to be localized, we do not do so for the initial error message, so we adjust that now. We simply call the Get() method of the Tr structure in our "tr" package on the initial error message, as we already do for the generic error message.
In PR git-lfs#4781 we updated a number of packages to replace global string variables with local variables so they could be translated via the "tr" package's localization functions. One concern raised in that PR pertained to the changes to the lfshttp/errors.go source file, where a map of error format strings is constructed in the defaultError() function, with each string mapped to a specific HTTP status code. These strings should contain at least one "%s" verb, into which the HTTP request URL should be interpolated. In commit cda9c3c of PR git-lfs#4781 we moved the map's initialization into the defaultError() function of our "lfshttp" package, and added a call to the Get() method of the Tr structure from our "tr" package around each string's definition. However, we deferred the interpolation of the request URL to a subsequent call to a string formatting function, with the intent that we would first select the correct localized format string from the map based on the HTTP request's status code, and then interpolate the request's URL into the final error message. Further, if a request's status code does not correspond to a string in the map, our defaultError() function creates a generic error message into which we try to interpolate both the request's URL and its status code. For these generic messages, the intent of the changes in commit cda9c3c was that we first interpolate the status code, and then treat the resultant string as if it had been found in the defaultErrors map, meaning it should still contain one "%s" verb into which the request URL can be interpolated. Such a design implies that in the generic error messages we need to use a "%d" verb for the status code, and a doubly-escaped "%%s" verb for the request URL, since we want the latter to be converted to a regular "%s" verb when the string is first used as a format string. This design exposed an unexpected issue, which is described in leonelquinteros/gotext#57. In trying to work around this issue, though, we introduced a bug of our own whereby the generic messages are passed through three formatting functions rather than two, and in doing so the "%%s" verbs are eliminated before we attempt to interpolate the HTTP request's URL, leading to further errors. The single commit in PR git-lfs#5822 attempts to fix this problem, but without complete success. We now adopt the changes from that PR's commit and resolve the remaining problems so we should always supply an appropriate set of arguments for the number of formatting verbs in both types of messages in the defaultError() function. The confusion here stems from the issue reported in leonelquinteros/gotext#57, namely that the Printf() function of the "github.com/leonelquinteros/gotext" package behaves differently depending on whether or not it is passed any arguments other than a format string. When no arguments are passed, the format string is not parsed and any verbs are left unaltered, whereas when at least one additional argument is passed, the format string is parsed and the trailing arguments interpolated into it. The Tr global variable in our "tr" package is a Locale structure from the "gotext" package. Starting with commit cda9c3c, we separately invoke this variable's Get() method on each of the error messages defined in the defaultErrors map, but without any other arguments. Because the Get() method does not parse the format string when no other arguments are passed, in theory we ought to be able to define these messages with regular "%s" verbs. If we do so, though, the Go linter complains that it appears we are passing format strings to the Get() method without matching arguments for their "%s" verbs. To work around this linter limitation, we currently use "%%s" verbs in the error messages in the defaultErrors map, which are then passed through the Get() method unchanged. We then make a subsequent call to the Sprintf() function from the Go standard library's "fmt" package, passing the error message as the format string but without any other arguments. This call effectively just converts any "%%s" verbs to "%s" verbs. By contrast, the generic error messages in the defaultError() function are passed to the Tr variable's Get() method with another argument, the HTTP request's status code. Due to the presence of the extra argument, the Get() method parses the format string, replacing the "%d" verb with the status code and converting the "%%s" verb into a "%s" verb. (Note that originally, in PR git-lfs#4781, we used the Sprintf() function of the standard "fmt" package rather than the Get() method of our Tr variable. We corrected this oversight in commit 04abbd8 of PR git-lfs#4846, but while that change allows the strings to be localized, it has no effect on the handling of the formatting verbs.) As with the error messages from the defaultErrors map, the generic error messages are then also passed without any other arguments to the Sprintf() function from the standard "fmt" package. However, this call does not have the same effect in both cases. When the error message is specific to an HTTP status code and has been selected from the defaultErrors map, this call just converts the "%%s" verb into a "%s" one, as noted above. When the error message is a generic one, though, that conversion has already been performed by the Tr structure's Get() method. The consequence is that the Sprintf() function replaces the "%s" verb with "%!s(MISSING)", as described in the "fmt" package's documentation: https://pkg.go.dev/fmt#hdr-Format_errors In the final step of the defaultError() function we then pass the result from this Sprintf() call to the Errof() function of our "errors" package, and here we include the HTTP request's URL as a second argument. In the case where the error message has been selected from the defaultErrors map, this last formatting function replaces the "%s" verb with the URL. In the case where the error message is a generic one, though, the leading portion of the format error marker "%!s(MISSING)" cannot be parsed as a verb, and so the "%!" is replaced with a full reflection of all the internal elements of the URL structure into string form. Since these format error markers and reflection fields are not desirable in our HTTP request error messages, we revise the defaultError() function so it each type of error message is passed through only a single formatting function, specifically the Tr variable's Get() method. This allows us to use regular "%s" and "%d" formatting verbs exclusively in all the messages. Rather than call the Get() method on each string as it is defined in the defaultErrors map, we only call the method once a specific message is selected based on the HTTP request's status code, and we pass the request's URL to the Get() method as another argument. For the generic error messages, we simply pass both the status code and the URL as arguments to the Get() function. Because we always pass either one or two arguments to that function in addition to a format string, it always performs the appropriate interpolation for us, and we avoid the need to work around its behaviour when it is only passed a format string without other arguments, as well as avoiding the use of any further formatting functions like Sprintf(). Co-authored-by: Philip Peterson <[email protected]>
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. In the Log() method of the SimpleTask structure in our "tasklog" package, we pass the method's non-constant input string as the only argument to the Logf() method, which then uses it as a format string when creating a task log Update structure. To resolve this issue, we instead call the Logf() method with a format string containing a single "%s" verb, and pass the non-constant input string as the second argument.
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. When the "git lfs pointer" command compares two Git LFS pointer files, it outputs the contents of the file specified by the --pointer option (or supplied on standard input, if the --stdin option is used) to the standard error file descriptor. To write this data, we call the Fprintf() function of the Go standard library's "fmt" package, and pass the pointer file data as the format string, although it is not a constant string value. To resolve this issue, we simply need to use the Fprint() function instead, as we already do in a number of other instances in the same command where we write data to the standard error file descriptor.
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. Our Git LFS Batch API test utility, the git-lfs-test-server-api program, defines an exit() function which accepts a format string and a variable number of arguments, generates an output message from those, and then halts the program with a non-zero exit code. In one instance we call this exit() function with a non-constant format string we construct from a fixed string and an error message. To resolve this issue, we revise the fixed string to include a "%s" verb, and pass the error as a second argument to the exit() function.
When we report an error, we often call the Errorf() function in our custom "errors" package, which accepts both a format string and a variable number of additional arguments. The intent of this function is for the additional arguments to be interpolated into the format string, and then an error created using the string which results from the interpolation. In many instances we originally made use of the interpolation capability of the Errorf() function by passing at least one additional argument to it other than the format string. In other cases, though, we used the Errorf() function with just a single fixed string argument, so the the New() function of the same "errors" package would have sufficed instead. In PR git-lfs#4781 we then updated our code to allow for localization of most of our output messages by inserting calls to the Get() method of the Tr structure of our "tr" package. In the instances where we call the Errorf() function, the consequence is that we now pass the format string and any additional arguments to the Get() method, and then just pass the string it returns to the Errorf() function as a single argument. As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. This includes all the cases where we pass the output from the Get() method of the Tr structure to an Errorf() function as a single argument. To resolve this concern, we revise all such instances to use the New() function of our "errors" package in place of the Errorf() function.
When we report an error, we often call the Wrapf() function in our custom "errors" package, which accepts an existing error, a format string, and a variable number of additional arguments. The intent of this function is for the additional arguments to be interpolated into the format string, and then a new error created using the string which results from the interpolation, combined with the initial error and its message. In many instances we originally made use of the interpolation capability of the Wrapf() function by passing at least one additional argument to it other than the existing error and the format string. In other cases, though, we used the Wrapf() function with just the existing error and a fixed string argument, so the Wrap() function of the same "errors" package would have sufficed instead. In PR git-lfs#4781 updated our code to allow for localization of most of our output messages by inserting calls to the Get() method of the Tr structure of our "tr" package. In the instances where we call the Wrapf() function, the consequence is that we now pass the format string and any additional arguments to the Get() method, and then just pass the string it returns to the Wrapf() function as the format string argument following the initial existing error argument. As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. This includes all the cases where we pass the output from the Get() method of the Tr structure to an Wrapf() function as a format string without any additional arguments to be interpolated. To resolve this concern, we revise all such instances to use the Wrap() function of our "errors" package in place of the Wrapf() function.
As of Go 1.24, if the Go version number in the "go.mod" file is set to 1.24 or higher, the "go vet" command now reports misuses of non-constant strings as format strings. In previous commits in this PR we have now resolved all but one of the instances where we provided a non-constant string as a format string. The remaining instance is our use of the Errorf() function of the "fmt" package from the Go standard library at the end of the Combine() function in our "errors" package. The Combine() function was added to our custom "errors" package in commit 08e3e5b of PR git-lfs#1870, originally for use in the "locking" package. This function merges multiple errors into a single error by concatenating their error messages with a newline separator character between each original message. In Go 1.20 the Join() function was added to the "errors" package of the Go standard library, and it performs the same concatenation of error messages as our Combine() function, including the use of a newline character as a separator between the original messages, except that it delays the concatenation until the Error() method is called. To resolve the remaining case where we pass a non-constant string as a format string, we remove the Combine() function from our "errors" package and replace it with a Join() function that simply invokes the Join() function of the standard library's "errors" package, which we can expect to be defined as we currently require the use of at least Go 1.21 (per the Go version specified in our "go.mod" file). To make this change, we alias the standard library's "errors" package as "goerrors", following the pattern established in our "lfshttp" package where we use that alias as well. We then revise the two callers of our Combine() function to make use of the new Join() function instead. One of these callers is the startConnection() function in our "ssh" package, which simply joins two errors, and the other caller is the FixLockableFileWriteFlags() method of the Client structure in our "locking" package, which collects zero or more errors while iterating over a list of files and repeatedly calling another method. Because the Join() functions accept variadic arguments rather than an array of errors, we revise these callers to pass the errors they intend to concatenate as direct arguments rather than in an array. For the FixLockableFileWriteFlags() method this means that as it iterates over its list of files, if any errors occur we immediately call the Join() function to add them to the error value we return at the end of the function, whereas previously we appended the errors to an array, and then called the Combine() function after the loop exited to concatenate any errors in the array into the function's final error return value.
In a previous commit in this PR we replaced the Combine() function in our custom "errors" package with a Join() function that passes its arguments through to the Join() function of the Go standard library's "errors" package. Both these Join() functions and our earlier Combine() function serve to concatenate multiple error values into a single error value, with the individual error messages separated by newline characters when the error is converted to a string value for output. In a number of instances we perform the same concatenation of error messages using loops and assembling the final error message by explicitly appending each individual error message along with the newline separator characters. We can now simplify all these instances where we need to concatenate error messages by making use of the Join() function in our "errors" package. In almost all of these cases, this means we are able to eliminate the conditional logic used to check if a newline separator character is required because more than one error has been encountered. (The exception is in the Fill() method of the CredentialHelpers structure of our "creds" package where we previously concatenated the error messages using the Join() method of the "strings" package from the Go standard library.) Note that we could also have used the Combine() function from our "errors" package to simplify these use cases; however, the Join() function is more convenient in most instances because it accepts variadic arguments rather than requiring an array of errors as input.
In PR git-lfs#1466 we introduced our custom "errors" package as a replacement for the "errutil" package used previously to provide various error handling functions and specialized error types. As part of this PR, in commit 8624a87 we defined the StackTrace() function in the new "errors" package, and updated the logPanicToWriter() function in our "commands" package to call it when generating an error log file. We write these log files to the .git/lfs/logs directory whenever a Git LFS command invokes one of our error reporting utility functions such as LoggedError() or Panic(). Later, in commit 377366d of PR git-lfs#2178, we revised the logPanicToWriter() function to use the "%+v" format string specifier when writing an error's details to the log file, and dropped the call to the StackTrace() function of our "errors" package. The StackTrace() call was deemed no longer necessary because we expect all of our errors to be created using our "errors" package's functions, which instantiate error structures defined by the legacy "github.com/pkg/errors" package. This package defines its own Format() methods for its structures so that when the "%+v" format string specifier is used with them, they output both the message and stack trace associated with each error. As a consequence of this change in PR git-lfs#2178, the StackTrace() function in our own custom "errors" package has no remaining callers, so we can remove it now.
We recently updated our "netrc" Go package in PR git-lfs/go-netrc#3 with a small change to use a constant format string (as reported by Go 1.24's "go vet" command). We therefore now update the Go module list in this repository to reference the latest version of our git-lfs/go-netrc project, by running: go get github.com/git-lfs/go-netrc@latest We then also run the "go mod tidy" command to update the set of Go module hashes in the "go.sum" file.
58503d8 to
3642879
Compare
larsxschneider
approved these changes
Mar 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As described in #5968, when we upgrade to Go 1.24, and when we bump the Go version number in our
go.modfile to 1.24 or higher, thego vetcommand will identify instances where we pass a non-constant value as the format string parameter to a formatting function such asfmt.Sprintf(). This feature was implemented by the change in golang/go#60529 and is noted in the Go 1.24 release documentation.Some of the misuses of format functions in our code result from the introduction of our localization support in PR #4781, as in this instance, because as shown below we moved format strings and their accompanying arguments into our message localization function, but did not change the existing string formatting function into one which only accepts a fixed string.
In other instances, though, this problem predates PR #4781, such as in this example where we accidentally used a formatting function with an non-constant, dynamic value.
Before addressing these issues, we first ensure one additional message output by our
smudgefilter is fully localizable, as we currently allow translation for only a portion of the message.Next, we fully resolve the problem reported in PR #5822, where we return an error message with multiple format error markers of the form
&{...}s(MISSING)when we encounter an HTTP status code that does not have a specific error message defined in thedefaultError()function of ourlfshttppackage. This behaviour stems from the final steps in the function, where we callfmt.Sprintf()on a format string which contains a%sverb, but do not pass an argument to be interpolated into that verb. We call that function without arguments other than a format string in an attempt to work around the issue reported in leonelquinteros/gotext#57.While the changes proposed in PR #5822 fix this issue for the generic error messages we use when an HTTP status code is not one for which a specific message is defined, it introduces a similar problem for those code-specific messages. To completely resolve these issues, we refactor our
defaultError()function so we avoid the use of our message localization function until we have selected an error message. This obviates any need for doubly-escaped format string verbs like%%sor for the use of any formatting functions other than the localization function.With those initial corrections in place, in a series of commits we then revise all the instances where we pass a dynamic value as the format string to a formatting function. Each commit corrects the all misuses of a given formatting function, such as
fmt.Fprintf()or theErrorf()andWrapf()functions from our customerrorspackage.The last of these commits removes the
Combine()function from ourerrorspackage and replaces it with aJoin()function that simply invokes theJoin()function that was added to the standard library'serrorspackage in Go 1.20. This change has the advantage that it eliminates the call at the end of ourCombine()function where we pass a dynamic string value to ourErrorf()function. We then update the two locations where we call theCombine()function to use our newJoin()function instead.We next refactor a number of instances throughout our code where we collect multiple errors and combine them into a single error. In all these cases, we now use the new
Join()function in ourerrorspackage. Note that we could have used theCombine()function to do this in the past, but did not take advantage of that option, and this PR provides an opportunity to make these simplifications.To further streamline our
errorspackage, we also remove theStackTrace()function function, as it has not been used since PR #2178.Finally, we update our
go.modfile to use the latest version of the Go package from ourgit-lfs/go-netrcrepository, which includes the correction in PR git-lfs/go-netrc#3 of another use of a non-constant format string, and rungo mod tidyto update the correspondinggo.sumfile.This PR will be most easily reviewed on a commit-by-commit basis, as each commit in this PR represents a discrete and logically related set of changes accompanied by a detailed description.
This PR includes the work from PR #5822 and resolves the issue reported in that PR.
Resolves #5968.
/cc @philip-peterson as the author of PR #5822
/cc @QuLogic as the reporter of #5968