Skip to content

Conversation

@chrisd8088
Copy link
Member

@chrisd8088 chrisd8088 commented Feb 28, 2025

As described in #5968, when we upgrade to Go 1.24, and when we bump the Go version number in our go.mod file to 1.24 or higher, the go vet command will identify instances where we pass a non-constant value as the format string parameter to a formatting function such as fmt.Sprintf(). This feature was implemented by the change in golang/go#60529 and is noted in the Go 1.24 release documentation.

Some of the misuses of format functions in our code result from the introduction of our localization support in PR #4781, as in this instance, because as shown below we moved format strings and their accompanying arguments into our message localization function, but did not change the existing string formatting function into one which only accepts a fixed string.

-		return nil, errors.Errorf("error parsing path %s", path)
+		return nil, errors.Errorf(tr.Tr.Get("error parsing path %s", path))

In other instances, though, this problem predates PR #4781, such as in this example where we accidentally used a formatting function with an non-constant, dynamic value.

Before addressing these issues, we first ensure one additional message output by our smudge filter is fully localizable, as we currently allow translation for only a portion of the message.

Next, we fully resolve the problem reported in PR #5822, where we return an error message with multiple format error markers of the form &{...}s(MISSING) when we encounter an HTTP status code that does not have a specific error message defined in the defaultError() function of our lfshttp package. This behaviour stems from the final steps in the function, where we call fmt.Sprintf() on a format string which contains a %s verb, but do not pass an argument to be interpolated into that verb. We call that function without arguments other than a format string in an attempt to work around the issue reported in leonelquinteros/gotext#57.

While the changes proposed in PR #5822 fix this issue for the generic error messages we use when an HTTP status code is not one for which a specific message is defined, it introduces a similar problem for those code-specific messages. To completely resolve these issues, we refactor our defaultError() function so we avoid the use of our message localization function until we have selected an error message. This obviates any need for doubly-escaped format string verbs like %%s or for the use of any formatting functions other than the localization function.

With those initial corrections in place, in a series of commits we then revise all the instances where we pass a dynamic value as the format string to a formatting function. Each commit corrects the all misuses of a given formatting function, such as fmt.Fprintf() or the Errorf() and Wrapf() functions from our custom errors package.

The last of these commits removes the Combine() function from our errors package and replaces it with a Join() function that simply invokes the Join() function that was added to the standard library's errors package in Go 1.20. This change has the advantage that it eliminates the call at the end of our Combine() function where we pass a dynamic string value to our Errorf() function. We then update the two locations where we call the Combine() function to use our new Join() function instead.

We next refactor a number of instances throughout our code where we collect multiple errors and combine them into a single error. In all these cases, we now use the new Join() function in our errors package. Note that we could have used the Combine() function to do this in the past, but did not take advantage of that option, and this PR provides an opportunity to make these simplifications.

To further streamline our errors package, we also remove the StackTrace() function function, as it has not been used since PR #2178.

Finally, we update our go.mod file to use the latest version of the Go package from our git-lfs/go-netrc repository, which includes the correction in PR git-lfs/go-netrc#3 of another use of a non-constant format string, and run go mod tidy to update the corresponding go.sum file.

This PR will be most easily reviewed on a commit-by-commit basis, as each commit in this PR represents a discrete and logically related set of changes accompanied by a detailed description.


This PR includes the work from PR #5822 and resolves the issue reported in that PR.
Resolves #5968.

/cc @philip-peterson as the author of PR #5822
/cc @QuLogic as the reporter of #5968

chrisd8088 and others added 11 commits March 4, 2025 16:36
In commit f972064 of PR git-lfs#5066 we
introduced the "lfs.remote.searchAll" configuration option and updated
the "smudge" Git LFS filter logic so that when the option is enabled,
the filter searches all Git remotes for Git LFS objects that are not
present locally rather than only attempting to fetch the objects from
the default Git LFS remote URL.

The downloadFileFallBack() method of the GitFilter structure in the
"lfs" package was added to implement this logic.  One of the error
conditions it must handle may occur if no Git remotes are defined.
In this case, we append (i.e., wrap) an initial error message with
a second generic error message indicating which file and Git LFS
object ID could not be downloaded.

While we allow for the generic error message to be localized, we
do not do so for the initial error message, so we adjust that now.
We simply call the Get() method of the Tr structure in our "tr" package
on the initial error message, as we already do for the generic
error message.
In PR git-lfs#4781 we updated a number of packages to replace global string
variables with local variables so they could be translated via the
"tr" package's localization functions.

One concern raised in that PR pertained to the changes to the
lfshttp/errors.go source file, where a map of error format strings
is constructed in the defaultError() function, with each string
mapped to a specific HTTP status code.  These strings should contain
at least one "%s" verb, into which the HTTP request URL should be
interpolated.

In commit cda9c3c of PR git-lfs#4781 we moved
the map's initialization into the defaultError() function of our
"lfshttp" package, and added a call to the Get() method of the Tr
structure from our "tr" package around each string's definition.
However, we deferred the interpolation of the request URL to a
subsequent call to a string formatting function, with the intent that
we would first select the correct localized format string from the
map based on the HTTP request's status code, and then interpolate
the request's URL into the final error message.

Further, if a request's status code does not correspond to a string
in the map, our defaultError() function creates a generic error
message into which we try to interpolate both the request's URL and
its status code.  For these generic messages, the intent of the
changes in commit cda9c3c was that
we first interpolate the status code, and then treat the resultant
string as if it had been found in the defaultErrors map, meaning it
should still contain one "%s" verb into which the request URL can be
interpolated.

Such a design implies that in the generic error messages we need to
use a "%d" verb for the status code, and a doubly-escaped "%%s" verb
for the request URL, since we want the latter to be converted to a
regular "%s" verb when the string is first used as a format string.

This design exposed an unexpected issue, which is described in
leonelquinteros/gotext#57.  In trying to work around this issue, though,
we introduced a bug of our own whereby the generic messages are passed
through three formatting functions rather than two, and in doing so
the "%%s" verbs are eliminated before we attempt to interpolate the
HTTP request's URL, leading to further errors.

The single commit in PR git-lfs#5822 attempts to fix this problem, but without
complete success.  We now adopt the changes from that PR's commit and
resolve the remaining problems so we should always supply an appropriate
set of arguments for the number of formatting verbs in both types of
messages in the defaultError() function.

The confusion here stems from the issue reported in
leonelquinteros/gotext#57, namely that the Printf() function of the
"github.com/leonelquinteros/gotext" package behaves differently
depending on whether or not it is passed any arguments other than
a format string.  When no arguments are passed, the format string
is not parsed and any verbs are left unaltered, whereas when
at least one additional argument is passed, the format string is
parsed and the trailing arguments interpolated into it.

The Tr global variable in our "tr" package is a Locale structure
from the "gotext" package.  Starting with commit
cda9c3c, we separately invoke this
variable's Get() method on each of the error messages defined in
the defaultErrors map, but without any other arguments.  Because
the Get() method does not parse the format string when no other
arguments are passed, in theory we ought to be able to define these
messages with regular "%s" verbs.  If we do so, though, the Go linter
complains that it appears we are passing format strings to the Get()
method without matching arguments for their "%s" verbs.

To work around this linter limitation, we currently use "%%s" verbs in
the error messages in the defaultErrors map, which are then passed
through the Get() method unchanged.  We then make a subsequent call
to the Sprintf() function from the Go standard library's "fmt" package,
passing the error message as the format string but without any other
arguments.  This call effectively just converts any "%%s" verbs to
"%s" verbs.

By contrast, the generic error messages in the defaultError() function
are passed to the Tr variable's Get() method with another argument,
the HTTP request's status code.  Due to the presence of the extra
argument, the Get() method parses the format string, replacing the "%d"
verb with the status code and converting the "%%s" verb into a "%s"
verb.  (Note that originally, in PR git-lfs#4781, we used the Sprintf()
function of the standard "fmt" package rather than the Get() method
of our Tr variable.  We corrected this oversight in commit
04abbd8 of PR git-lfs#4846, but while that
change allows the strings to be localized, it has no effect on the
handling of the formatting verbs.)

As with the error messages from the defaultErrors map, the generic
error messages are then also passed without any other arguments to the
Sprintf() function from the standard "fmt" package.  However, this call
does not have the same effect in both cases.  When the error message is
specific to an HTTP status code and has been selected from the
defaultErrors map, this call just converts the "%%s" verb into a "%s"
one, as noted above.  When the error message is a generic one, though,
that conversion has already been performed by the Tr structure's Get()
method.  The consequence is that the Sprintf() function replaces the
"%s" verb with "%!s(MISSING)", as described in the "fmt" package's
documentation:

  https://pkg.go.dev/fmt#hdr-Format_errors

In the final step of the defaultError() function we then pass the
result from this Sprintf() call to the Errof() function of our
"errors" package, and here we include the HTTP request's URL as a second
argument.  In the case where the error message has been selected from
the defaultErrors map, this last formatting function replaces the "%s"
verb with the URL.  In the case where the error message is a generic one,
though, the leading portion of the format error marker "%!s(MISSING)"
cannot be parsed as a verb, and so the "%!" is replaced with a full
reflection of all the internal elements of the URL structure into
string form.

Since these format error markers and reflection fields are not desirable
in our HTTP request error messages, we revise the defaultError() function
so it each type of error message is passed through only a single
formatting function, specifically the Tr variable's Get() method.
This allows us to use regular "%s" and "%d" formatting verbs exclusively
in all the messages.

Rather than call the Get() method on each string as it is defined in
the defaultErrors map, we only call the method once a specific message
is selected based on the HTTP request's status code, and we pass the
request's URL to the Get() method as another argument.  For the generic
error messages, we simply pass both the status code and the URL as
arguments to the Get() function.  Because we always pass either one or
two arguments to that function in addition to a format string, it always
performs the appropriate interpolation for us, and we avoid the need
to work around its behaviour when it is only passed a format string
without other arguments, as well as avoiding the use of any further
formatting functions like Sprintf().

Co-authored-by: Philip Peterson <[email protected]>
As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.

In the Log() method of the SimpleTask structure in our "tasklog" package,
we pass the method's non-constant input string as the only argument to
the Logf() method, which then uses it as a format string when creating
a task log Update structure.

To resolve this issue, we instead call the Logf() method with a format
string containing a single "%s" verb, and pass the non-constant input
string as the second argument.
As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.

When the "git lfs pointer" command compares two Git LFS pointer files,
it outputs the contents of the file specified by the --pointer option
(or supplied on standard input, if the --stdin option is used) to the
standard error file descriptor.  To write this data, we call the
Fprintf() function of the Go standard library's "fmt" package, and
pass the pointer file data as the format string, although it is not
a constant string value.

To resolve this issue, we simply need to use the Fprint() function
instead, as we already do in a number of other instances in the same
command where we write data to the standard error file descriptor.
As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.

Our Git LFS Batch API test utility, the git-lfs-test-server-api program,
defines an exit() function which accepts a format string and a variable
number of arguments, generates an output message from those, and then
halts the program with a non-zero exit code.  In one instance we call
this exit() function with a non-constant format string we construct
from a fixed string and an error message.

To resolve this issue, we revise the fixed string to include a "%s"
verb, and pass the error as a second argument to the exit() function.
When we report an error, we often call the Errorf() function in our
custom "errors" package, which accepts both a format string and a
variable number of additional arguments.  The intent of this function
is for the additional arguments to be interpolated into the format
string, and then an error created using the string which results from
the interpolation.

In many instances we originally made use of the interpolation capability
of the Errorf() function by passing at least one additional argument to
it other than the format string.  In other cases, though, we used the
Errorf() function with just a single fixed string argument, so the
the New() function of the same "errors" package would have sufficed
instead.

In PR git-lfs#4781 we then updated our code to allow for localization of most
of our output messages by inserting calls to the Get() method of the Tr
structure of our "tr" package.  In the instances where we call the
Errorf() function, the consequence is that we now pass the format string
and any additional arguments to the Get() method, and then just pass
the string it returns to the Errorf() function as a single argument.

As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.  This includes all the cases where we pass
the output from the Get() method of the Tr structure to an Errorf()
function as a single argument.  To resolve this concern, we revise
all such instances to use the New() function of our "errors" package
in place of the Errorf() function.
When we report an error, we often call the Wrapf() function in our
custom "errors" package, which accepts an existing error, a format
string, and a variable number of additional arguments.  The intent of
this function is for the additional arguments to be interpolated into
the format string, and then a new error created using the string which
results from the interpolation, combined with the initial error and
its message.

In many instances we originally made use of the interpolation capability
of the Wrapf() function by passing at least one additional argument to
it other than the existing error and the format string.  In other cases,
though, we used the Wrapf() function with just the existing error and
a fixed string argument, so the Wrap() function of the same "errors"
package would have sufficed instead.

In PR git-lfs#4781 updated our code to allow for localization of most of our
output messages by inserting calls to the Get() method of the Tr
structure of our "tr" package.  In the instances where we call the
Wrapf() function, the consequence is that we now pass the format string
and any additional arguments to the Get() method, and then just pass
the string it returns to the Wrapf() function as the format string
argument following the initial existing error argument.

As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.  This includes all the cases where we pass
the output from the Get() method of the Tr structure to an Wrapf()
function as a format string without any additional arguments to be
interpolated.  To resolve this concern, we revise all such instances to
use the Wrap() function of our "errors" package in place of the Wrapf()
function.
As of Go 1.24, if the Go version number in the "go.mod" file is set to
1.24 or higher, the "go vet" command now reports misuses of non-constant
strings as format strings.  In previous commits in this PR we have now
resolved all but one of the instances where we provided a non-constant
string as a format string.  The remaining instance is our use of the
Errorf() function of the "fmt" package from the Go standard library
at the end of the Combine() function in our "errors" package.

The Combine() function was added to our custom "errors" package in
commit 08e3e5b of PR git-lfs#1870, originally
for use in the "locking" package.  This function merges multiple
errors into a single error by concatenating their error messages
with a newline separator character between each original message.

In Go 1.20 the Join() function was added to the "errors" package
of the Go standard library, and it performs the same concatenation
of error messages as our Combine() function, including the use of a
newline character as a separator between the original messages, except
that it delays the concatenation until the Error() method is called.

To resolve the remaining case where we pass a non-constant string as
a format string, we remove the Combine() function from our "errors"
package and replace it with a Join() function that simply invokes
the Join() function of the standard library's "errors" package,
which we can expect to be defined as we currently require the use of
at least Go 1.21 (per the Go version specified in our "go.mod" file).

To make this change, we alias the standard library's "errors" package
as "goerrors", following the pattern established in our "lfshttp"
package where we use that alias as well.

We then revise the two callers of our Combine() function to make
use of the new Join() function instead.  One of these callers is the
startConnection() function in our "ssh" package, which simply joins
two errors, and the other caller is the FixLockableFileWriteFlags()
method of the Client structure in our "locking" package, which
collects zero or more errors while iterating over a list of files
and repeatedly calling another method.

Because the Join() functions accept variadic arguments rather than
an array of errors, we revise these callers to pass the errors
they intend to concatenate as direct arguments rather than in an
array.  For the FixLockableFileWriteFlags() method this means that
as it iterates over its list of files, if any errors occur we
immediately call the Join() function to add them to the error value
we return at the end of the function, whereas previously we appended
the errors to an array, and then called the Combine() function after
the loop exited to concatenate any errors in the array into the
function's final error return value.
In a previous commit in this PR we replaced the Combine() function
in our custom "errors" package with a Join() function that passes
its arguments through to the Join() function of the Go standard
library's "errors" package.  Both these Join() functions and our
earlier Combine() function serve to concatenate multiple error
values into a single error value, with the individual error messages
separated by newline characters when the error is converted to a
string value for output.

In a number of instances we perform the same concatenation of error
messages using loops and assembling the final error message by
explicitly appending each individual error message along with the
newline separator characters.

We can now simplify all these instances where we need to concatenate
error messages by making use of the Join() function in our "errors"
package.  In almost all of these cases, this means we are able to
eliminate the conditional logic used to check if a newline separator
character is required because more than one error has been encountered.
(The exception is in the Fill() method of the CredentialHelpers
structure of our "creds" package where we previously concatenated
the error messages using the Join() method of the "strings" package
from the Go standard library.)

Note that we could also have used the Combine() function from our
"errors" package to simplify these use cases; however, the Join()
function is more convenient in most instances because it accepts
variadic arguments rather than requiring an array of errors as input.
In PR git-lfs#1466 we introduced our custom "errors" package as a replacement
for the "errutil" package used previously to provide various error
handling functions and specialized error types.  As part of this PR,
in commit 8624a87 we defined the
StackTrace() function in the new "errors" package, and updated the
logPanicToWriter() function in our "commands" package to call it
when generating an error log file.  We write these log files to
the .git/lfs/logs directory whenever a Git LFS command invokes one
of our error reporting utility functions such as LoggedError() or
Panic().

Later, in commit 377366d of PR git-lfs#2178,
we revised the logPanicToWriter() function to use the "%+v" format
string specifier when writing an error's details to the log file,
and dropped the call to the StackTrace() function of our "errors"
package.

The StackTrace() call was deemed no longer necessary because we
expect all of our errors to be created using our "errors" package's
functions, which instantiate error structures defined by the legacy
"github.com/pkg/errors" package.  This package defines its own
Format() methods for its structures so that when the "%+v" format
string specifier is used with them, they output both the message
and stack trace associated with each error.

As a consequence of this change in PR git-lfs#2178, the StackTrace()
function in our own custom "errors" package has no remaining callers,
so we can remove it now.
We recently updated our "netrc" Go package in PR git-lfs/go-netrc#3
with a small change to use a constant format string (as reported by
Go 1.24's "go vet" command).  We therefore now update the Go module
list in this repository to reference the latest version of our
git-lfs/go-netrc project, by running:

  go get github.com/git-lfs/go-netrc@latest

We then also run the "go mod tidy" command to update the set of Go
module hashes in the "go.sum" file.
@chrisd8088 chrisd8088 force-pushed the constant-fmt-strings branch from 58503d8 to 3642879 Compare March 5, 2025 01:37
@chrisd8088 chrisd8088 changed the title [DRAFT] Use constant format strings and fix HTTP error messages Use constant format strings and fix HTTP error messages Mar 5, 2025
@chrisd8088 chrisd8088 removed the wip label Mar 5, 2025
@chrisd8088 chrisd8088 marked this pull request as ready for review March 5, 2025 02:30
@chrisd8088 chrisd8088 requested a review from a team as a code owner March 5, 2025 02:30
@chrisd8088 chrisd8088 merged commit 41999a0 into git-lfs:main Mar 10, 2025
10 checks passed
@chrisd8088 chrisd8088 deleted the constant-fmt-strings branch March 10, 2025 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Go 1.24: non-constant format strings break test build

2 participants