Skip to content

Conversation

@chrisd8088
Copy link
Member

We add a --fixup option to the git lfs migrate info command which has similar properties to the --fixup option to the git lfs migrate import command, namely that it looks for files in the repository which should be Git LFS pointers (according to the filter settings in .gitattributes files) but which are not.

In the case of the migrate info command this new option only reports the sum totals for any such files, and does not make any changes to the repository. When the --fixup option is supplied, the migrate info command will not output a summary of any other files, including .gitattributes files, which are otherwise normally included in the command's final output.

The new --fixup option is taken to imply the --pointers=ignore option setting, and is incompatible with any other settings of that option.

We add tests of the new option, including invalid option combinations, and also make some related improvements to the existing tests in the t/t-migrate-info.sh script, notably by breaking up "omnibus" tests added in #4436 into separate tests, and by refactoring all the tests of invalid options so they align with other scripts' and check that the command exits with an expected failure code. We also adjust the text of one such invalid option error message (again from #4436) to align better with the other existing and new error messages for such cases.

Fixes #4419.
/cc @kid1412621 as reporter.

We adjust the text of the error message output in the case
of an unrecognized value for the --pointers option to the
"git lfs migrate info" command, in order to make the text
and calling convention align with other similar option
parsing and error handling.
We refactor the tests of various invalid options to the
"git lfs migrate info" command to use the same checks as
those found in the t/t-migrate-fixup.sh tests, among others.
This allows us to confirm both the error message text and
that the command fails as expected.
We split out the existing tests of various values of the
"git lfs migrate info" --pointers option, added in PR git-lfs#4436,
in order to treat each value in a separate test.  This has
the small advantage that if support for a particular option
value should be broken unexpectedly in the future, it
should result in a specific test failure instead of a more
generic one.
We add a --fixup option to the "git lfs migrate info"
command which has similar properties as the --fixup option
to the "git lfs migrate import" command, namely that it
looks for files in the repository which should be Git LFS
pointers (according to the filter settings in .gitattributes)
but which are not.

In the case of the "migrate info" command this new option
only reports the sum totals for any such files, and does
not make any changes to the repository.  When the --fixup
option is supplied, the "migrate info" command will not
output a summary of any other files, including .gitattributes
files, which are otherwise normally included in the command's
final output.

The new --fixup option is taken to imply the --pointers=ignore
option, and is incompatible with any other settings of that
option.
@chrisd8088 chrisd8088 marked this pull request as ready for review May 17, 2021 20:23
@chrisd8088 chrisd8088 requested a review from a team May 17, 2021 20:40
Copy link
Member

@bk2204 bk2204 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@chrisd8088 chrisd8088 merged commit cadbbe7 into git-lfs:main May 18, 2021
@chrisd8088 chrisd8088 deleted the migrate-info-fixup branch May 18, 2021 14:56
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Jun 1, 2023
The "migrate info (--fixup, no .gitattributes)" test was added
to the t/t-migrate-info.sh test suite in commit
4800c5e of PR git-lfs#4501, when we
added --fixup option support to the "git lfs migrate info" command.

As we expect to introduce additional, similar tests in subsequent
commits in this PR, we rename the test to clarify that no fixup
action is expected in this case, and refactor the test's invocation
of the "git lfs migrate info --fixup" command so that it does not
run inside a pipeline.  This latter change ensures that if the
command fails, the test also fails due to the "set -e" shell option.

While this change to avoid the use of a pipeline is not critical for
this test, it will be important for the similar tests we add in
subsequent commits, because they will need to validate that the
"git lfs migrate info --fixup" command does not simply crash and
produce no output on stdout, which, if it occured in a pipeline,
would not be detected.  This is particularly true in these tests
because we expect the command to produce no output on stdout, which
it would not do both on success and if it panics, so we can not
distinguish these cases by examining the number of lines of output.

We then make a copy of this test in the t/t-migrate-fixup.sh test
suite as well, except using the "git lfs migrate import --fixup"
command, as we also expect to add similar tests to this suite in
subsequent commits.  Note that for this test it is critical that
we supply the --yes option to the command, to avoid it waiting
forever for input because it has detected the log file as a new
file in the Git working tree and therefore has prompted for
confirmation to proceed.

Finally, we remove one unneeded "git add" command from the
setup_single_local_branch_tracked_corrupt() test helper function,
which we refactor further in subsequent commits.
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Jun 1, 2023
Support for Git macro attributes was added in a series of commits
in PR git-lfs#3391, including commit 9d3e52d,
where the Line structure of the "git/gitattr" package was updated
to include a Macro element which would be set non-nil for macro
definition lines in a .gitattributes file file, while the existing
Pattern element would be set non-nil for all other lines.

The "git lfs track" command, among others, was then adjusted to
create a MacroProcessor structure (from the same "git/gitattr" package)
and call its ProcessLines() method to resolve any macro references
and thus convert the "raw" parsed Line structures into a set for
which the Pattern element was always non-nil, and no Macro elements
appeared.

Later, the "git lfs fsck" command gained the ability to process
macro definitions in .gitattributes files, in PR git-lfs#4525.

However, the "git lfs migrate import" command was not adjusted,
specifically in the implementation of its "--fixup" option, which
initializes a Tree structure (also of the "git/gitattr" package) for
the root tree of each commit in a repository's history using the
package's New() function.  This function traverses all the trees
in the hierarchy and finds and parses all the .gitattributes files
in them.  Then, when the command visits each file within the
commit's tree using the Rewrite() method of the Rewriter structure
in the "git/githistory" package, it calls the (*Tree).Applied()
method to match the file's path against any applicable Git attributes,
to see if the file should be treated as a Git LFS object.

This lack of support for macro attributes in the "git lfs migrate
import --fixup" command was then propagated to the "git lfs migrate
info --fixup" command in commit 4800c5e
of PR git-lfs#4501, when the "git lfs migrate info" command was updated to
respect the --fixup option.

As a result, both of these commands (when used with the --fixup option)
would panic if they encountered a .gitattributes file with any macro
definition, as they would call the (*Tree).Applied() method and it
would attempt to access the nil Pattern element of the lines with
non-nil Macro elements.  (Prior to the changes in commit
c374d1f of PR git-lfs#5375 the "git lfs migrate
import --fixup" command would then stall indefinitely, but it now
also exits after the panic condition.)  These problems were reported
in issue git-lfs#5332.

To resolve this problem and avoid similar ones in the future, we
refactor the Line structure into a Line interface, which only provides
a Attrs() method to retrieve a slice of Attr attributes, and no other
methods.  We then also define two additional interfaces, each of which
embeds the Line interface, PatternLine and MacroLine, with corresponding
getter methods for their respective elements.

The ParseLine() function of the "git/gitattr" package now returns a
slice of generic Line types, each of which is either a PatternLine
or a MacroLine, but never both.  Callers like the Applied() method of
the Tree structure therefore need to perform type assertions or
switches to determine which type of Line they are handling, which
ensures they always access the line's data through safe methods.

We then update the Go tests for the "git/gitattr" package as
appropriate, and also add two tests each to the t/t-migrate-fixup.sh
and t/t-migrate-import.sh test suites.  All four of these new
shell tests fail without the changes in this commit.  In particular,
several of these tests make sure to run the "git lfs migrate" commands
outside of any shell pipeline so the test will fail if the command
panics and produces no output, even if no output is the expected
condition for a successful execution of the command.
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Jun 1, 2023
Support for Git macro attributes was added in a series of commits
in PR git-lfs#3391, including commit 9d3e52d,
where the Line structure of the "git/gitattr" package was updated
to include a Macro element which would be set non-nil for macro
definition lines in a .gitattributes file file, while the existing
Pattern element would be set non-nil for all other lines.

The "git lfs track" command, among others, was then adjusted to
create a MacroProcessor structure (from the same "git/gitattr" package)
and call its ProcessLines() method to resolve any macro references
and thus convert the "raw" parsed Line structures into a set for
which the Pattern element was always non-nil, and no Macro elements
appeared.

Later, the "git lfs fsck" command gained the ability to process
macro definitions in .gitattributes files, in PR git-lfs#4525.

However, the "git lfs migrate import" command was not adjusted,
specifically in the implementation of its "--fixup" option, which
initializes a Tree structure (also of the "git/gitattr" package) for
the root tree of each commit in a repository's history using the
package's New() function.  This function traverses all the trees
in the hierarchy and finds and parses all the .gitattributes files
in them.  Then, when the command visits each file within the
commit's tree using the Rewrite() method of the Rewriter structure
in the "git/githistory" package, it calls the (*Tree).Applied()
method to match the file's path against any applicable Git attributes,
to see if the file should be treated as a Git LFS object.

This lack of support for macro attributes in the "git lfs migrate
import --fixup" command was then propagated to the "git lfs migrate
info --fixup" command in commit 4800c5e
of PR git-lfs#4501, when the "git lfs migrate info" command was updated to
respect the --fixup option.

As a result, both of these commands (when used with the --fixup option)
panic if they encounter a .gitattributes file with any macro
definitions, as they call the (*Tree).Applied() method and it
attempts to access the nil Pattern element of the lines with
non-nil Macro elements.  (Prior to the changes in commit
c374d1f of PR git-lfs#5375 the "git lfs migrate
import --fixup" command would then stall indefinitely, but it now
also exits after the panic condition.)  These problems were reported
in issue git-lfs#5332.

To resolve this problem and avoid similar ones in the future, we
refactor the Line structure into a Line interface, which only provides
an Attrs() method to retrieve a slice of Attr attributes, and no other
methods.  We then also define two additional interfaces, each of which
embeds the Line interface, PatternLine and MacroLine, with corresponding
getter methods for their respective elements.

The ParseLine() function of the "git/gitattr" package now returns a
slice of generic Line types, each of which is either a PatternLine
or a MacroLine, but never both.  Callers like the Applied() method of
the Tree structure therefore need to perform type assertions or
switches to determine which type of Line they are handling, which
ensures they always access the line's data through safe methods.

We then update the Go tests for the "git/gitattr" package as
appropriate, and also add two tests each to the t/t-migrate-fixup.sh
and t/t-migrate-import.sh test suites.  All four of these new
shell tests fail without the changes in this commit.  In particular,
several of these tests make sure to run the "git lfs migrate" commands
outside of any shell pipeline so the test will fail if the command
panics and produces no output, even if no output is the expected
condition for a successful execution of the command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--fixup flag to support git lfs migrate info

2 participants