Skip to content

Conversation

@chrisd8088
Copy link
Member

This PR updates our "Releasing Git LFS" documentation page to reflect the current state of our release process, and also adjusts two of the scripts used in that process so they function successfully on macOS systems (or provide more useful error messages if the necessary dependencies are not available).

In particular, our script/upload script expects that the sha3sum(1) utility is available and that it accepts input in the format generated by our script/hash-files script, i.e.:

<hash-name> (<file-name>) = <hash-value>

However, the version of the sha1sum(1) program packaged and installed by Homebrew is different from the one packaged by Debian, which is a Perl script that accepts a wider range of input formats, including the format output by our script/hash-files script. The program provided by Homebrew's sha3sum formula requires input in the following format, and is not able to determine the appropriate algorithm to use without it being specified by an -a command-line option.

<hash-value>  <file-name>

Therefore we adjust our script/upload script so it is able to function with either implementation of the sha1sum(1) utility, since the Perl script also accepts the same format as the Homebrew version.

This PR will be most easily reviewed on a commit-by-commit basis, with whitespace changes ignored.

Before updating the documentation of our release process, we first adjust
the indentation of several paragraphs to match the preceding ones.
Before updating the documentation of our release process, we first correct
a minor misspelling.
In commit d61b3ea of PR git-lfs#5607 we added
support in our Makefile and scripts to build and upload a version of our
Linux release assets for the LoongArch64 platform, and at the same time
we added an entry for this platform to the end of the list of Linux assets
in our release process documentation.

Before updating the documentation of our release process, we first reorder
this list to keep it in alphanumeric order, so as to match how assets are
listed in our public release notes.
In PR git-lfs#5438 we added support in our Makefile and scripts to build and
upload a version of our Linux release assets for the RISC-V 64 platform.

As the documentation of our release process includes a list of the
Linux assets we build, we now add an entry for the RISC-V 64 platform
so as to match the assets listed in our public release notes.
Based on the recent experience of releasing the 3.6.0 version of Git LFS,
we update several steps in our documentation to more accurately reflect
our current release process.

First, we add a reminder that we should include in our release notes
any changes from the previous Git LFS release in terms of platform
requirements or the set of operating system versions we support.

As well, we more fully document the dependencies of our script/finalize
script, and note that a PR to update Homebrew should now be generated
automatically within a few hours of our making a new release public.
Before we release a new version of Git LFS, we generally want to upgrade
the version of Go used in the Docker container images generated by
our git-lfs/build-dockers repository, so we add notes regarding this
step to our release process documentation.

Although Markdown does not require us to renumber the items in an
ordered list, we do so anyway as this makes our documentation more
readable when viewed as a plain text file.
Our script/hash-files script is executed by our script/upload script
to generate hashes of the asset files built by our release process, so
that we can then sign the resulting "hashes" file with a GPG signing key
and publish the signed file in our release announcement.

This script makes use of the OpenSSL::Digest Ruby gem to perform hashing
with a variety of algorithms, including BLAKE2b, BLAKE2s, SHA-512, and
others.

On macOS, a number of these digests require the Ruby 3.x interpreter and
"openssl" gem available via Homebrew rather than the Ruby 2.x versions
provided by current versions of macOS; otherwise, the script returns
"first num too large" errors from OpenSSL.

We therefore add a simple check to our script which exits with a non-zero
code and an error message if the Ruby 3.x interpreter is not in use.
The verify_assets() function in our script/upload script performs a
final verification of the contents of the GPG-signed "sha25sum.asc" and
"hashes.asc" files we include in the assets attached to our public
release notes.

To perform these checks, it runs the shasum(1) utility for all the SHA-2
hashes, the sha3sum(1) utility for the SHA-3 hashes, and the b2sum(1)
utility for the BLAKE2b hashes, passing each utility the appropriate
lines from the "hashes.asc" file.  These lines have the format output
by our script/hash-files script, namely:

  <hash-name> (<file-name>) = <hash-value>

While this format is accepted and parsed successfully by all of the
utilities on most Linux systems, the sha3sum(1) program available on
macOS via Homebrew does not permit this input format with its -c option.
Instead, it requires as input exactly the same format as it outputs when
hashing a set of files, i.e.:

  <hash-value>  <file-name>

This disparity is due to the fact that Homebrew packages as the "sha3sum"
utility a different implementation than is packaged for Debian Linux.
Specifically, the Homebrew package is built from the following source
project:

  https://codeberg.org/maandree/sha3sum
  https://formulae.brew.sh/formula/sha3sum

On Debian, however, the sha3sum(1p) utility is a Perl script provided
with the Digest::SHA3 Perl extension:

  https://metacpan.org/dist/Digest-SHA3/view/sha3sum
  https://sources.debian.org/src/libdigest-sha3-perl/
  https://manpages.debian.org/unstable/libdigest-sha3-perl/sha3sum.1p.en.html

This Perl script accepts a wider range of input formats than the program
packaged by Homebrew, including the one created by our script/hash-files
script.  It is also able to deduce the SHA-3 algorithm to use based on
the hash name in each line, which the sha3sum(1) utility on macOS is
unable to do.

As we sometimes want to run our scripts on macOS when generating a new
Git LFS release, we adjust our script/upload script so it rewrites the
lines from the "hashes.asc" file into the format accepted by the Homebrew
version of sha3sum(1), and also so it invokes that utility separately
for each hash algorithm.  As this also works with the Perl script version
of sha3sum(1p), our script/upload script should now succeed on both
macOS and Linux in the future.
@chrisd8088 chrisd8088 requested a review from a team as a code owner November 24, 2024 07:27
and then use that branch as the base for the PATCH release.

1. If the `release-M.N` branch does not already exist, create it from
1. Upgrade or downgrade the Go version used in the `git-lfs/build-dockers`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would we downgrade?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most obvious situation occurs when Go releases a new minor (or major) version which drops support for older platforms, and we've moved our development branches ahead to keep pace with that.

However, we normally want our patch releases to support the same set of platform versions as all the other Git LFS releases we've made in that series, so we generally don't want to upgrade to a new minor (or major) version of Go for a patch release, and might have to downgrade Go temporarily as a result.

@chrisd8088 chrisd8088 merged commit 9811573 into git-lfs:main Dec 2, 2024
10 checks passed
@chrisd8088 chrisd8088 deleted the update-release-docs-scripts branch December 2, 2024 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants