Upgrade GitHub Actions workflows to use `ruby/setup-ruby@v1` #5236

chrisd8088 · 2022-12-22T05:32:18Z

As the actions/setup-ruby GitHub Action is deprecated in favour of the ruby/setup-ruby Action, we upgrade to use the latter in our CI and release workflows.

As testing the release.yml workflow is not convenient without generating a release, it has instead been run successfully on a separate, private repository cloned from this one.

MSYS2 Build Environment

In the Windows builds, the ruby/setup-ruby@v1 Action installs the MSYS2 build tools and sets some environment variables which persist for subsequent steps in the workflow, such as the TMPDIR variable.

These differences from the actions/setup-ruby Action introduce failures into our test suite, principally by causing Actions to run the Bash shell in C:\msys64\usr\bin instead of the one in C:\Program Files\Git\bin, which in turn results in the git command running the git.exe binary found in C:\Program Files\Git\bin instead of C:\Program Files\Git\mingw64\bin.

The Git binary in C:\Program Files\Git\bin, when asked to run a git lfs command, then finds the git-lfs.exe binary in C:\Program Files\Git\cmd instead of the one we have built, despite our changing the PATH environment variable so that our built git-lfs.exe binary should be found first.

We resolve this problem by renaming the C:\msys64 directory with a temporary name after installing the asciidoctor Ruby gem.

Also note that we do not need to specify the Bash shell to run the gem install acsiidoctor step, and that we move this step below the ruby/setup-ruby@v1 step in all cases so as to make the dependency more clear. (We also move the gem install packagecloud-ruby step into the same relative location, for the same reason.)

Temporary Files

The ruby/setup-ruby@v1 Action also sets the TMPDIR environment variable for all subsequent steps, and this results in other test suite failures on Windows because we create paths for our temporary test repositories in our GIT_LFS_TEST_DIR path, which we set using mktemp(1), and that derives its default path from TMPDIR. However, some of our test helper programs such as lfstest-customadapter use the Go language ioutil.TempFile() function to construct paths, which calls os.TempDir(), and on Windows that does not refer to the TMPDIR environment variable but the TMP and TEMP ones instead.

The consequence is that tests using those helper programs try to move files between directories creating using both techniques of temporary path creation, and these return paths on different volumes, so the test fails when trying to move files between D:\a\_temp and C:\Users\RUNNER~1\AppData\Local\Temp.

Therefore we take care to unset the TMPDIR variable before running commands which might make use of mktemp(1).

Thanks to @MSP-Greg for the suggestion to rename C:\msys64!

/cc ruby/setup-ruby#293
/cc #5228

As the actions/setup-ruby GitHub Action is deprecated in favour of the ruby/setup-ruby Action, we upgrade to use the latter in our CI and release workflows. In the Windows builds, the ruby/setup-ruby@v1 Action installs the MSYS2 build tools and sets some environment variables which persist for subsequent steps in the workflow, such as the TMPDIR variable. These differences from the actions/setup-ruby Action introduce failures into our test suite, principally by causing Actions to run the Bash shell in C:\msys64\usr\bin instead of the one in C:\Program Files\Git\bin, which in turn results in the "git" command running the git.exe binary found in C:\Program Files\Git\bin instead of C:\Program Files\Git\mingw64\bin. The Git binary in C:\Program Files\Git\bin, when asked to run a "git lfs" command, then finds the git-lfs.exe binary in C:\Program Files\Git\cmd instead of the one we have built, despite our changing the PATH environment variable so that our built git-lfs.exe binary should be found first. We resolve this problem by renaming the C:\msys64 directory with a temporary name after installing the asciidoctor Ruby gem. Also note that we do not need to specify the Bash shell to run the "gem install acsiidoctor" step, and that we move this step below the ruby/setup-ruby@v1 step in all cases so as to make the dependency more clear. (We also move the "gem install packagecloud-ruby" step into the same relative location, for the same reason.) The ruby/setup-ruby@v1 Action also sets the TMPDIR environment variable for all subsequent steps, and this results in other test suite failures on Windows because we create paths for our temporary test repositories in our GIT_LFS_TEST_DIR path, which we set using mktemp(1), and that derives its defaut path from TMPDIR. However, some of our test helper programs such as lfstest-customadapter use the Go language ioutil.TempFile() function to construct paths, which calls os.TempDir(), and on Windows that does not refer to the TMPDIR environment variable but the TMP and TEMP ones instead. The consequence is that tests using those helper programs try to move files between directories creating using both techniques of temporary path creation, and these return paths on different volumes, so the test fails when trying to move files between D:\a\_temp and C:\Users\RUNNER~1\AppData\Local\Temp. Therefore we take care to unset the TMPDIR variable before running commands which might make use of mktemp(1). h/t MSP-Greg for the suggestion to rename C:\msys64

In commit b7fa3a5 of PR git-lfs#5236 we replaced the deprecated actions/setup-ruby@v1 GitHub Action in our CI and release workflows with the current ruby/setup-ruby@v1 Action. We now update our other Actions to their respective latest versions as well: actions/checkout: v1 -> v2 actions/download-artifact: v1 -> v3 actions/setup-go: v2 -> v3 actions/upload-artifact: v1 -> v3 docker/setup-qemu-action: v1 -> v2

In commit b7fa3a5 of PR git-lfs#5236 we replaced the deprecated actions/setup-ruby@v1 GitHub Action in our CI and release workflows with the current ruby/setup-ruby@v1 Action. We now update our other Actions to their respective latest versions as well: actions/checkout: v1 -> v3 actions/download-artifact: v1 -> v3 actions/setup-go: v2 -> v3 actions/upload-artifact: v1 -> v3 docker/setup-qemu-action: v1 -> v2 By default, the actions/checkout Action (as of v2) performs a Git fetch with a --depth=1 option, in which case Git tags for our repository would not be available and so our Makefile and release workflow would fail when calling "git describe". Therefore we specify a fetch depth of 0 where necessary to ensure all tags are available and the full Git history is fetched. This is somewhat more expensive, but our project's history is not excessively large.

In commit b7fa3a5 of PR git-lfs#5236 we replaced the deprecated actions/setup-ruby@v1 GitHub Action in our CI and release workflows with the current ruby/setup-ruby@v1 Action. We now update our other Actions to their respective latest versions as well: actions/checkout: v1 -> v3 actions/download-artifact: v1 -> v3 actions/setup-go: v2 -> v3 actions/upload-artifact: v1 -> v3 docker/setup-qemu-action: v1 -> v2 By default, the actions/checkout Action (as of v2) performs a Git fetch with a --depth=1 option, in which case Git tags for our repository would not be available and so our Makefile and release workflow would fail when calling "git describe". Therefore we specify a fetch depth of 0 where necessary to ensure all tags are available and the full Git history is fetched. This is somewhat more expensive, but our project's history is not excessively large. As well, as of v2 of the actions/download-artifact Action, the downloaded assets are not placed in a new subdirectory created with the name of the Action, but directly in the current working directory. We want to retain the previous behaviour so we add a "path" argument with the same name as each of the macos-assets and windows-assets download steps.

In commit b7fa3a5 of PR git-lfs#5236 we replaced the deprecated actions/setup-ruby@v1 GitHub Action in our CI and release workflows with the current ruby/setup-ruby@v1 Action. We now update our other Actions to their respective latest versions as well: actions/checkout: v1 -> v3 actions/download-artifact: v1 -> v3 actions/setup-go: v2 -> v3 actions/upload-artifact: v1 -> v3 docker/setup-qemu-action: v1 -> v2 As of v2 of the actions/download-artifact Action, downloaded assets are not placed in a new subdirectory created with the name from the step's "name" argument, but in the current working directory instead. We want to retain the previous behaviour so we add a "path" argument with the same name as each of the macos-assets and windows-assets download steps. By default, the actions/checkout Action (as of v2) performs a Git fetch with a --depth=1 option, so a shallow clone is made. As a result, when our Makefile calls "git describe HEAD" to set its VERSION variable, no tags are available and Git responds with an error message. Many of our workflow jobs succeed despite logging that error, including the build-docker and build-docker-cross jobs in both our CI and Release workflows. (The Docker builds create upload artifacts with the correct filenames despite the lack of any tags because they rely on the Git LFS version strings in our debian/changelog file and in our binary; the rpm/build_rpms.bsh script builds a binary just to run "git-lfs version" and determine the version string from its output.) However, our workflow jobs which run the "make release" command fail outright in the absence of any Git tags, as they search for build artifacts using filenames constructed with the empty VERSION variable, such as "git-lfs-windows-amd64-.zip". When no files are found, the tar command fails, halting the job. This affects both the build-default job in our CI workflow (for Linux and macOS), and all of build-main, build-macos, and build-windows jobs in our Release workflow. To resolve this in the case of a PR or other push to a branch, we set a fetch-depth value of 0 for our actions/checkout@v3 steps, which downloads the full Git history and all tags. This is somewhat more expensive than a shallow clone, but our project's history is not excessively large. Due to the GitHub Actions bug documented in actions/checkout#882, though, this resolution is insufficient in the case of a push to a tag. At present, the actions/checkout@v3 incorrectly determines the SHA of an annotated tag to be the SHA of its associated commit, and then proceeds as if the tag had been updated on the server since the Action was started, and so rewrites the tag locally to refer to the commit SHA. This has the effect of making the local tag into a lightweight tag, which "git describe" then ignores (since we don't pass the --tags option to it). As a temporary fix for this problem, we add a step after the actions/checkout@v3 step which updates the local tag again to match the remote one. We only run this step when the pushed reference was a tag, because on a branch push it would fail as Git would refuse to update the currently checked-out branch. In our Release workflow, since it only runs on pushes to tags, we can run this step unconditionally. (We could also continue to use the default fetch-depth of 1 for the actions/checkout@v3 step, since we always subsequently fetch the relevant tag, but to be consistent and to avoid future issues once actions/checkout#882 is fixed upstream, we do not do so.)

.github/workflows/ci.yml

In commit b7fa3a5 in this PR we updated our GitHub Actions workflows to use the ruby/setup-ruby@v1 Action, and in doing so had to resolve several issues for our Windows runners regarding that Action, specifically to clear the TMPDIR environment variable it sets so mktemp(1) and Go use the same volume for temporary files, and to move the MSYS2 installation under C:\msys64 aside so the Git in that path is not used by default. Otherwise, that Git prefers its own PATH lookup to the one our build scripts set, and continues to find its own git-lfs binary instead of the one we have built and want to test. To document these additional workflow steps better we now also add comments into our workflow definitions, as suggested by larsxschneider on PR review.

In commit b7fa3a5 of PR git-lfs#5236 we replaced the deprecated actions/setup-ruby@v1 GitHub Action in our CI and release workflows with the current ruby/setup-ruby@v1 Action. We now update our other Actions to their respective latest versions as well: actions/checkout: v1 -> v3 actions/download-artifact: v1 -> v3 actions/setup-go: v2 -> v3 actions/upload-artifact: v1 -> v3 docker/setup-qemu-action: v1 -> v2 As of v2 of the actions/download-artifact Action, downloaded assets are not placed in a new subdirectory created with the name from the step's "name" argument, but in the current working directory instead. We want to retain the previous behaviour so we add a "path" argument with the same name as each of the macos-assets and windows-assets download steps. By default, the actions/checkout Action (as of v2) performs a Git fetch with a --depth=1 option, so a shallow clone is made. As a result, when our Makefile calls "git describe HEAD" to set its VERSION variable, no tags are available and Git responds with an error message. Many of our workflow jobs succeed despite logging that error, including the build-docker and build-docker-cross jobs in both our CI and Release workflows. (The Docker builds create upload artifacts with the correct filenames despite the lack of any tags because they rely on the Git LFS version strings in our debian/changelog file and in our binary; the rpm/build_rpms.bsh script builds a binary just to run "git-lfs version" and determine the version string from its output.) However, our workflow jobs which run the "make release" command fail outright in the absence of any Git tags, as they search for build artifacts using filenames constructed with the empty VERSION variable, such as "git-lfs-windows-amd64-.zip". When no files are found, the tar command fails, halting the job. This affects both the build-default job in our CI workflow (for Linux and macOS), and all of build-main, build-macos, and build-windows jobs in our Release workflow. To resolve this in the case of a PR or other push to a branch, we set a fetch-depth value of 0 for our actions/checkout@v3 steps, which downloads the full Git history and all tags. This is somewhat more expensive than a shallow clone, but our project's history is not excessively large. Due to the GitHub Actions bug documented in actions/checkout#882, though, this resolution is insufficient in the case of a push to a tag. At present, the actions/checkout@v3 incorrectly determines the SHA of an annotated tag to be the SHA of its associated commit, and then proceeds as if the tag had been updated on the server since the Action was started, and so rewrites the tag locally to refer to the commit SHA. This has the effect of making the local tag into a lightweight tag, which "git describe" then ignores (since we don't pass the --tags option to it). As a temporary fix for this problem, we add a step after the actions/checkout@v3 step which updates the local tag again to match the remote one. We only run this step when the pushed reference was a tag, because on a branch push it would fail as Git would refuse to update the currently checked-out branch. In our Release workflow, since it only runs on pushes to tags, we can run this step unconditionally. (We could also continue to use the default fetch-depth of 1 for the actions/checkout@v3 step, since we always subsequently fetch the relevant tag, but to be consistent and to avoid future issues once actions/checkout#882 is fixed upstream, we do not do so.)

In PR git-lfs#3840 we introduced a GitHub Actions workflow to build our release assets when we push a new version tag. As a final step in this workflow we run our script/packagecloud.rb script, which requires the use of the "packagecloud-ruby" Ruby gem. At the time, these requirements meant we needed to first execute the actions/setup-ruby action so we could then install the gem and run our Ruby script. Later, in PR git-lfs#4230, we added steps to our CI workflow to build our manual pages, and since these required the use of the "ronn" Ruby gem, we also added the actions/setup-ruby action to our CI workflow. (We subsequently migrated our manual page source files to the AsciiDoc format, in PR git-lfs#5054, but we continue to use Ruby and the "asciidoctor" gem to build our manual pages.) Then in commit b7fa3a5 of PR git-lfs#5236 we upgraded both of our Actions workflows to use the ruby/setup-ruby action instead of the now-deprecated actions/setup-ruby one. Because the ruby/setup-ruby action installs the MSYS2 environment on Windows and sets several key environment variables like PATH and TMPDIR, we also introduced steps to make sure our CI and release processes continued to work as expected in this context, by clearing the TMPDIR variable and renaming the directory containing the MSYS2 executables. Fortunately, the default runners provided by GitHub Actions for the macOS, Windows, and Ubuntu Linux platforms are now all provisioned with an installation of Ruby 3.x which includes the "gem" utility we need to install the "asciidoctor" and "packagecloud-ruby" gems. As these are all we require to run our Ruby scripts, we no longer need the more extensive Ruby development environment provided by the ruby/setup-ruby action. We can therefore simply remove the ruby/setup-ruby steps from our workflows, along with all the special handling of the MSYS2 environment and the TMPDIR environment variable on Windows.

chrisd8088 requested a review from a team as a code owner December 22, 2022 05:32

This was referenced Dec 22, 2022

Check for github action updates monthly #5228

Merged

Allow to unset env variables actions/runner#1126

Open

chrisd8088 mentioned this pull request Dec 30, 2022

Upgrade workflows to latest Ubuntu and Actions versions #5243

Merged

larsxschneider approved these changes Jan 3, 2023

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

chrisd8088 added 2 commits January 3, 2023 19:52

Merge branch 'main' into actions-ruby-setup

64c0d68

chrisd8088 merged commit cbab54d into git-lfs:main Jan 4, 2023

chrisd8088 deleted the actions-ruby-setup branch January 4, 2023 05:47

chrisd8088 mentioned this pull request Feb 7, 2025

Update Windows CI jobs to find Perl prove command #5983

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upgrade GitHub Actions workflows to use `ruby/setup-ruby@v1` #5236

Upgrade GitHub Actions workflows to use `ruby/setup-ruby@v1` #5236

Uh oh!

chrisd8088 commented Dec 22, 2022 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Upgrade GitHub Actions workflows to use ruby/setup-ruby@v1 #5236

Upgrade GitHub Actions workflows to use ruby/setup-ruby@v1 #5236

Uh oh!

Conversation

chrisd8088 commented Dec 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MSYS2 Build Environment

Temporary Files

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Upgrade GitHub Actions workflows to use `ruby/setup-ruby@v1` #5236

Upgrade GitHub Actions workflows to use `ruby/setup-ruby@v1` #5236

chrisd8088 commented Dec 22, 2022 •

edited

Loading