Stop assuming that Pacman package names cannot contain -<digit>#499
Merged
dscho merged 1 commit intoMay 10, 2023
Merged
Conversation
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows-automation
that referenced
this pull request
May 9, 2023
See git-for-windows/build-extra#499 for details. Signed-off-by: Johannes Schindelin <[email protected]>
Member
Author
FWIW I have fixed this manually in https://github.com/git-for-windows/git-for-windows-automation/actions/runs/4930413416/jobs/8811298893. This PR is still kind of semi-urgent because it's blocking any further package deployment in Git for Windows (unless we want to see the same breakage happen again 😃) |
rimrul
reviewed
May 10, 2023
As we found the hard way, Pacman package names _can_ legally contain a
dash followed by a digit. That has implications on all the code in Git
for Windows that assumes it can deduce the package name from an archive
name simply by stripping any suffix starting with `-<digit>`.
Concretely, the problem we encountered was that the newly-created
`msys2-runtime-3.3` package was first added correctly to Git for
Windows' Pacman repository, i.e. the `git-for-windows.db` file (which
really is a specially-formatted `.tar` archive) now
contained the following entries:
msys2-runtime-3.3-3.3.6-1/
msys2-runtime-3.3-3.3.6-1/desc
msys2-runtime-3.3-devel-3.3.6-1/
msys2-runtime-3.3-devel-3.3.6-1/desc
msys2-runtime-3.3.6-4/
msys2-runtime-3.3.6-4/desc
msys2-runtime-devel-3.3.6-4/
msys2-runtime-devel-3.3.6-4/desc
However, when the _next_ package was added (in this instance, Git
Credential Manager), the `sanitize_db` function in `pacman-helper.sh`
(which is run as part of the `quick_add` function in the same file when
Git for Windows' automation deploys Pacman package) misidentified all
the entries starting with `msys2-runtime-3.3` as referring to the
`msys2-runtime` package.
The reason for this is that I originally assumed that `${dir%%-[0-9]*}`
would be a good way to reduce those directory names to the corresponding
package name, i.e. to strip any suffix starting with a dash followed by
a digit.
However, that is not how Pacman works. As can be seen in the
`find_pkgentry` function of `repo-remove`, the correct way is to strip
the last two dash-separated fields: `${dir%-*-*}`, see
https://github.com/git-for-windows/git-sdk-64/blob/HEAD/usr/bin/repo-remove
Note in particular the difference between using the greedy `${...%%...}`
operator and the reluctant `${...%...}` operator (i.e. two vs one `%`
characters).
Let's fix this (better late than never), by using the `${...%-*-*}`
pattern in shell code, and an equivalent in Perl.
Note: When looking for existing directories in `/var/lib/pacman/local/*`
(to determine whether or not a particular Pacman package is installed or
not), it is not _quite_ as trivial because we used a shell pattern
there: `$package-[0-9]*`. In those instances, I introduced a shell
function that iterates through the paths expanded from that shell
pattern and returns the intended one (if any).
Signed-off-by: Johannes Schindelin <[email protected]>
dscho
force-pushed
the
fix-assumptions-about-pacman-package-naming
branch
from
May 10, 2023 07:11
701c630 to
6769038
Compare
rimrul
approved these changes
May 10, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We currently have a rather big issue, where the Pacman repositories are broken.
The underlying problem is that the Pacman databases are corrupt. To understand why: Pacman databases (i.e.
git-for-windows.dbwhich live in the Pacman repositories and are synced into/var/lib/pacman/sync/) are.tarfiles containing information about the current package versions. This package information is contained in the.tararchives asdescfiles stored in directories whose names have the format<pkgname>-<pkgver>-<pkgrel>wherepkgverandpkgrelcorrespond to the definitions in the originalPKGBUILDfile:pkgverrefers to the upstream package version, whilepkgrelis an incrementing number (starting with 1) that allows for rebuilding the packages even while thepkgveris unchanged.Now, the issue was that those Pacman repositories were "sanitized", de-duplicating packages, but mistaking
msys2-runtime-3.3-3.3.6-1andmsys2-runtime-3.3.6-4to refer to different versions of the same package,msys2-runtime. To make matters worse,msys2-runtime-3.3-devel-3.3.6-1was not mistaken to be a different version of the same package asmsys2-runtime-devel-3.3.6-4, which means that e.g. git-sdk-64'ssyncjob now consistently fails with:This PR fixes this, and all related problems that are also caused by my original incorrect assumption about Pacman package naming.
I might have missed some, or done some incorrectly, though, so I appreciate every thorough review.