Skip to content

Stop assuming that Pacman package names cannot contain -<digit>#499

Merged
dscho merged 1 commit into
git-for-windows:mainfrom
dscho:fix-assumptions-about-pacman-package-naming
May 10, 2023
Merged

Stop assuming that Pacman package names cannot contain -<digit>#499
dscho merged 1 commit into
git-for-windows:mainfrom
dscho:fix-assumptions-about-pacman-package-naming

Conversation

@dscho

@dscho dscho commented May 9, 2023

Copy link
Copy Markdown
Member

We currently have a rather big issue, where the Pacman repositories are broken.

The underlying problem is that the Pacman databases are corrupt. To understand why: Pacman databases (i.e. git-for-windows.db which live in the Pacman repositories and are synced into /var/lib/pacman/sync/) are .tar files containing information about the current package versions. This package information is contained in the .tar archives as desc files stored in directories whose names have the format <pkgname>-<pkgver>-<pkgrel> where pkgver and pkgrel correspond to the definitions in the original PKGBUILD file: pkgver refers to the upstream package version, while pkgrel is an incrementing number (starting with 1) that allows for rebuilding the packages even while the pkgver is unchanged.

Now, the issue was that those Pacman repositories were "sanitized", de-duplicating packages, but mistaking msys2-runtime-3.3-3.3.6-1 and msys2-runtime-3.3.6-4 to refer to different versions of the same package, msys2-runtime. To make matters worse, msys2-runtime-3.3-devel-3.3.6-1 was not mistaken to be a different version of the same package as msys2-runtime-devel-3.3.6-4, which means that e.g. git-sdk-64's sync job now consistently fails with:

[...]
:: Starting core system upgrade...
warning: terminate other MSYS2 programs before proceeding
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: installing msys2-runtime (3.4.6-2) breaks dependency 'msys2-runtime=3.3.6' required by msys2-runtime-devel
[...]

This PR fixes this, and all related problems that are also caused by my original incorrect assumption about Pacman package naming.

I might have missed some, or done some incorrectly, though, so I appreciate every thorough review.

@dscho
dscho requested a review from rimrul May 9, 2023 14:24
@dscho dscho self-assigned this May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
dscho added a commit to git-for-windows/git-for-windows-automation that referenced this pull request May 9, 2023
@dscho

dscho commented May 9, 2023

Copy link
Copy Markdown
Member Author

We currently have a rather big issue, where the Pacman repositories are broken.

The underlying problem is that the Pacman databases are corrupt.

FWIW I have fixed this manually in https://github.com/git-for-windows/git-for-windows-automation/actions/runs/4930413416/jobs/8811298893.

This PR is still kind of semi-urgent because it's blocking any further package deployment in Git for Windows (unless we want to see the same breakage happen again 😃)

@jeffhostetler jeffhostetler left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this look fine.

Comment thread make-file-list.sh
As we found the hard way, Pacman package names _can_ legally contain a
dash followed by a digit. That has implications on all the code in Git
for Windows that assumes it can deduce the package name from an archive
name simply by stripping any suffix starting with `-<digit>`.

Concretely, the problem we encountered was that the newly-created
`msys2-runtime-3.3` package was first added correctly to Git for
Windows' Pacman repository, i.e. the `git-for-windows.db` file (which
really is a specially-formatted `.tar` archive) now
contained the following entries:

	msys2-runtime-3.3-3.3.6-1/
	msys2-runtime-3.3-3.3.6-1/desc
	msys2-runtime-3.3-devel-3.3.6-1/
	msys2-runtime-3.3-devel-3.3.6-1/desc
	msys2-runtime-3.3.6-4/
	msys2-runtime-3.3.6-4/desc
	msys2-runtime-devel-3.3.6-4/
	msys2-runtime-devel-3.3.6-4/desc

However, when the _next_ package was added (in this instance, Git
Credential Manager), the `sanitize_db` function in `pacman-helper.sh`
(which is run as part of the `quick_add` function in the same file when
Git for Windows' automation deploys Pacman package) misidentified all
the entries starting with `msys2-runtime-3.3` as referring to the
`msys2-runtime` package.

The reason for this is that I originally assumed that `${dir%%-[0-9]*}`
would be a good way to reduce those directory names to the corresponding
package name, i.e. to strip any suffix starting with a dash followed by
a digit.

However, that is not how Pacman works. As can be seen in the
`find_pkgentry` function of `repo-remove`, the correct way is to strip
the last two dash-separated fields: `${dir%-*-*}`, see
https://github.com/git-for-windows/git-sdk-64/blob/HEAD/usr/bin/repo-remove

Note in particular the difference between using the greedy `${...%%...}`
operator and the reluctant `${...%...}` operator (i.e. two vs one `%`
characters).

Let's fix this (better late than never), by using the `${...%-*-*}`
pattern in shell code, and an equivalent in Perl.

Note: When looking for existing directories in `/var/lib/pacman/local/*`
(to determine whether or not a particular Pacman package is installed or
not), it is not _quite_ as trivial because we used a shell pattern
there: `$package-[0-9]*`. In those instances, I introduced a shell
function that iterates through the paths expanded from that shell
pattern and returns the intended one (if any).

Signed-off-by: Johannes Schindelin <[email protected]>
@dscho
dscho force-pushed the fix-assumptions-about-pacman-package-naming branch from 701c630 to 6769038 Compare May 10, 2023 07:11
@dscho
dscho merged commit 6fe3f63 into git-for-windows:main May 10, 2023
@dscho
dscho deleted the fix-assumptions-about-pacman-package-naming branch May 10, 2023 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants