[entt, libharu, libopensp, minizip, smf, vlpp, vulkan-headers] Fix incorrect versions by njakob · Pull Request #28571 · microsoft/vcpkg

njakob · 2022-12-27T01:19:50Z

Describe the pull request

What does your PR fix?

While working on vcpkg.link, I realized that some ports' versions point to invalid git trees. Therefore, this PR fixes these different issues.

In order to find these invalid git trees, it is possible to check the output of the git ls-tree command. For example, the duplicate sdl2 3.10.0#0 has a git tree of 965c440d8611528f1069a2a494f11da420110408 and running the command produces the following output:
```
$ git ls-tree 965c440d8611528f1069a2a494f11da420110408
fatal: not a tree object
```
This is incorrect as we should be able to retrieve the git tree containing all the files as the working 3.10.0#0 version entry:
```
$ git ls-tree a871a9d0c7187960052099119854369e854c3e50
100644 blob 63ba1818e98160d2f040c801aa16d0453d40e010	portfile.cmake
100644 blob 92b5a87baee9670140ba5bcb032453d399405d16	vcpkg.json
```
It is also possible to directly see the content of a file in a given tree by using git show
```
$ git show a871a9d0c7187960052099119854369e854c3e50:vcpkg.json
```
When updating git refs, since ./vcpkg x-ci-verify-versions --verify-git-trees validates that it links to a matching manifest or control file with the given package version and port version, we have to either find an existing tree with the right version or remove the invalid entry. In this PR, it was only possible to reverse the changes for libopensp and invalid versions has been removed for entt, libharu, minizip, smf, vlpp and vulkan-headers.

Semi-related to this PR, How do I get history versions of a port now #27451 suggests using git blame in order to get the history of a port. While changing the file would alter this history, using git blame already doesn't provide any data before the introduction of the versions files. In other words, vcpkg does not and should not rely on git blame for simulating this functionality.
Which triplets are supported/not supported? Have you updated the CI baseline?

Support was not changed in this PR
Does your PR follow the maintainer guide?

Yes

If you have added/updated a port: Have you run `./vcpkg x-add-version --all` and committed the result?

N/A

The output of ./vcpkg x-ci-verify-versions --verify-git-trees now only show unknown errors ([many ports] Fix version files #24277)

FAIL: abseil

Error: While reading versions for port abseil from file: ./vcpkg/versions/a-/abseil.json
     While validating version: 2020-03-03#7.
     While trying to load port from: 28fa609b06eec70bb06e61891e94b94f35f7d06e:vcpkg.json
     Found the following error(s):

FAIL: blend2d

Error: While reading versions for port blend2d from file: ./vcpkg/versions/b-/blend2d.json
     While validating version: beta_2020-07-31.
     While trying to load port from: ffce764b880d8cc24e3b00328aa3861f15bae91d:vcpkg.json
     Found the following error(s):

FAIL: libwebp

Error: While reading versions for port libwebp from file: ./vcpkg/versions/l-/libwebp.json
     While validating version: 1.1.0.
     While trying to load port from: a05e0de81085231df86f6902aba1e0a364e2ca7b:CONTROL
     Found the following error(s):
        a05e0de81085231df86f6902aba1e0a364e2ca7b:CONTROL:1:94: error: invalid character in feature name (must be lowercase, digits, '-', or '*')
     on expression: libwebp[anim, gif2webp, img2webp, info, mux, nearlossless, simd, cwebp, dwebp], libwebp[vwebp_sdl] (!osx), libwebp[extras] (!osx)

njakob · 2022-12-27T01:21:57Z

@microsoft-github-policy-service agree

njakob · 2022-12-27T01:22:31Z

versions/e-/entt.json

Version 3.10.0#0 is duplicated with different git trees. The duplication has been introduced by #24581 (1) and seems to be safe to be removed.

njakob · 2022-12-27T01:28:25Z

versions/l-/libharu.json

This has been introduced by #25613 (1) as most likely a mistake since the PR is about boost-build.

Simply updated the git tree with the previous version since the port was not changed.

We could also simply remove the version completely.

We should find the proper SHA for version 2017-08-15 if it exsits.

Error: While reading versions for port libharu from file: /workspaces/vcpkg/versions/l-/libharu.json While validating version: 2017-08-15#11. The version declared in file does not match checked-out version: 2017-08-15#10 Checked out Git SHA: 312f4b697d2f46818c218e270bd447cdeb76322c

While checking the history of this port, it seems we do not have a valid git tree for 2017-08-15#11: We move from 2017-08-15#10 (#19979) to 2.4.0-rc1#0 (#25569) directly. Therefore, I simply removed the version entry.

njakob · 2022-12-27T01:30:14Z

versions/l-/libopensp.json

The git tree got changed by mistake in #26382 (1).

This reverts the original git tree of 1.5.2#0 while keeping the 1.5.2#1 version.

njakob · 2022-12-27T01:33:00Z

versions/m-/minizip.json

It seems that 1.2.12#0 got introduced by mistake in #27226 (1).

The fix simply uses the git tree of 1.2.13#0.

We could also simply remove the version completely.

Since pointing to different manifest or control files with different package and port versions trigger validation errors when running ./vcpkg x-ci-verify-versions --verify-git-trees, the version was removed.

njakob · 2022-12-27T01:34:39Z

versions/s-/smf.json

The version 0.1.0#0 was most likely added by mistake by #28162 (1) since the PR reference clearly 0.1.1.

The version pointing to the invalid git tree got removed.

njakob · 2022-12-27T01:37:48Z

versions/v-/vlpp.json

The version 0.11.0.0#4 mostly likely added by mistake by #26887 (1) since 1.1.0.0#0 was added on the same PR.

The version pointing to the invalid git tree got removed.

njakob · 2022-12-27T01:40:27Z

versions/v-/vulkan-headers.json

The version 1.3.233#0 was most likely added by mistake in #27746 (1) since the PR clearly references 1.3.234.

Therefore, the version pointing to an invalid git tree got removed.

Thomas1664 · 2022-12-27T08:15:08Z

@njakob out of curiosity, how did you find those incorrect versions? There already is a GitHub bot that makes sure the version database is updated correctly but as you have seen, there are ways to fool the bot.

dg0yt · 2022-12-27T09:14:17Z

<off-topic>
@njakob On vcpkg.link:
"No license" is displayed when a port manifest has "license": null. IMO this is misleading. These ports do have license terms. They are just too complex to be expressed in the SPDX license field, or strongly depend on selected features.
Example: gdal
Suggestion: Display "Complex license" or similar.
Reference: https://github.com/microsoft/vcpkg/blob/master/docs/maintainers/manifest-files.md#license
</off-topic>

njakob · 2022-12-27T10:47:47Z

@njakob out of curiosity, how did you find those incorrect versions? There already is a GitHub bot that makes sure the version database is updated correctly but as you have seen, there are ways to fool the bot.

That's a good question @Thomas1664! Since this should have been part of the PR description, I added some details on how I did find these broken versions.

First of all, I haven't checked in detail how the bot is generating suggestions. However, it seems we're not performing an integrity check of the changed versions file.

We could imagine adding a separate GitHub action performing a git ls-tree of each entry of the changed files. This way we would catch such cases before they get merged into master and keep the existing suggestion system.

I could take a look at this in the following days if that's something you believe could be valuable!

njakob · 2022-12-27T11:12:52Z

@njakob On vcpkg.link: "No license" is displayed when a port manifest has "license": null. IMO this is misleading. These ports do have license terms. They are just too complex to be expressed in the SPDX license field, or strongly depend on selected features. Example: gdal Suggestion: Display "Complex license" or similar. Reference: https://github.com/microsoft/vcpkg/blob/master/docs/maintainers/manifest-files.md#license

That's very valuable feedback @dg0yt, I have not considered this case and missed the meaning of the null value. Your idea to use a different terminology makes a lot of sense!

Quickly searching over the repository, it seems we also have cases with a license file without any indication in the manifest (e.g. vulkan). I'm wondering if we should somehow either always add the "complex license" as part of the port or enable contributors to specify an external URL for it. This, however, deserves its own issue to be discussed and tracked.

I'll make sure to address the issue you raised @dg0yt today! Edit: I pushed an update that includes your suggestion!

vicroms · 2023-01-02T05:45:35Z

There is an existing --verify-git-trees flag that checks for consistency in the version database. I'm working on some changes to enable it on every PR run, but in its current implementation is too costly to run.

vicroms

I ran ./vcpkg x-ci-verify-versions --verify-git-trees on this PR.

The errors on abseil, blend2d and libwebp are known from #24277 but libharu and minizip have incorrect fixes on this PR.

vicroms · 2023-01-02T05:53:18Z

versions/l-/libharu.json

We should find the proper SHA for version 2017-08-15 if it exsits.

Error: While reading versions for port libharu from file: /workspaces/vcpkg/versions/l-/libharu.json While validating version: 2017-08-15#11. The version declared in file does not match checked-out version: 2017-08-15#10 Checked out Git SHA: 312f4b697d2f46818c218e270bd447cdeb76322c

njakob

Thank you for having a look @vicroms and pointing out ./vcpkg x-ci-verify-versions --verify-git-trees, I was not aware of this command!

Looking quickly at the current implementation, we could imagine only validating files that change in PRs instead of every single port.

Regarding the two new errors you raised for libharu and minizip, I simply removed the version entries since we do not have any git tree with a matching package and port version.

njakob · 2023-01-02T14:07:18Z

versions/l-/libharu.json

While checking the history of this port, it seems we do not have a valid git tree for 2017-08-15#11: We move from 2017-08-15#10 (#19979) to 2.4.0-rc1#0 (#25569) directly. Therefore, I simply removed the version entry.

njakob · 2023-01-02T14:08:51Z

versions/m-/minizip.json

Since pointing to different manifest or control files with different package and port versions trigger validation errors when running ./vcpkg x-ci-verify-versions --verify-git-trees, the version was removed.

vicroms · 2023-01-04T22:40:16Z

Thanks for the PR!

gharwo · 2023-01-11T15:28:27Z

Related to this, in the process of trying to establish a maintenance branch for our code consuming libharu at version 2017-08-15#10, I seem to have come across issue 25518, which was fixed by PR2553. The problem appears to be that the fix is the removal of the TIFF symbols and not fixing the original hash:

url : [ https://github.com/libharu/libharu/pull/157.diff ]
1> File path : [ C:\Users<user>\AppData\Local\Programs\vcpkg\downloads\libharu-shading-pr-157.patch.28904.part ]
1> Expected hash : [ f2ddb22b54b4eccc79400b6a4b2d245a221898f75456a5a559523eab7a523a87dfc5dfd0ec5fb17a771697e03c7ea6ed4c6095eff73e0a4302cd6eb24584c957 ]
1> Actual hash : [ 10562c8c9c6975b74d625912e7d3b850376941d5e46136679371ce1315f416b6b8e91d73fd1257b907327b2d71d50b8c026547203f280e42fb7b1901245de03c ]

This means building the versions prior to the fix still produce the hash error. I hope I've understood this correctly. This is on a fresh install of vcpkg and after doing a git pull.

This is the vcpkg.json I was using:
{ "name": "repsu", "version-string": "5.0.0", "dependencies": [ { "name": "libharu", "version>=": "2017-08-15#10" } ], "builtin-baseline": "ba383ed2332f42593a30dd3f35b09f4a59d31d91" }

vicroms · 2023-01-11T18:34:15Z

@gharwo

The SHA bfeaf0d13fce9156ac216daa37a2c945290fc0ed is invalid in a local clone of vcpkg, and because of that, removing them from the version database was the right thing to do.

However, since GitHub keeps the PR refs around (and thus the git objects), one can obtain the files for that specific SHA from GitHub via https://github.com/Microsoft/vcpkg/archive/bfeaf0d13fce9156ac216daa37a2c945290fc0ed.zip.

If you must take that version and cannot upgrade the port, then you should download the port files form the URL provided above and create a port overlay for libharu.

dg0yt · 2023-01-11T19:26:23Z

@vicroms @gharwo The problem in #28571 (comment) is the occassional vcpkg habbit of downloading diffs from other repos instead of committing them to vcpkg. You will always get a valid diff from GH, but it is not binary stable over time. So SHA512 checks can fail after some time.

The variable part in the downloaded diffs ist the format of the index lines. Example for the given diff:
original: index 44b816c..60b1d84 100644
current: index 44b816c2..60b1d847 100644

A regex substitution e.g. in VS Code transforms the download from the actual hash to the expected hash: (Hey, I produced a SHA512 collission ;-)
(^index .......).(\.\........). -> $1$2

gharwo · 2023-01-12T12:29:19Z

OK. Thanks for the hint. I'll see how I get on.

…

On Wed, 11 Jan 2023 at 18:34, Victor Romero ***@***.***> wrote: @gharwo <https://github.com/gharwo> The SHA bfeaf0d13fce9156ac216daa37a2c945290fc0ed is invalid in a local clone of vcpkg, and because of that, removing them from the version database was the right thing to do. However, since vcpkg keeps the PR refs around (and thus the git objects), one can obtain the files for that specific SHA from GitHub via https://github.com/Microsoft/vcpkg/archive/bfeaf0d13fce9156ac216daa37a2c945290fc0ed.zip . If you must take that version and cannot upgrade the port, then you should download the port files form the URL provided above and create a port overlay for libharu. — Reply to this email directly, view it on GitHub <#28571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXENQ5YVA5RLUPUBAGLI4UTWR34LFANCNFSM6AAAAAATKAF2II> . You are receiving this because you were mentioned.Message ID: ***@***.***>

gharwo · 2023-01-18T09:05:10Z

Pulling that specific PR ref worked perfectly and enabling the overlay port in vcpkg-configuration embedded in vcpkg.json of the maintenance branch means less churn for maintenance releases. Thanks.

[entt] Remove incorrect version

7385077

njakob commented Dec 27, 2022

View reviewed changes

MonicaLiu0311 assigned FrankXie05 Dec 27, 2022

njakob commented Dec 27, 2022

View reviewed changes

njakob marked this pull request as ready for review December 27, 2022 01:43

FrankXie05 added the category:port-bug The issue is with a library, which is something the port should already support label Dec 27, 2022

github-actions bot previously approved these changes Dec 27, 2022

View reviewed changes

FrankXie05 previously approved these changes Dec 27, 2022

View reviewed changes

FrankXie05 added the info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. label Dec 27, 2022

autoantwort mentioned this pull request Dec 27, 2022

[vcpkg|vcpkg-tool] Remove baseline.json from repo #27455

Closed

njakob mentioned this pull request Dec 28, 2022

[staticjson] add new port. #28578

Merged

vicroms requested changes Jan 2, 2023

View reviewed changes

njakob added 3 commits January 2, 2023 14:50

[libharu] Fix incorrect version git tree

3411d78

[libopensp] Fix incorrect version git tree

d9de7af

[minizip] Fix incorrect version git tree

279304c

njakob added 3 commits January 2, 2023 14:50

[smf] Remove incorrect version

df28294

[vlpp] Remove incorrect version

bebc5c2

[vulkan-headers] Remove incorrect version

c18d3e4

njakob dismissed stale reviews from FrankXie05 and github-actions[bot] via c18d3e4 January 2, 2023 14:04

njakob force-pushed the fix-versions branch from 7955555 to c18d3e4 Compare January 2, 2023 14:04

njakob commented Jan 2, 2023

View reviewed changes

njakob requested a review from vicroms January 2, 2023 14:17

vicroms approved these changes Jan 4, 2023

View reviewed changes

vicroms merged commit 4548ef8 into microsoft:master Jan 4, 2023

github-actions bot approved these changes Jan 4, 2023

View reviewed changes

njakob mentioned this pull request Mar 9, 2025

[restc-cpp] Fix incorrect version #44264

Merged

7 tasks

Conversation

njakob commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does your PR fix?

Which triplets are supported/not supported? Have you updated the CI baseline?

Does your PR follow the maintainer guide?

If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

Uh oh!

njakob commented Dec 27, 2022

Uh oh!

njakob Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njakob Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njakob Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thomas1664 commented Dec 27, 2022

Uh oh!

dg0yt commented Dec 27, 2022

Uh oh!

njakob commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

njakob commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vicroms commented Jan 2, 2023

Uh oh!

vicroms left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njakob left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vicroms commented Jan 4, 2023

Uh oh!

gharwo commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vicroms commented Jan 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dg0yt commented Jan 11, 2023

Uh oh!

gharwo commented Jan 12, 2023 via email

Uh oh!

gharwo commented Jan 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

njakob commented Dec 27, 2022 •

edited

Loading

If you have added/updated a port: Have you run `./vcpkg x-add-version --all` and committed the result?

njakob Dec 27, 2022 •

edited

Loading

njakob Dec 27, 2022 •

edited

Loading

njakob Dec 27, 2022 •

edited

Loading

njakob commented Dec 27, 2022 •

edited

Loading

njakob commented Dec 27, 2022 •

edited

Loading

gharwo commented Jan 11, 2023 •

edited

Loading

vicroms commented Jan 11, 2023 •

edited

Loading