Lookup latest versions of packages on illumos #10215
Conversation
| wget "$BaseUrl"/"$package".tgz | ||
| ar -x "$package".tgz | ||
| tar --skip-old-files -xzf "$package".tmp.tgz -C "$__RootfsDir" 2>/dev/null | ||
| tar --skip-old-files -xzf "$package".tmp.tg* -C "$__RootfsDir" 2>/dev/null |
There was a problem hiding this comment.
For some reason, icu in trunk branch has .tmp.tg instead of .tmp.tgz, while other packages have .tmp.tgz. Therefore I have used the glob.
ps - this script is only used in docker builder (https://github.com/dotnet/dotnet-buildtools-prereqs-docker), so it doesn't need to be resilient or portable).
There was a problem hiding this comment.
The odd bit is
This one is giving me a tmp.tgz at https://pkgsrc.joyent.com/packages/SmartOS/trunk/x86_64/All/icu-71.1.tgz in 7zip. Do you know what version it was fetching before? Looks like ar is seeing the inner stream name as .tg
hoyosjs
left a comment
hoyosjs
left a comment
There was a problem hiding this comment.
Given that Illumos is an unsupported OS on .NET, I am not sure what customers depend on this or who validates that the resulting runtime works correctly. The change looks sane, but have you had a chance to check it doesn't regress support? For example, I see openssl1.1.1n. I see it being potentially brittle with there being 2 mit-krb5 prefixed packages in the repo. The main one is 1.18.4. There's also zlib 1.2.12 and icu 71 which we should support.
eng/common/cross/build-rootfs.sh
Outdated
| for package in "${array[@]}"; do | ||
| echo "Installing $package..." | ||
| echo "Installing '$package'" | ||
| package="$(grep ">$package" All | head -1 | sed -En 's/.*href="(.*)\.tgz".*/\1/p')" |
There was a problem hiding this comment.
grep maybe with $package-[0-9] to avoid things like mit-krb5-app-<ver> over mit-krb5-<ver>
There was a problem hiding this comment.
head -1 is there to filter out the unneeded ones. Can use [0-9]. End result is the same..
There was a problem hiding this comment.
I meant mostly that order of entries maters with the head scheme. Mostly wanted to be resilient to that.
So far I am doing it. |
4 participants
We were using 2020Q1 repository to build illumos rootfs, which has older version of packages. To obtain newer versions, we need to either update the fixed repository reference in the URL each time we want a newer version, or use
trunkas a floating reference and lookup the latest version of packages available in the manifest.This PR switches the repository branch to
trunkand looks up the package version by scraping the index page (HTML) and finding the latest version of archives we need.I have used this approach for two reasons:
trunkbranch.cc @GrabYourPitchforks