Skip to content

[CI] Cache also packages#51270

Merged
giordano merged 2 commits intomasterfrom
mg/cache-pkgs
Dec 27, 2021
Merged

[CI] Cache also packages#51270
giordano merged 2 commits intomasterfrom
mg/cache-pkgs

Conversation

@giordano
Copy link
Copy Markdown
Member

@giordano giordano commented Dec 27, 2021

I'm seeing that also the instantiation step

General/.github/workflows/ci.yml

Line 59 in 2138ce3

- run: .ci/instantiate.sh

often gets stuck when there are connections problems. See for example https://github.com/JuliaRegistries/General/runs/4641542758?check_suite_focus=true

   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Installed Artifacts ────────────────── v1.3.0
  Installed LibCURL_jll ──────────────── v7.70.0+2
  Installed MozillaCACerts_jll ───────── v2021.1.19+0
  Installed LazyArtifacts ────────────── v1.3.0
  Installed Downloads ────────────────── v1.6.0
  Installed NetworkOptions ───────────── v1.2.0
  Installed Mocking ──────────────────── v0.7.3
  Installed RegistryCI ───────────────── v7.1.7
  Installed Tar ──────────────────────── v1.9.2
  Installed IniFile ──────────────────── v0.5.0
  Installed StatsAPI ─────────────────── v1.1.0
  Installed MbedTLS ──────────────────── v1.0.3
  Installed libsodium_jll ────────────── v1.0.19+0
  Installed VisualStringDistances ────── v0.1.1
  Installed nghttp2_jll ──────────────── v1.40.0+2
  Installed JLLWrappers ──────────────── v1.3.0
  Installed ExprTools ────────────────── v0.1.6
  Installed URIs ─────────────────────── v1.3.0
  Installed HTTP ─────────────────────── v0.9.17
  Installed StaticArrays ─────────────── v1.2.13
  Installed licensecheck_jll ─────────── v0.3.1+1
  Installed Parsers ──────────────────── v2.1.3
  Installed Preferences ──────────────── v1.2.2
  Installed Zlib_jll ─────────────────── v1.2.11+18
  Installed InlineStrings ────────────── v1.1.1
  Installed LibSSH2_jll ──────────────── v1.9.0+3
  Installed TimeZones ────────────────── v1.6.2
  Installed Compat ───────────────────── v3.41.0
  Installed GitHub ───────────────────── v5.7.0
  Installed RegistryTools ────────────── v1.7.0
  Installed TOML ─────────────────────── v1.0.3
  Installed SodiumSeal ───────────────── v0.1.1
  Installed Distances ────────────────── v0.10.7
  Installed JSON ─────────────────────── v0.21.2
  Installed LicenseCheck ─────────────── v0.2.2
  Installed RecipesBase ──────────────── v1.2.1
  Installed MbedTLS_jll ──────────────── v2.16.8+1
    Cloning [88034a9c-02f8-509d-84a9-84ec65e18404] StringDistances from matthieugomez/StringDistances.jl.git
25l25h25l    Fetching: [>                                        ]  0.0 %
  Installed StringDistances ──────────── v0.10.0
    Cloning [15f4f7f2-30c1-5605-9d31-71845cf9641f] AutoHashEquals from andrewcooke/AutoHashEquals.jl.git
Error: The operation was canceled.

The fact that this is trying to clone the Git repository instead of downloading the snapshot is another indication we're struggling with downloads from GitHub (and git clones are also slow). caching packages in addition to artifacts should help with this step.

@giordano giordano added the CI continuous integration label Dec 27, 2021
@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

Hmmmm. Why is it trying to clone from GitHub? We're using the Pkg server, right?

(I know we are using the Git registry, but even when using the Git registry, Pkg should still try to download the packages from the Pkg server first, before trying GitHub, right?)

@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

Also, even if we clone all the packages as Git repos, it still shouldn't take 27 minutes.... so I think something else (or multiple somethings) is going on here.

@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

E.g. https://twitter.com/jflamy/status/1475299591687913473/photo/1

DilumAluthge
DilumAluthge approved these changes Dec 27, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@DilumAluthge DilumAluthge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, but I would like us to figure out the underlying issue (or issues).

@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

E.g. https://twitter.com/jflamy/status/1475299591687913473/photo/1

But again, slow GitHub doesn't explain why the Pkg client isn't first trying to download the packages from the Pkg server.

@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

@giordano On line 44, can you change - name: Cache artifacts to something more accurate, e.g. - name: Cache artifacts and packages or something like that?

I tried to make a "suggestion", but GitHub won't let me, since that line isn't modified by the PR.

@DilumAluthge
Copy link
Copy Markdown
Member

DilumAluthge commented Dec 27, 2021

At least there's finally an incident listed: https://www.githubstatus.com/incidents/gh0vvxtlj5v7

@giordano giordano temporarily deployed to stopwatch December 27, 2021 13:31 Inactive
@giordano
Copy link
Copy Markdown
Member Author

giordano commented Dec 27, 2021

As a confirmation this is working as intended, the instantiation step now takes 1 second, without output (because everything is already there): https://github.com/JuliaRegistries/General/runs/4642386103?check_suite_focus=true

@giordano giordano merged commit 3f3d657 into master Dec 27, 2021
@giordano giordano deleted the mg/cache-pkgs branch December 27, 2021 13:38
@giordano giordano mentioned this pull request Dec 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DilumAluthge DilumAluthge DilumAluthge approved these changes

Assignees

No one assigned

Labels

CI continuous integration

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@giordano @DilumAluthge