ci: Switch remaining Linux tasks to self-hosted #21652

maflcko · 2021-04-11T11:16:08Z

Cirrus CI will be capping the free compute soon. For now, switch more tasks to persistent worker, as recommended by Cirrus CI.

(See slightly related discussion in #28098)

DrahtBot · 2021-04-11T15:24:10Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	dergoegge, pinheadmz, hebasto

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#28210 (build: Bump minimum supported Clang to clang-13 by MarcoFalke)
#28173 (ci: Run Windows native task on GitHub Actions by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

dergoegge

ACK fa8e89d

pinheadmz · 2023-08-22T17:09:34Z

concept ACK fa8e89d

Questions:

Is Win64 native still running on cirrus? The code changes look like it shouldnt be but it has a warning:

Monthly free compute limit exceeded and will be limited next month!

The MSan task took 2.5 hours! yow. Is the bottleneck there just hardware?

maflcko · 2023-08-23T07:44:53Z

Is Win64 native still running on cirrus? The code changes look like it shouldnt be but it has a warning:

Yes, sorry for the confusion. This is only about Linux. I've adjusted the title. There is another pull about the msvc on Windows.

maflcko · 2023-08-23T07:45:53Z

The MSan task took 2.5 hours! yow. Is the bottleneck there just hardware?

On the first run it will build llvm/clang/msan + depends + Bitcoin Core on a fresh cache. See also the comment in the cirrus yaml.

hebasto

ACK fa8e89d.

Observing very low Ccache hit rate. For example, in https://cirrus-ci.com/task/6728143201894400:

ccache version 3.7.7
cache directory                     /tmp/ccache_dir
primary config                      /tmp/ccache_dir/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Aug 18 17:06:37 2023
cache hit (direct)                   110
cache hit (preprocessed)              15
cache miss                           737
cache hit rate                     14.50 %
called for link                       12
cleanups performed                     0
files in cache                      1489
cache size                          33.6 MB
max cache size                     200.0 MB

maflcko · 2023-08-23T10:35:09Z

I wonder why the hit rate is non-zero. On the first run, the cache is empty, obviously. Maybe a leftover ccache from a previous push to this pull request?

hebasto · 2023-08-23T10:38:09Z

@MarcoFalke

I wonder why the hit rate is non-zero. On the first run, the cache is empty, obviously. Maybe a leftover ccache from a previous push to this pull request?

Can you please re-run all Cirrus jobs to test caching facilities?

maflcko · 2023-08-23T10:51:10Z

Can you please re-run all Cirrus jobs to test caching facilities?

There are multiple workers per label type and multiple tasks using the same label type. Thus, it will take many re-runs to populate the initial cache on all workers.

maflcko · 2023-08-23T10:52:40Z

I could limit the label type to each task, but that would be micro-managment. And free resources on one label type can not be used on another label type.

hebasto · 2023-08-23T10:53:28Z

Can you please re-run all Cirrus jobs to test caching facilities?

There are multiple workers per label type and multiple tasks using the same label type. Thus, it will take many re-runs to populate the initial cache on all workers.

Fair enough. We can observe caching quality in-progress later.

hebasto · 2023-08-23T10:56:30Z

There are multiple workers per label type and multiple tasks using the same label type.

Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

maflcko · 2023-08-23T11:02:15Z

Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

hebasto · 2023-08-23T11:12:24Z

Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

IIUC, multiple tasks share the same CCACHE_DIR. If they use different compilers or different compiler flags, they need more space to do not purge cached items from another task, no?

maflcko · 2023-08-23T11:31:19Z

Sounds like we need to increase CCACHE_MAXSIZE to cope such a usage, no?

Why? Each task has its own name (space). The CI is now doing exactly what happens when you run the CI locally.

IIUC, multiple tasks share the same CCACHE_DIR. If they use different compilers or different compiler flags, they need more space to do not purge cached items from another task, no?

No, and as I said this is unrelated to the changes here. If there is a bug in the CI system when running locally, it should be fixed separately, not as part of this pull.

See also:

$ git grep _ccache ci
ci/test/04_install.sh:  docker volume create "${CONTAINER_NAME}_ccache" || true
ci/test/04_install.sh:                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \

hebasto · 2023-08-23T13:30:55Z

https://github.com/bitcoin/bitcoin/runs/16142913468 -- timeouts. Probably unrelated...

maflcko · 2023-08-23T13:35:56Z

yeah, the blob is missing.

Fetching xcb-proto-1.15.2.tar.xz from https://bitcoincore.org/depends-sources
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0  222k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404

It would be good to switch the mirror URL to something that supports ipv6 and is maintained. Hint: #17704

maflcko · 2023-08-25T12:12:49Z

Apart from the blob missing, this seems to be an odd issue, because it works outside of podman and inside podman, any other site works as well:

# podman run --rm -it ubuntu:mantic  bash -c  'apt update && apt install curl -y && curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab http://xorg.freedesktop.org/archive/individual/proto/xcb-proto-1.15.2.tar.xz ' 

curl: (28) Failed to connect to xorg.freedesktop.org port 80 after 3001 ms: Timeout was reached

# podman run --rm -it ubuntu:mantic  bash -c  'apt update && apt install curl -y && curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab https://drahtbot.space/depends_download_fallback/xcb-proto-1.15.2.tar.xz '

100  144k  100  144k    0     0   614k      0 --:--:-- --:--:-- --:--:--  612k

# curl --location --fail  --connect-timeout 3 --retry 0  -o /tmp/ab http://xorg.freedesktop.org/archive/individual/proto/xcb-proto-1.15.2.tar.xz

100  144k  100  144k    0     0   118k      0  0:00:01  0:00:01 --:--:--  208k

See https://serverfault.com/questions/1108095/docker-container-unable-to-connect-to-one-specific-external-ip-yet-host-can

fanquake added the Tests label Apr 11, 2021

maflcko force-pushed the 2104-ciMoreSelf branch from 12ddc12 to 2f3552f Compare April 11, 2021 12:19

DrahtBot mentioned this pull request Apr 11, 2021

Use std::filesystem. Remove Boost Filesystem & System #20744

Merged

maflcko force-pushed the 2104-ciMoreSelf branch 2 times, most recently from 9dd511a to a497f98 Compare April 17, 2021 08:53

DrahtBot mentioned this pull request Apr 20, 2021

test: Use clang-12 for ASAN, Add missing suppression #21738

Merged

maflcko force-pushed the 2104-ciMoreSelf branch from a497f98 to 1c04196 Compare June 4, 2021 14:11

DrahtBot mentioned this pull request Jun 24, 2021

[TESTBED][NO-MERGE][POC] Use std::filesystem. Remove Boost Filesystem & System #22336

Closed

This was referenced Aug 16, 2021

ci: Invalidate depends caches when sources have been changed #22710

Merged

build, qt: Add Wayland support for Linux builds with depends #22708

Closed

DrahtBot added the Needs rebase label Dec 13, 2021

maflcko closed this Dec 27, 2021

maflcko deleted the 2104-ciMoreSelf branch December 27, 2021 19:31

bitcoin locked and limited conversation to collaborators Dec 27, 2022

bitcoin unlocked this conversation Jul 24, 2023

maflcko restored the 2104-ciMoreSelf branch July 24, 2023 08:49

maflcko changed the title ~~[WIP NOMERGE DRAFT] ci: Switch more tasks to self-hosted~~ ci: Switch more tasks to self-hosted Jul 24, 2023

maflcko reopened this Jul 24, 2023

maflcko force-pushed the 2104-ciMoreSelf branch 2 times, most recently from fa339d8 to fa28b13 Compare July 24, 2023 09:49

DrahtBot added CI failed and removed Needs rebase labels Jul 24, 2023

This was referenced Jul 24, 2023

ci: Use qemu-user through container engine #28087

Merged

ci: Move ASan USDT to persistent_worker #28161

Merged

maflcko force-pushed the 2104-ciMoreSelf branch 2 times, most recently from 411b023 to 848fbc9 Compare July 29, 2023 14:56

DrahtBot mentioned this pull request Jul 29, 2023

util: Replace std::filesystem with util/fs.h #28076

Merged

maflcko force-pushed the 2104-ciMoreSelf branch 2 times, most recently from beb0b0e to b83e83c Compare July 30, 2023 08:21

maflcko added this to the 26.0 milestone Aug 18, 2023

DrahtBot removed the CI failed label Aug 18, 2023

maflcko requested review from hebasto and pinheadmz August 21, 2023 10:08

DrahtBot mentioned this pull request Aug 21, 2023

ci: Run Windows native task on GitHub Actions #28173

Merged

maflcko changed the title ~~ci: Switch more tasks to self-hosted~~ ci: Switch remaining tasks to self-hosted Aug 22, 2023

dergoegge approved these changes Aug 22, 2023

View reviewed changes

DrahtBot removed the request for review from pinheadmz August 22, 2023 17:09

maflcko changed the title ~~ci: Switch remaining tasks to self-hosted~~ ci: Switch remaining Linux tasks to self-hosted Aug 23, 2023

maflcko requested a review from fanquake August 23, 2023 07:49

hebasto approved these changes Aug 23, 2023

View reviewed changes

fanquake merged commit 33da5d0 into bitcoin:master Aug 23, 2023

maflcko deleted the 2104-ciMoreSelf branch August 23, 2023 13:28

hebasto mentioned this pull request Aug 23, 2023

ci: Add missing docker.io prefix for native tasks to CI_IMAGE_NAME_TAG #28330

Merged

bitcoin locked and limited conversation to collaborators Aug 24, 2024

ci: Switch remaining Linux tasks to self-hosted #21652

ci: Switch remaining Linux tasks to self-hosted #21652

Conversation

maflcko commented Apr 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews

Conflicts

Uh oh!

dergoegge left a comment

Choose a reason for hiding this comment

Uh oh!

pinheadmz commented Aug 22, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

hebasto left a comment

Choose a reason for hiding this comment

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

hebasto commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

hebasto commented Aug 23, 2023

Uh oh!

hebasto commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

hebasto commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

hebasto commented Aug 23, 2023

Uh oh!

maflcko commented Aug 23, 2023

Uh oh!

maflcko commented Aug 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

maflcko commented Apr 11, 2021 •

edited

Loading

DrahtBot commented Apr 11, 2021 •

edited

Loading