Skip to content

ci: Print ccache stats, add pip cache, and cleanups #18581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
maflcko merged 3 commits into from
Apr 13, 2020
Merged

ci: Print ccache stats, add pip cache, and cleanups #18581

maflcko merged 3 commits into from
Apr 13, 2020

Conversation

@hebasto
Copy link
Member

@hebasto hebasto commented Apr 10, 2020 •
edited
Loading

The Travis pip cache is free and saves a dozen of seconds :)

Here are some excerpts from the Travis logs with ccache statistics (I found useful):

  1. Arm64
ccache version 3.6
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 18:21:06 2020
stats zeroed                        Fri Apr 10 18:03:04 2020
cache hit (direct)                   156
cache hit (preprocessed)             198
cache miss                           143
cache hit rate                     71.23 %
called for link                        8
cleanups performed                     9
files in cache                      1255
cache size                          80.6 MB
max cache size                     100.0 MB
  1. s390x
ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:15:35 2020
cache hit (direct)                   115
cache hit (preprocessed)             163
cache miss                            94
cache hit rate                     74.73 %
called for link                        6
cleanups performed                     0
files in cache                       804
cache size                         304.8 MB
max cache size                       5.0 GB
  1. Win64
ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:38:07 2020
cache hit (direct)                   147
cache hit (preprocessed)             199
cache miss                           138
cache hit rate                     71.49 %
called for link                        7
cleanups performed                     7
files in cache                      1242
cache size                          87.0 MB
max cache size                     100.0 MB
  1. CentOS 7
ccache version 3.7.7
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 17:45:59 2020
stats zeroed                        Fri Apr 10 17:34:27 2020
cache hit (direct)                   152
cache hit (preprocessed)             200
cache miss                           144
cache hit rate                     70.97 %
called for link                        8
cleanups performed                     4
files in cache                      1352
cache size                          86.1 MB
max cache size                     100.0 MB
  1. bionic
ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:21:10 2020
cache hit (direct)                   136
cache hit (preprocessed)              35
cache miss                           330
cache hit rate                     34.13 %
called for link                        8
cleanups performed                    18
files in cache                      1302
cache size                          83.8 MB
max cache size                     100.0 MB
  1. xenial
ccache version 3.2.4
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
cache hit (direct)                   161
cache hit (preprocessed)             174
cache miss                            95
called for link                        7
files in cache                      3411
cache size                           1.1 GB
max cache size                       5.0 GB
  1. focal
ccache version 3.7.7
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 17:35:57 2020
stats zeroed                        Fri Apr 10 17:21:32 2020
cache hit (direct)                    78
cache hit (preprocessed)             125
cache miss                           107
cache hit rate                     65.48 %
called for link                      120
cleanups performed                     0
files in cache                      6218
cache size                           1.8 GB
max cache size                       5.0 GB
  1. macOS 10.12
ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:47:19 2020
cache hit (direct)                    28
cache hit (preprocessed)               1
cache miss                           469
cache hit rate                      5.82 %
called for link                        8
cleanups performed                    17
files in cache                      1946
cache size                          82.6 MB
max cache size                     100.0 MB
  1. macOS 10.14 native
ccache version 3.7.8
cache directory                     /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /usr/local/Cellar/ccache/3.7.8/etc/ccache.conf
stats updated                       Fri Apr 10 17:38:44 2020
stats zeroed                        Fri Apr 10 17:22:36 2020
cache hit (direct)                   213
cache hit (preprocessed)             293
cache miss                           144
cache hit rate                     77.85 %
called for link                       11
called for preprocessing              56
compile failed                        30
preprocessor error                    56
bad compiler arguments                14
autoconf compile/link                 77
no input file                         66
cleanups performed                     0
files in cache                      1564
cache size                         284.6 MB
max cache size                       5.0 GB

@fanquake fanquake added the Tests label Apr 10, 2020
@promag
Copy link
Contributor

promag commented Apr 10, 2020

ACK, looks good to me.

maflcko
maflcko reviewed Apr 10, 2020
View reviewed changes
maflcko
maflcko reviewed Apr 10, 2020
View reviewed changes
Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Telling from the version, it looks like you are executing ccache on the host, not in the docker

@maflcko
Copy link
Member

maflcko commented Apr 10, 2020

Could update OP with the new ccache stats, since they are now stale?

@hebasto
Copy link
Member Author

hebasto commented Apr 10, 2020

Could update OP with the new ccache stats, since they are now stale?

Sure. Waiting for the Travis build in the bitcoin repo :)

@hebasto
Copy link
Member Author

hebasto commented Apr 10, 2020

Updated b1f7d35 -> 73f8303 (pr18581.01 -> pr18581.02, diff):

Another use:

ci/test/04_install.sh:env | grep -E '^(BITCOIN_CONFIG|BASE_|QEMU_|CCACHE_|WINEDEBUG|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|TEST_PREVIOUS_RELEASES|PREVIOUS_RELEASES_DIR)' | tee /tmp/env

Telling from the version, it looks like you are executing ccache on the host, not in the docker

@hebasto
Copy link
Member Author

hebasto commented Apr 10, 2020

Could update OP with the new ccache stats, since they are now stale?

Updated.

@maflcko maflcko merged commit 6ef45bc into bitcoin:master Apr 13, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 13, 2020
Merge bitcoin#18581: ci: Print ccache stats, add pip cache, and cleanups
a0875dc 
73f8303 ci: Add pip cache (Hennadii Stepanov)
6d48066 ci: Print ccache statistics summary (Hennadii Stepanov)
a6eea71 ci: Drop unused WINEDEBUG variable (Hennadii Stepanov)

Pull request description:

  The Travis [pip cache](https://docs.travis-ci.com/user/caching/#pip-cache) is free and saves a dozen of seconds :)

  Here are some excerpts from the Travis logs with `ccache` statistics (I found useful):

  2) [Arm64](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507749)
  ```
  ccache version 3.6
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 18:21:06 2020
  stats zeroed                        Fri Apr 10 18:03:04 2020
  cache hit (direct)                   156
  cache hit (preprocessed)             198
  cache miss                           143
  cache hit rate                     71.23 %
  called for link                        8
  cleanups performed                     9
  files in cache                      1255
  cache size                          80.6 MB
  max cache size                     100.0 MB
  ```
  3) [s390x](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507750)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:15:35 2020
  cache hit (direct)                   115
  cache hit (preprocessed)             163
  cache miss                            94
  cache hit rate                     74.73 %
  called for link                        6
  cleanups performed                     0
  files in cache                       804
  cache size                         304.8 MB
  max cache size                       5.0 GB
  ```
  4) [Win64](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507751)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:38:07 2020
  cache hit (direct)                   147
  cache hit (preprocessed)             199
  cache miss                           138
  cache hit rate                     71.49 %
  called for link                        7
  cleanups performed                     7
  files in cache                      1242
  cache size                          87.0 MB
  max cache size                     100.0 MB
  ```
  5) [CentOS 7](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507752)
  ```
  ccache version 3.7.7
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 17:45:59 2020
  stats zeroed                        Fri Apr 10 17:34:27 2020
  cache hit (direct)                   152
  cache hit (preprocessed)             200
  cache miss                           144
  cache hit rate                     70.97 %
  called for link                        8
  cleanups performed                     4
  files in cache                      1352
  cache size                          86.1 MB
  max cache size                     100.0 MB
  ```
  6) [bionic](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507753)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:21:10 2020
  cache hit (direct)                   136
  cache hit (preprocessed)              35
  cache miss                           330
  cache hit rate                     34.13 %
  called for link                        8
  cleanups performed                    18
  files in cache                      1302
  cache size                          83.8 MB
  max cache size                     100.0 MB
  ```
  7) [xenial](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507754)
  ```
  ccache version 3.2.4
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  cache hit (direct)                   161
  cache hit (preprocessed)             174
  cache miss                            95
  called for link                        7
  files in cache                      3411
  cache size                           1.1 GB
  max cache size                       5.0 GB
  ```
  10) [focal](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507757)
  ```
  ccache version 3.7.7
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 17:35:57 2020
  stats zeroed                        Fri Apr 10 17:21:32 2020
  cache hit (direct)                    78
  cache hit (preprocessed)             125
  cache miss                           107
  cache hit rate                     65.48 %
  called for link                      120
  cleanups performed                     0
  files in cache                      6218
  cache size                           1.8 GB
  max cache size                       5.0 GB
  ```
  14) [macOS 10.12](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507761)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:47:19 2020
  cache hit (direct)                    28
  cache hit (preprocessed)               1
  cache miss                           469
  cache hit rate                      5.82 %
  called for link                        8
  cleanups performed                    17
  files in cache                      1946
  cache size                          82.6 MB
  max cache size                     100.0 MB
  ```
  15) [macOS 10.14 native](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507762)
  ```
  ccache version 3.7.8
  cache directory                     /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /usr/local/Cellar/ccache/3.7.8/etc/ccache.conf
  stats updated                       Fri Apr 10 17:38:44 2020
  stats zeroed                        Fri Apr 10 17:22:36 2020
  cache hit (direct)                   213
  cache hit (preprocessed)             293
  cache miss                           144
  cache hit rate                     77.85 %
  called for link                       11
  called for preprocessing              56
  compile failed                        30
  preprocessor error                    56
  bad compiler arguments                14
  autoconf compile/link                 77
  no input file                         66
  cleanups performed                     0
  files in cache                      1564
  cache size                         284.6 MB
  max cache size                       5.0 GB
  ```

Top commit has no ACKs.

Tree-SHA512: f9f2c6d72680bafc7a982efc04dc37ab9bba77ffd2a4cbbc56bb0e86c5c3501d57aa2f0ebeac0113e9d8daac1ee4eac1eddf74c1172c267b37ee2d9324366c8a
@hebasto hebasto deleted the 20200409-ci-plus branch April 14, 2020 04:51
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@maflcko maflcko maflcko left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hebasto @promag @maflcko @fanquake