Skip to content

Conversation

@sipsorcery
Copy link
Contributor

@sipsorcery sipsorcery commented Nov 26, 2020

The motivation for this PR is twofold:

  1. Update the Qt binaries used by the appveyor CI job after a recent update to Visual Studio 2019 used in the Appveyor build image resulted in ABI incompatibilities,

  2. Remove optimisations and debug information from the Bitcoin Core Release msvc build to reduce the chance of future ABI incompatibility issues for future Visual Studio updates.

The changes made in this PR are:

  • Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.
  • Adjusted msvc compiler and linker settings to remove optimisations and debug information generation to help avoid future ABI issues on Visual Studio updates.
  • Tidied up debug and release configuration blocks in common project file to avoid duplication.
  • Updated appveyor config to use latest Visual Studio 2019 image.
  • Bumped vcpkg version to tag 2020.11-1 for binary caching feature*.

See #20392 for related discussion.

*Binary caching is a new vcpkg feature that allows dependency caching. This PR is not using the feature but by updating the vcpkg version it means it can be optionally used by other contributors in their own Appveyor configs. By caching the vcpkg dependencies using this feature my build times are reduced by approx 10 minutes.

@sipsorcery sipsorcery marked this pull request as draft November 29, 2020 21:10
…at are causing sporadic ABI issues on Visual Studio updates.

Tidied up debug and release configuration blocks in common project file to avoid duplication.

Updated appveyor config to use latest Visual Studio 2019 image.

Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.

Bumped vcpkg version to tag '2020.11-1' for binary caching feature.

See bitcoin#20392 for related discussion.
@sipsorcery sipsorcery changed the title WIP: AppVeyor CI fixes in preparation for next image bump AppVeyor CI fixes for Visual Studio 2019 16.8.1 image Dec 2, 2020
@sipsorcery sipsorcery marked this pull request as ready for review December 2, 2020 15:41
@sipsorcery
Copy link
Contributor Author

@MarcoFalke as a follow on to your comment here this PR should reduce the appveyor build times by approximately 20 minutes. From around 65 to around 45.

The reduction is down to building the Qt libraries without msvc compiler optimisations. As mentioned above, removing the optimisations was to help with ABI compatibility. The build time reduction was a welcome side effect.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 8b99e60, tested in my personal repo; did not check changes in the build_msvc/ directory.

AppVeyor finished build in 46 min 31 sec.

@sipsorcery
Copy link
Contributor Author

@hebasto thanks for checking. If you have time could you do me a favour (it's a quick set up) and check the vcpkg binary caching feature on your appveyor account?

If so the steps are:

appveyor_env

The test is to see whether your build job pulls the pre-built vcpkg dependencies from my Azure devops account. The reason it may not is the way vcpkg generates the hash to determine whether the pre-built dependency is a match or not.

If the pre-built dependencies are used your build time should be closer to 30 minutes.

@maflcko maflcko changed the title AppVeyor CI fixes for Visual Studio 2019 16.8.1 image ci: AppVeyor fixes for Visual Studio 2019 16.8.1 image Dec 3, 2020
.appveyor.yml Outdated
git pull origin master > $null
git -c advice.detachedHead=false checkout $env:VCPKG_COMMIT_ID
git -c advice.detachedHead=false checkout $env:VCPKG_TAG
git pull origin $env:VCPKG_TAG
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it first checkout the tag and then pull it? Shouldn't it be the other way round?

@maflcko
Copy link
Member

maflcko commented Dec 3, 2020

Concept ACK 2c69381

Haven't reviewed the code, nor have I tested if building locally on Windows still works

@hebasto
Copy link
Member

hebasto commented Dec 3, 2020

@sipsorcery

@hebasto thanks for checking. If you have time could you do me a favour (it's a quick set up) and check the vcpkg binary caching feature on your appveyor account?

Configured and checked: re-build has finished "in 30 min".

@sipsorcery
Copy link
Contributor Author

Configured and checked: re-build has finished "in 30 min".

Perfect, thanks!

@laanwj
Copy link
Member

laanwj commented Dec 3, 2020

Concept ACK 2c69381
Checked that this only touches the MSVC files.

@laanwj laanwj merged commit 3fa6a9f into bitcoin:master Dec 3, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Dec 3, 2020
… image

2c69381 Removed redundant git pull from appveyor config. (Aaron Clauson)
8b99e60 Adjusted msvc compiler and linker settings to remove optimisations that are causing sporadic ABI issues on Visual Studio updates. (Aaron Clauson)

Pull request description:

  The motivation for this PR is twofold:

    1. Update the Qt binaries used by the appveyor CI job after a recent update to Visual Studio 2019 used in the Appveyor build image resulted in ABI incompatibilities,

    2. Remove optimisations and debug information from the Bitcoin Core `Release` msvc build to reduce the chance of future ABI incompatibility issues for future Visual Studio updates.

  The changes made in this PR are:

   - Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.
   - Adjusted msvc compiler and linker settings to remove optimisations and debug information generation to help avoid future ABI issues on Visual Studio updates.
   - Tidied up debug and release configuration blocks in common project file to avoid duplication.
   - Updated appveyor config to use latest Visual Studio 2019 image.
   - Bumped vcpkg version to tag `2020.11-1` for binary caching feature*.

  See bitcoin#20392 for related discussion.

  *Binary caching is a new [vcpkg feature](https://github.com/microsoft/vcpkg/blob/master/docs/specifications/binarycaching.md) that allows dependency caching. This PR is not using the feature but by updating the vcpkg version it means it can be optionally used by other contributors in their own Appveyor configs. By caching the vcpkg dependencies using this feature my build times are reduced by approx 10 minutes.

ACKs for top commit:
  MarcoFalke:
    Concept ACK 2c69381
  laanwj:
    Concept ACK 2c69381

Tree-SHA512: 372f5b8b3b5f7e56b78045f9fc06a22fd9f5366d06e99e2f1eaad6d741680da74d0e6371e7bc580cb41f4d6696b2101b950167309d33e98242578b458ace813a
@sipsorcery
Copy link
Contributor Author

@MarcoFalke if you think it's a good idea you can add the environment setting below to the Bitcoin Core appveyor job to further reduce the build time by between 10 and 15 minutes. As mentioned above this setting is better put in the Appveyor build job settings rather than in the yml file.

VCPKG_BINARY_SOURCES nuget,https://aaronrc.pkgs.visualstudio.com/bitcoin/_packaging/vcpkg-deps/nuget/v3/index.json,read

Unlike previous attempts at caching vcpkg dependencies this one should be more reliable:

  • If a particular package doesn't have a matching nuget package it will be built from source,
  • If a new dependency is added to the vcpkg.json file it will be built from source,
  • If the Appveyor image gets updated the version hash of the vcpkg dependency changes and will cause it to be built from source,
  • The stored nuget packages are treated as readonly with the above environment variable. It requires a manual script to be run to add new packages.

I've created a custom script to run as a standalone appveyor build job which builds the dependencies in vcpkg.json and stores the results as nuget packages in my Azure site. It would be trivial to switch to a different nuget storage location or even standard file storage if required. It's probably worth verifying that this approach does prove more reliable than previous attempts before expending too much effort.

maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Dec 16, 2020
…at are causing sporadic ABI issues on Visual Studio updates.

Tidied up debug and release configuration blocks in common project file to avoid duplication.

Updated appveyor config to use latest Visual Studio 2019 image.

Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.

Bumped vcpkg version to tag '2020.11-1' for binary caching feature.

See bitcoin#20392 for related discussion.

Github-Pull: bitcoin#20506
Rebased-From: 8b99e60
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Dec 16, 2020
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Dec 16, 2020
…at are causing sporadic ABI issues on Visual Studio updates.

Tidied up debug and release configuration blocks in common project file to avoid duplication.

Updated appveyor config to use latest Visual Studio 2019 image.

Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.

Bumped vcpkg version to tag '2020.11-1' for binary caching feature.

See bitcoin#20392 for related discussion.

Github-Pull: bitcoin#20506
Rebased-From: 8b99e60
hebasto pushed a commit to hebasto/bitcoin that referenced this pull request Dec 16, 2020
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants