Skip to content

Inject FetchContent logic#105

Merged
bretbrownjr merged 5 commits intobemanproject:mainfrom
bretbrownjr:inject-fetch-content
Mar 5, 2025
Merged

Inject FetchContent logic#105
bretbrownjr merged 5 commits intobemanproject:mainfrom
bretbrownjr:inject-fetch-content

Conversation

@bretbrownjr
Copy link
Copy Markdown
Member

Problem

Given:

A sandbox environment is an environment where dependencies are all provided but access to the internet is not. Also take as a given that the build for this project happens in an environment that meets that description.

When:

Configuring this project with any of the provided approaches. For instance:

cmake -B build -S .

Then:

The configuration step hangs or fails, depending on the nature of the sandboxed environment.

Solution

Move to a find_package first implementation for the CMakeLists.txt of this project.

Using FetchContent is still trivial, using the new functionality in cmake/use-fetch-content.cmake as documented in the updates to README.md included in this commit.

@bretbrownjr
Copy link
Copy Markdown
Member Author

This is an alternative and mutually exclusive approach to #92.

Note that it uses yet another approach to letting the same repo use either FetchContent or find_package without hardcoding anything special in CMakeLists.txt. See https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#id11 and the documents it links for more on how "Dependency Providers" work.

This approach is more code compared to #92, but I would expect if we liked this approach, the entirety of the use-fetch-content.cmake file could spun off to a standalong utility project, perhaps inside https://github.com/bemanproject/infra. This seems to be an ergonomic gap in CMake, so analogous features could be proposed upstream to CMake itself assuming we get a usage experience we're happy with.

Another merit of this approach compared to #92 is that pinned dependency information is in legible JSON format, so automation (like a make update target) to update the pinned version of GoogleTest would be very possible compared to hardcoding that information in a CMakeLists.txt. Similarly, documentation would be relatively simpler and require no special CMake knowledge.

@bretbrownjr
Copy link
Copy Markdown
Member Author

On the failing CI checks, they are due to a missing googletest dependency, which isn't a large concern. It's not too difficult to have presets that prefer FetchContent on MacOS and Windows, pending selection of package management solutions on those architectures. Same for installing libgtest-dev on Ubuntu.

First, I'd like to hear whether this is a design we would like to pursue before investing time in troubleshooting CI failures.

Copy link
Copy Markdown
Member

@wusatosi wusatosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this too much complexity for supporting using local GTest package? Looks like you are trying to implement a package manager.

@bretbrownjr
Copy link
Copy Markdown
Member Author

bretbrownjr commented Jan 28, 2025

@wusatosi Using FetchContent is implementing dependency management in-repo, yes.

Using it explicitly is an incomplete approach to dependency management since it doesn't have the features (sandbox-friendliness, portability to existing packaging systems, trivial version bumping) that I'm looking for with this PR.

This PR demonstrates how find_package might be redirected to FetchContent calls. I don't expect this is a good long-term approach. As I state in the Discourse on this change, if we take this direction as a medium-term strategy, I expect the interesting CMake logic would be pulled out into another repo like https://github.com/bemanproject/infra.

Another approach to consider would be to use find_package explicitly but lean harder into package managers by supporting Conan with a conanfile.txt and/or supporting vcpkg with a vcpkg.json directly in this repo. One or both of those approaches could be used in CI, assumed in presets, etc.

FetchContent would still be possible for consumers of this project, but it would be an exercise for them to wire that up if they desired, perhaps with the same CMAKE_PROJECT_TOP_LEVEL_INCLUDES and dependency provider approach in the current revision of this PR.

@bretbrownjr
Copy link
Copy Markdown
Member Author

This latest commit fixes the linting checks, especially CMake formatting issues.

The remaining CI failures revolve around actually installing GTest. Those are going to require a consensus decision on how to resolve. Look out for another draft PR by me to demonstrate how to resolve that issue with a conanfile.txt.

Comment thread cmake/use-fetch-content.cmake
@bretbrownjr
Copy link
Copy Markdown
Member Author

I made a PR against the Beman Standard to land alongside this change that recommends find_package instead of FetchContent:
bemanproject/beman#98

@bretbrownjr bretbrownjr force-pushed the inject-fetch-content branch 7 times, most recently from 32a71f1 to 44d4fa5 Compare February 24, 2025 03:36
@bretbrownjr bretbrownjr marked this pull request as ready for review February 24, 2025 03:40
@bretbrownjr
Copy link
Copy Markdown
Member Author

Forgive the CI-driven-development, but the HEAD commit on this PR now works.

This is ready for review, and if acceptable, merging.

Comment thread .github/workflows/ci_tests.yml
Comment on lines -100 to +107
name: "Unit: ${{ matrix.platform.description }} ${{ matrix.cpp_version }} ${{ matrix.cmake_args.description }}"
name: "Unit:
${{ matrix.platform.description }}
${{ matrix.cpp_version }}
${{ matrix.cmake_args.description }}"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I edited this line and a few others like it to make the file easier to read, diff, and review. Hopefully everyone's OK with doing that as part of this PR.

-S .
-DCMAKE_CXX_STANDARD=${{ matrix.cpp_version }}
-DCMAKE_TOOLCHAIN_FILE="${{ matrix.platform.toolchain }}"
-DCMAKE_PROJECT_TOP_LEVEL_INCLUDES="./cmake/use-fetch-content.cmake"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me note since the formatting changes obscure the change a little. This argument referencing use-fetch-content.cmake is added. It makes this CI workflow use FetchContent.

Pay attention to the other cmake commands in this file as well. If you do not see a mention of use-fetch-content.cmake, the workflow uses find_package and system installed dependencies.

Copy link
Copy Markdown
Member

@wusatosi wusatosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config priority

Do we really want searching for local libgtest to be the default behavior?

For people hacking around beman or downstream paper/implementation authors, I don't think they will be more likely to care about offline run with libgtest installed than being able configure the project with one click.

I think the priority is off, it should be:
First: People with generic environment (cmake, gcc, a network connection and no libgtest) should have the ability to configure the project with one click.
Second: People that want to configure the project for offline configuration (e.g. they are so interested in the project that they bring the project on a plane or into their company) should have the option via an explicit parameter to do an offline build.

Need docs for downstream library authors

Missing docs for downstream library author that want to add a beman project (e.g. iterator-interface or execution) as sub-dependency. Do they have to do -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES="./cmake/use-fetch-content.cmake" all the time?

The internals of lockfile.json is not documented (why not dependencies.json btw), and it's not clear how offline functionality will interact with them. I think this is significant enough of a change to require documentation before merging.

Using fetch content shouldn't look like magic

I would prefer if we can switch this behavior with something like: BEMAN_DEPENDENCY_FETCH=LOCAL (as enum option with local, fetch. so we can add maybe vcpkg in the future).

This would be a lot more declarative and obvious to change for downstream than adding: CMAKE_PROJECT_TOP_LEVEL_INCLUDES="./cmake/use-fetch-content.cmake.

Comment thread .github/workflows/ci_tests.yml
Comment thread cmake/use-fetch-content.cmake
@bretbrownjr
Copy link
Copy Markdown
Member Author

Do we really want searching for local libgtest to be the default behavior?

We want find_package(GTest) to be the way we declare an external dependency, yes. This is the recommended and portable practice as discussed. FetchContent does not provide the portability or ergonomics we'd want to replicate across the other Beman libraries and the C++ ecosystem more broadly.

I would prefer if we can switch this behavior with something like: BEMAN_DEPENDENCY_FETCH=LOCAL (as enum option with local, fetch. so we can add maybe vcpkg in the future).

I'm opposed to implementing bits and pieces of a dependency management system in every Beman library. It makes more sense to move the whole community to supported and widely adopted technologies like Conan, vcpkg, and/or a number of other systems that already support fetching dependencies from the internet (or not) in all sorts of configurations.

I'll be clear about this. Downloading dependencies from github.com is supported by both Conan and vcpkg (and all the other options too!). It's really easy and covered by the introductory tutorials of both options. They both support building against a local cloned and modified of the googletest repo as well. We wouldn't have to develop and maintain tools and paragraphs of docs, just point our users at the most relevant sections, if that's required.

I would prefer if we can switch this behavior with something like: BEMAN_DEPENDENCY_FETCH=LOCAL (as enum option with local, fetch. so we can add maybe vcpkg in the future).

A BEMAN_DEPENDENCY_FETCH=conan or vcpkg wouldn't really work. Conan is already running a command like conan install before it invokes cmake. vpkg is incorporated via a CMake toolchain file named vcpkg.cmake.

One of the biggest problems with FetchContent is that it encourages conflation of build systems and dependency management systems. This wouldn't be too bad except for the fact that the dependency management features it realistically provides does not stack up well at all compared to alternatives (see the discussion below about lockfiles).

I also don't want users to type use-fetch-content.cmake long term. This is an intermediate technology until we come up with something better. The main goal of adding it in this PR is to provide some sort of stopgap and to demonstrate that we can make find_package(GTest) resolve to... well, whatever we want.

The internals of lockfile.json is not documented...

That's fair, and I can update the PR quickly to add some docs, though they are more developer documentation for maintainers of Exemplar than something I expect users will need to interact with. Meaning I'm not sure if I'll put them in a user-facing README.md or not.

I'm fairly flexible on the name, for what it's worth. If we can get consensus but only if the file is named examplar.lock.json or fetchcontent.dependencies.json or something, I can live with that.

To explain my reasoning, I chose the name "lockfile" because that's one term of art for files like these. If you ask your search engine or your favorite programmer-oriented LLM tool will turn up full explanations, I expect, but here's the Conan Lockfile documentation for a quick and relevant example

PS. By linking the Conan docs on the subject of dependency pinning, I'm throwing shade at FetchContent because Conan supports (and documents!) all sorts of dependency manipulation workflows and features that FetchContent really doesn't. And Conan is actually not exceptional in supporting this. On the contrary, FetchContent is notable for its limited built-in feature set.

@wusatosi
Copy link
Copy Markdown
Member

Cool! I don't and can't claim to know enough about CMake or package managers to provide more judgement, but as long as my concerns are seen I am happy to sit aside and learn by observation.

My only concern would be loss of one-click setup, I think a cheaty way would be to do this would be adding CMAKE_PROJECT_TOP_LEVEL_INCLUDES=./cmake/use-fetch-content.cmake to current CMake presets, and having presets that deal specificity with package management. Agh, only if you can pass arguments when using preset, how much do we need to pay kitware to have this implemented?

@bretbrownjr bretbrownjr requested a review from wusatosi March 1, 2025 22:38
@bretbrownjr bretbrownjr force-pushed the inject-fetch-content branch from 44d4fa5 to 6ff2015 Compare March 2, 2025 01:53
Problem
-------

**Given**:

A sandbox environment is an environment where dependencies
are all provided but access to the internet is not. Also take as
a given that the build for this project happens in an environment
that meets that description.

**When**:

Configuring this project with any of the provided approaches. For
instance:

```
cmake -B build -S .
```

**Then**:

The configuration step hangs or fails, depending on the nature of
the sandboxed environment.

Solution
--------

Move to a `find_package` first implementation for the
`CMakeLists.txt` of this project.

Using `FetchContent` is still trivial, using the new functionality
in `cmake/use-fetch-content.cmake` as documented in the updates to
`README.md` included in this commit.

Signed-off-by: Bret Brown <[email protected]>
* Use the `libgtest-dev` package on Ubuntu for the compiler-test
  workflow.
* Use the `use-fetch-content.cmake` flag in other workflows.
* Discontinue use of `--workflow` for CMake as it does not support
  use of any `-D` flags, which are required to use `FetchContent`
  via `use-fetch-content.cmake` on Windows and MacOS.

The approach in this commit provides coverage for both FetchContent
and find_package approaches to the project.

Signed-off-by: Bret Brown <[email protected]>
@bretbrownjr bretbrownjr force-pushed the inject-fetch-content branch from 37bf0d0 to fc0b45f Compare March 2, 2025 02:33
@bretbrownjr
Copy link
Copy Markdown
Member Author

This PR is updated and ready for final review. If it's basically OK, I would like to merge so I can push up pull requests to initially support Conan and vcpkg. I'm happy to collaborate on followon PRs to provide more polish, iterate on how we inject FetchContent, etc.

@wusatosi I added some documentation for lockfile.json as requested.

Copy link
Copy Markdown
Member

@wusatosi wusatosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI and docs looks good to me with minor suggestions.

This is a CMake heavy PR that I am not qualified to approve. I think @camio should take a crack at this.

I am still bit salty about the loss of ergonomics here, but I guess we can bring it back later.

I encourage reviewer to read through my comments and @bretbrownjr 's reply, there are significant decisions made in this PR.

Comment thread docs/README.md
Comment thread README.md
Comment thread README.md
@wusatosi
Copy link
Copy Markdown
Member

wusatosi commented Mar 2, 2025

Can you add this patch so the devcontainer still work?

diff --git a/.devcontainer/postcreate.sh b/.devcontainer/postcreate.sh
index bddba03..81d2420 100644
--- a/.devcontainer/postcreate.sh
+++ b/.devcontainer/postcreate.sh
@@ -1,3 +1,6 @@
 # Setup pre-commit
 pre-commit
 pre-commit install
+
+# Install GoogleTest
+sudo apt-get -y install libgtest-dev

@bretbrownjr bretbrownjr force-pushed the inject-fetch-content branch from fc0b45f to e5bc7df Compare March 2, 2025 22:18
@bretbrownjr
Copy link
Copy Markdown
Member Author

@wusatosi I added more docs walking through how to add a Catch2 dependency. Does that help?

I also added libgtest-dev to the devcontainer JSON. I did a little tidying up in there as well. Hopefully that's OK.

@bretbrownjr bretbrownjr requested a review from wusatosi March 2, 2025 22:19
These docs were added to `docs/README.md` since that file is
targeted at people looking to contribute to this project.

Signed-off-by: Bret Brown <[email protected]>
* Add libgtest-dev to devcontainer.
* Restructure devcontainer to group repository tweaks, debian
  package installs, and pip package installs into separate
  sections.
* Within each group, sort installed packages alphabetically,
  and put each on its own line. This makes code review and
  conflict resolution easier.
@bretbrownjr bretbrownjr force-pushed the inject-fetch-content branch from e5bc7df to 51c5d96 Compare March 2, 2025 22:21
Copy link
Copy Markdown
Member

@wusatosi wusatosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌 no more complaints from me

Comment thread README.md
Comment thread docs/README.md Outdated
Comment thread docs/README.md
Comment thread README.md
@@ -88,16 +88,13 @@ This project requires at least the following to build:

* C++17
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* C++17
* C++20

Below we're now defaulting to 20, which seems to be the right default - suggest we update here too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't change this detail. for what it's worth.

I believe the library supports C++17 entirely, but certain test cases require C++20 to run.

I'm pretty sure we could backport the library to C++03, even. I think all of the build requirements are related to the unit tests, but I've been focusing mostly on tooling details, so I haven't followed up on this detail.

Given you've approved the PR, I'll assume that means we're OK addressing the C++ standard compatibility in a separate PR or issue.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bret is right here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't affect this PR, but I was writing a convenience utility for processing json in cmake in a more declarative way instead of the convoluted string function that currently exists. Maybe unnecessary if we don't expect to write too much of this cmake code like this

Copy link
Copy Markdown
Member

@JeffGarland JeffGarland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks like a good step forward -- I've made a couple small suggestions, but overall I think we should merge and improve from here. Approving.

* Bulleted lists should be a complete sentence with a period.

Co-authored-by: Jeff Garland <[email protected]>
@bretbrownjr bretbrownjr requested a review from wusatosi March 3, 2025 12:03
@bretbrownjr bretbrownjr merged commit 3ceff82 into bemanproject:main Mar 5, 2025
@bretbrownjr bretbrownjr deleted the inject-fetch-content branch March 5, 2025 11:01
@bretbrownjr
Copy link
Copy Markdown
Member Author

I'm merging this change given the approving reviews. @wusatosi, I'm happy to team up on follow up on issues and PRs addressing ergonomic concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants