Skip to content

feat: expand the usefulness of resource_sets#1235

Merged
alexeagle merged 1 commit intobazel-contrib:mainfrom
novas0x2a:resource-set
Jan 23, 2026
Merged

feat: expand the usefulness of resource_sets#1235
alexeagle merged 1 commit intobazel-contrib:mainfrom
novas0x2a:resource-set

Conversation

@novas0x2a
Copy link
Copy Markdown
Contributor

@novas0x2a novas0x2a commented Jan 16, 2026

This adds a new public function, resource_set_for, which takes a cpu and memory value and returns an appropriate resource_set function for those values, adding some support for the cross-product mentioned in the previous pass. The bazel resource_set API isn't very easy to work with, and this hides the ugliness here, rather than expecting every module to implement something like rules_rust did.

The ugliness of the implementation is mitigated slightly by using a helper script to generate it, and I reason that people attempting to use the resource_set API would be better off with additional abstraction, so if/when this gets fixed in bazel, most people don't need to notice.

@novas0x2a novas0x2a mentioned this pull request Jan 16, 2026
Merged
@aspect-workflows
Copy link
Copy Markdown

aspect-workflows bot commented Jan 16, 2026 •
edited
Loading

Bazel 8 (Test)

3 test targets passed

Targets 
//lib/tests/copy_to_directory:case_22_test [k8-fastbuild]                    59ms
//lib/tests/copy_to_directory_bin_action:test [k8-fastbuild]                 109ms
//lib/tests:resource_sets_tests_test_0 [k8-fastbuild]                        76ms

Total test execution time was 244ms. 133 tests (97.8%) were fully cached saving 16s.

Bazel 9 (Test)

3 test targets passed

Targets 
//lib/tests/copy_to_directory:case_22_test [k8-fastbuild]                    56ms
//lib/tests/copy_to_directory_bin_action:test [k8-fastbuild]                 55ms
//lib/tests:resource_sets_tests_test_0 [k8-fastbuild]                        82ms

Total test execution time was 193ms. 133 tests (97.8%) were fully cached saving 17s.

Bazel 7 (Test)

e2e/api_entries

All tests were cache hits

1 test (100.0%) was fully cached saving 24ms.

Bazel 8 (Test)

e2e/api_entries

All tests were cache hits

1 test (100.0%) was fully cached saving 30ms.

Bazel 9 (Test)

e2e/api_entries

All tests were cache hits

1 test (100.0%) was fully cached saving 50ms.

Bazel 7 (Test)

e2e/copy_action

All tests were cache hits

1 test (100.0%) was fully cached saving 31ms.

Bazel 8 (Test)

e2e/copy_action

All tests were cache hits

1 test (100.0%) was fully cached saving 26ms.

Bazel 9 (Test)

e2e/copy_action

All tests were cache hits

1 test (100.0%) was fully cached saving 32ms.

Bazel 7 (Test)

e2e/copy_to_directory

All tests were cache hits

6 tests (100.0%) were fully cached saving 297ms.

Bazel 8 (Test)

e2e/copy_to_directory

All tests were cache hits

6 tests (100.0%) were fully cached saving 286ms.

Bazel 9 (Test)

e2e/copy_to_directory

All tests were cache hits

6 tests (100.0%) were fully cached saving 349ms.

Bazel 7 (Test)

e2e/coreutils

All tests were cache hits

4 tests (100.0%) were fully cached saving 247ms.

Bazel 8 (Test)

e2e/coreutils

All tests were cache hits

4 tests (100.0%) were fully cached saving 216ms.

Bazel 9 (Test)

e2e/coreutils

All tests were cache hits

4 tests (100.0%) were fully cached saving 198ms.

Bazel 7 (Test)

e2e/external_copy_to_directory

All tests were cache hits

1 test (100.0%) was fully cached saving 31ms.

Bazel 8 (Test)

e2e/external_copy_to_directory

All tests were cache hits

1 test (100.0%) was fully cached saving 26ms.

Bazel 9 (Test)

e2e/external_copy_to_directory

All tests were cache hits

1 test (100.0%) was fully cached saving 32ms.

Bazel 7 (Test)

e2e/smoke

All tests were cache hits

4 tests (100.0%) were fully cached saving 611ms.

Bazel 8 (Test)

e2e/smoke

All tests were cache hits

4 tests (100.0%) were fully cached saving 508ms.

Bazel 9 (Test)

e2e/smoke

All tests were cache hits

4 tests (100.0%) were fully cached saving 618ms.

Bazel 7 (Test)

e2e/write_source_files

All tests were cache hits

1 test (100.0%) was fully cached saving 39ms.

Bazel 8 (Test)

e2e/write_source_files

All tests were cache hits

1 test (100.0%) was fully cached saving 33ms.

Bazel 9 (Test)

e2e/write_source_files

All tests were cache hits

1 test (100.0%) was fully cached saving 40ms.

@novas0x2a
Copy link
Copy Markdown
Contributor Author

novas0x2a commented Jan 21, 2026 •
edited
Loading

As a follow-up, here are some of the other things I was looking at doing:

  1. making resource_set respond to tags as well as the attribute, as in: cpu:<small|medium|large|enormous>, cpu:<cores:int>, mem:<small|medium|large|enormous>, mem:<num:int><m|g> (i could be convinced to drop cpu:<int> and mem:<int> in favor of just having the symbolic sizes, though that'd represent a compat break if i did that for the attr, too)
  2. provide some sort of tuning mechanism to allow end users to configure what the sizes mean (as in: all "larges" are given 8 cores: e.g. //setting:cpu_large=8; this is important to give users an out for projects overtuning for large ci machines)
  3. finally, I plan to integrate this into rules_foreign_cc and sync it with the underlying build systems, defaulting (e.g.) CMAKE_BUILD_PARALLEL_LEVEL to the resource_set's cpu request.

I'm interested in helping achieve cross-consistency for these behaviors, which means I'd like (at least) 1 to be handled in bazel-lib, and maybe 2 if there's consensus.

fmeum
fmeum approved these changes Jan 22, 2026
View reviewed changes
@fmeum fmeum requested a review from alexeagle January 22, 2026 19:13
@novas0x2a
feat: expand the usefulness of resource_sets
c083e9f 
This adds a new public function, resource_set_for, which takes a cpu and
memory value and returns an appropriate resource_set function for those
values. The bazel resource_set API isn't very easy to work with, and
this hides the ugliness here, rather than expecting every module to
implement something like [rules_rust
did](https://github.com/bazelbuild/rules_rust/blob/f29a63cb3c473bd0158c8c9d3e0793a33187d505/rust/private/rustc_resource_set.bzl)

This implementation is ugly, but it's made slightly less bad by using a
helper script to generate it, and I reason that people attempting to use
the resource_set API would be better off with additional abstraction, so
if/when this gets fixed in bazel, most people don't need to notice.
@novas0x2a novas0x2a mentioned this pull request Jan 22, 2026
Closed
@alexeagle alexeagle merged commit e6420f3 into bazel-contrib:main Jan 23, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fmeum fmeum fmeum approved these changes

@alexeagle alexeagle Awaiting requested review from alexeagle

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@novas0x2a @fmeum @alexeagle