[24.11] libblake3: 1.5.5 -> 1.7.0 by silvanshade · Pull Request #391418 · NixOS/nixpkgs

silvanshade · 2025-03-20T00:45:24Z

This PR is a backport of libblake3 version 1.7.0 to release 24.11 from #390458

This PR depends on another backport PR for tbb_2022_0: #391413

The backport is needed for NixOS/nix#12676

wolfgangwalther · 2025-03-23T19:56:32Z

Please do the following when manually backporting changes:

mention in the PR description which other PR you are backporting
cherry-pick with -x, which will put the commit reference in the new commit message and allow the "check cherry-picks" job to work.

This is also documented: https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#manually-backporting-changes

(cherry picked from commit f312872)

(cherry picked from commit cacda7a)

(cherry picked from commit cd9a523)

(cherry picked from commit b49ff29) Modified to apply from commit 5060af0

wolfgangwalther · 2025-03-24T17:59:47Z

Please rebase after the merge of #391413.

Edit: Although there doesn't seem to be a merge conflict.. so let's try merging this way...

wolfgangwalther · 2025-03-24T18:08:45Z

Hm, unfortunately realized too late, that this breaks the build for libblake3 on x86_64-darwin (not aarch64-darwin, though). It fails with:

In file included from /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:4:
In file included from /nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:23:
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task.h:285:13: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer
    virtual ~task() = default;
            ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task_handle.h:49:5: note: in defaulted destructor for 'tbb::detail::d1::task' first required here
    task_handle_task(d1::wait_tree_vertex_interface* vertex, d1::task_group_context& ctx, d1::small_object_allocator& alloc)
    ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task.h:285:13: note: if you supply your own aligned allocation functions, use -faligned-allocation to silence this diagnostic
    virtual ~task() = default;
            ^
In file included from /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:4:
In file included from /nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:27:
In file included from /nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/task_group.h:29:
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task_handle.h:57:5: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer
    ~task_handle_task() override {
    ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task_handle.h:57:5: note: if you supply your own aligned allocation functions, use -faligned-allocation to silence this diagnostic
In file included from /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:4:
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer
struct function_invoker : public task {
       ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: note: in implicit destructor for 'tbb::detail::d1::function_invoker<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), tbb::detail::d1::invoke_root_task>' first required here
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:149:44: note: in instantiation of template class 'tbb::detail::d1::function_invoker<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), tbb::detail::d1::invoke_root_task>' requested here
    function_invoker<F1, invoke_root_task> invoker1(f1, root);
                                           ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:194:5: note: in instantiation of function template specialization 'tbb::detail::d1::invoke_recursive_separation<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>' requested here
    invoke_recursive_separation(root_wait_ctx, context, fs..., f1);
    ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:207:9: note: in instantiation of function template specialization 'tbb::detail::d1::parallel_invoke_impl<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7)>' requested here
        parallel_invoke_impl(std::forward<T>(t), std::forward<Fs>(args)...);
        ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:216:5: note: in instantiation of member function 'tbb::detail::d1::invoke_helper<std::tuple<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7)>, (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>::operator()' requested here
    invoke_helper<std::tuple<>, Fs...>()(std::forward<Fs>(fs)...);
    ^
/tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:28:16: note: in instantiation of function template specialization 'tbb::detail::d1::parallel_invoke<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>' requested here
  oneapi::tbb::parallel_invoke(
               ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: note: if you supply your own aligned allocation functions, use -faligned-allocation to silence this diagnostic
struct function_invoker : public task {
       ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer
struct function_invoker : public task {
       ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: note: in implicit destructor for 'tbb::detail::d1::function_invoker<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7), tbb::detail::d1::invoke_root_task>' first required here
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:150:44: note: in instantiation of template class 'tbb::detail::d1::function_invoker<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7), tbb::detail::d1::invoke_root_task>' requested here
    function_invoker<F2, invoke_root_task> invoker2(f2, root);
                                           ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:194:5: note: in instantiation of function template specialization 'tbb::detail::d1::invoke_recursive_separation<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>' requested here
    invoke_recursive_separation(root_wait_ctx, context, fs..., f1);
    ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:207:9: note: in instantiation of function template specialization 'tbb::detail::d1::parallel_invoke_impl<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7)>' requested here
        parallel_invoke_impl(std::forward<T>(t), std::forward<Fs>(args)...);
        ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:216:5: note: in instantiation of member function 'tbb::detail::d1::invoke_helper<std::tuple<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7)>, (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>::operator()' requested here
    invoke_helper<std::tuple<>, Fs...>()(std::forward<Fs>(fs)...);
    ^
/tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:28:16: note: in instantiation of function template specialization 'tbb::detail::d1::parallel_invoke<(lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:29:7), (lambda at /tmp/nix-build-libblake3-1.7.0.drv-0/source/c/blake3_tbb.cpp:33:7)>' requested here
  oneapi::tbb::parallel_invoke(
               ^
/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/parallel_invoke.h:39:8: note: if you supply your own aligned allocation functions, use -faligned-allocation to silence this diagnostic
struct function_invoker : public task {
       ^
4 errors generated.
make[2]: *** [CMakeFiles/blake3.dir/build.make:170: CMakeFiles/blake3.dir/blake3_tbb.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:115: CMakeFiles/blake3.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

This does not happen on master, though.

silvanshade · 2025-03-24T19:43:40Z

@wolfgangwalther Thanks for merging these.

I don't have an x86_64-darwin system to test with unfortunately but did some investigation.

The x86_64-darwin system seems to be compiling with a macOS SDK that is too old based on the error:

/nix/store/kllykddp7hb8lcy24nf5mbg54a937qn0-tbb-2022.0.0-dev/include/oneapi/tbb/detail/_task.h:285:13: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer

I think the minimum macOS available for aarch64 is 11.00 so it not being an issue there would make sense.

The fix seems like it would be related to this PR: #346043

There they mention this:

This PR adds a new SDK pattern for Darwin that provides a single SDK package for each version. A default SDK is provided as an extraBuildInput in the Darwin stdenv. To use a newer SDK version, add it as a buildInput. A hook is used to pick which SDK of those available is used (currently always the newest); and it is possible to have a different SDK for build, for host, and for target platforms. Most importantly, it is never possible to have more than one SDK active. Weird errors due to semicolons or other punctuation should be a thing of the past.

I think the solution then would be to make this change:

buildInputs = lib.optionals useTBB [ tbb_2022_0 ]
  ++ lib.optional stdenv.hostPlatform.isDarwin apple-sdk_11

If that fixes it I can submit another PR (although not sure how I should format that since it's not a backport technically) or maybe you can just submit a quick follow up?

wolfgangwalther · 2025-03-24T20:25:28Z

Thanks for looking into it. Your analysis was 100% spot on, your solution was close. Instead of another SDK version, I had to use the darwinMinVersionHook to make it build. See #392846.

github-actions bot added the 8.has: maintainer-list (update) This PR changes `maintainers/maintainer-list.nix` label Mar 20, 2025

silvanshade force-pushed the libblake3-backport-24.11 branch from cf75cd6 to db559ee Compare March 20, 2025 00:46

silvanshade mentioned this pull request Mar 20, 2025

Implement memory-mapped IO and multi-threading for BLAKE3 hashing NixOS/nix#12676

Merged

github-actions bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 20, 2025

nix-owners bot requested a review from fgaz March 20, 2025 00:52

mweinelt and others added 4 commits March 24, 2025 10:46

tbb_2022_0: init at 2022.0.0

bdb5b17

(cherry picked from commit f312872)

maintainers: add silvanshade

6e5965d

(cherry picked from commit cacda7a)

libblake3: add silvanshade to maintainers

b93d31e

(cherry picked from commit cd9a523)

libblake3: 1.5.5 -> 1.7.0

d1b6af4

(cherry picked from commit b49ff29) Modified to apply from commit 5060af0

silvanshade force-pushed the libblake3-backport-24.11 branch from db559ee to d1b6af4 Compare March 24, 2025 17:27

silvanshade changed the title ~~[Backport release-24.11] libblake3: 1.5.5 -> 1.7.0~~ [24.11] libblake3: 1.5.5 -> 1.7.0 Mar 24, 2025

silvanshade self-assigned this Mar 24, 2025

silvanshade requested a review from Ericson2314 March 24, 2025 17:32

wolfgangwalther merged commit 839a41f into NixOS:release-24.11 Mar 24, 2025
35 of 45 checks passed

wolfgangwalther mentioned this pull request Mar 24, 2025

[24.11] libblake3: fix build on x86_64-darwin #392846

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[24.11] libblake3: 1.5.5 -> 1.7.0#391418

[24.11] libblake3: 1.5.5 -> 1.7.0#391418
wolfgangwalther merged 4 commits intoNixOS:release-24.11from
silvanshade:libblake3-backport-24.11

silvanshade commented Mar 20, 2025 •

edited

Loading

Uh oh!

wolfgangwalther commented Mar 23, 2025

Uh oh!

wolfgangwalther commented Mar 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

wolfgangwalther commented Mar 24, 2025

Uh oh!

silvanshade commented Mar 24, 2025 •

edited

Loading

Uh oh!

wolfgangwalther commented Mar 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

silvanshade commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wolfgangwalther commented Mar 23, 2025

Uh oh!

wolfgangwalther commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

wolfgangwalther commented Mar 24, 2025

Uh oh!

silvanshade commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wolfgangwalther commented Mar 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

silvanshade commented Mar 20, 2025 •

edited

Loading

wolfgangwalther commented Mar 24, 2025 •

edited

Loading

silvanshade commented Mar 24, 2025 •

edited

Loading