Skip to content

Build ceph with Python 3.12; move to pkgs/by-name#443671

Open
djds wants to merge 1 commit intoNixOS:masterfrom
djds:pkgs/by-name/ce/ceph
Open

Build ceph with Python 3.12; move to pkgs/by-name#443671
djds wants to merge 1 commit intoNixOS:masterfrom
djds:pkgs/by-name/ce/ceph

Conversation

@djds
Copy link
Member

@djds djds commented Sep 17, 2025

Fixes #442652

Also patch s3select for arrow-cpp 20.

See: #426609

@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. labels Sep 17, 2025
Also patch s3select for arrow-cpp 20.

See: NixOS#426609
@djds djds force-pushed the pkgs/by-name/ce/ceph branch from 62ac27f to 3df23a8 Compare September 17, 2025 07:12
@djds djds marked this pull request as ready for review September 17, 2025 22:45
@djds
Copy link
Member Author

djds commented Sep 18, 2025

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 443671
Commit: 3df23a82c966e957bd039d2305099b00408b33a0


x86_64-linux

✅ 14 packages built:
  • ceph (ceph-dev)
  • ceph-client
  • ceph-csi
  • ceph.dev (ceph-dev.dev)
  • ceph.doc (ceph-dev.doc)
  • libceph (ceph.lib, libceph.dev, libceph.doc, libceph.lib, libceph.man)
  • ceph.man (ceph-dev.man)
  • qemu_full
  • qemu_full.debug
  • qemu_full.doc
  • qemu_full.ga
  • sambaFull (samba4Full)
  • sambaFull.dev (samba4Full.dev)
  • sambaFull.man (samba4Full.man)

@djds
Copy link
Member Author

djds commented Sep 19, 2025

@ccicnce113424 I have a feeling that this won't be accepted as is because of the inclusion of the ceph-s3select-arrow-20-compat.patch. I've been using that since your #426609 and while I understand the caution, I'd love your opinion since carrying a bunch of out of tree packages has it's own maintenance overhead (with the obvious caveat that I've done the same thing with rocksdb 7.9 here).

@bugsbugsbux
Copy link

hello? how about reacting in any way dear five! reviewers?

@djds
Copy link
Member Author

djds commented Sep 28, 2025

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    nixpkgs-djds-ceph.url = "github:djds/nixpkgs/pkgs/by-name/ce/ceph";
  };

  outputs =
    {
      nixpkgs,
      ...
    }@inputs:
    let
      system = "x86_64-linux";

      pks = import nixpkgs {
        inherit system;

        overlays = [
          ({
             inputs,
             system,
             ...
           }:
           # https://github.com/NixOS/nixpkgs/issues/442652
           _final: _prev: {
             inherit (inputs.nixpkgs-djds-ceph.legacyPackages.${system})
               ceph
               ceph-client
               ;
          })
        ];
      };
    in
    {
      ...
    };
}

@benaryorg
Copy link
Contributor

benaryorg commented Sep 29, 2025

I don't think I'll get around to taking a closer look at this one soon, however it would be great if you could split the "move to pkgs/by-name" off into a separate commit.
With everything in the same commit the diff is really hard to review.
Separate commits would allow to review the code changes of either commit properly without having to cross-reference files, and it would allow proper comments using the GitHub review feature, which currently recognises all files as deleted and new, none as moved, as far as I can tell.

Edit: please note also the nixpkgs commit conventions for pkgs

@djds
Copy link
Member Author

djds commented Sep 30, 2025

Yeah, I'm also a volunteer.

It's easier for me to carry this overlay than go back and redo commits because it would fit the GitHub workflow better or because I ignored the commit convention. Originally I opened this as a draft, but after 2 weeks, no one even attempted to fix the ceph build, which is a pretty important package for NixOS to be considered a serious system. I'm not saying I won't go back and redo this PR, but I also don't think I'll get around to taking a closer look anytime soon. Feel free to use this work as you see fit. It's open source, after all.

@nh2
Copy link
Contributor

nh2 commented Sep 30, 2025

@djds I don't quite understand if/how it addresses the PyO3 issue (#380823).

Doesn't that need ceph/ceph#62951 which is not in an upstream Ceph release?

@djds
Copy link
Member Author

djds commented Oct 1, 2025

@djds I don't quite understand if/how it addresses the PyO3 issue (#380823).

Doesn't that need ceph/ceph#62951 which is not in an upstream Ceph release?

Yeah I think I'm just wrong about this.

@nh2
Copy link
Contributor

nh2 commented Oct 1, 2025

@djd

It's easier for me to carry this overlay than go back and redo commits because it would fit the GitHub workflow better

Note this is not about any workflow imposed by Github; it is that it makes human review harder when one cannot easily discern refactorings (which can be trivially be provent to be absent of bugs by checking that the nix eval produces the same .drv) from actual changes one has to review and test carefully. That turns reviewing a few lines very carefully into having to review hundreds of lines carefully.

or because I ignored the commit convention.

That's not a blocker at all (anybody can easily fix that), but you just won't get the nice automated ofborg CI (the packages it attempts to build are determined by the text in the commit message, thus the convention).

I'm not saying I won't go back and redo this PR, but I also don't think I'll get around to taking a closer look anytime soon. Feel free to use this work as you see fit.

A code drop of something that is working is of course already very useful, thank you for that!

But somebody will have to pick the commits apart to merge it (you, one of us, or anybody else).

@nixpkgs-ci nixpkgs-ci bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 7, 2025
@emilazy
Copy link
Member

emilazy commented Oct 11, 2025

Cross‐linking #451118. (That one is presumably much easier to land.)

@emilazy emilazy mentioned this pull request Oct 11, 2025
13 tasks
@bugsbugsbux
Copy link

can we expect this in 25.11?

@benaryorg benaryorg mentioned this pull request Oct 24, 2025
13 tasks
@nh2
Copy link
Contributor

nh2 commented Oct 26, 2025

With PR

merged, the diff is smaller:

# git diff --stat origin/master HEAD pkgs/by-name/ce/ceph/  
 pkgs/by-name/ce/ceph/arrow-cpp-19.nix                                           | 346 --------------------------------------
 pkgs/by-name/ce/ceph/old-python-packages/cryptography-vectors.nix               |  36 ----
 pkgs/by-name/ce/ceph/old-python-packages/cryptography.nix                       | 140 ---------------
 ...hon-cryptography-Cherry-pick-fix-for-CVE-2023-49083-on-cryptography-40.patch |  44 -----
 pkgs/by-name/ce/ceph/old-python-packages/trustme.nix                            |  52 ------
 pkgs/by-name/ce/ceph/package.nix                                                | 255 ++++++----------------------
 pkgs/by-name/ce/ceph/{ => patches}/boost-1.85.patch                             |   0
 pkgs/by-name/ce/ceph/{ => patches}/boost-1.86-PyModule.patch                    |   0
 pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix                                    | 138 +++++++++++++++
 pkgs/by-name/ce/ceph/rocksdb_7_9/patches/fix-findliburing.patch                 |  29 ++++
 10 files changed, 222 insertions(+), 818 deletions(-)

@nh2
Copy link
Contributor

nh2 commented Oct 26, 2025

Side note: Ceph 20 will have Python 3.13 support:

ceph/ceph@ddb4721

# See:
# * <https://github.com/boostorg/python/issues/394>
# * <https://aur.archlinux.org/cgit/aur.git/commit/?h=ceph&id=8c5cc7d8deec002f7596b6d0860859a0a718f12b>
# * <https://github.com/ceph/ceph/pull/60999>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments like the above should not be removed.

Spotting such removals is another good point why refactorings to e.g. by-name should be split from feature updates.

nh2 pushed a commit to nh2/nixpkgs that referenced this pull request Oct 26, 2025
Also patch s3select for arrow-cpp 20.

See: NixOS#426609

Also:

* Build with current `fmt`

Split (from the original larger contribution
NixOS#443671
) done by:
Niklas Hambüchen <[email protected]>
@nh2 nh2 mentioned this pull request Oct 26, 2025
13 tasks
@nh2
Copy link
Contributor

nh2 commented Oct 26, 2025

But somebody will have to pick the commits apart to merge it (you, one of us, or anybody else).

(Hopefully I got exactly the difference right after the PRs merged meanwhile: #440224 and #440226)

@djds please have a review whether that new PR still looks right to you.


Once that is done, we can close this PR here as well.

aviallon pushed a commit to aviallon/nixpkgs that referenced this pull request Nov 27, 2025
Also patch s3select for arrow-cpp 20.

See: NixOS#426609

Also:

* Build with current `fmt`

Split (from the original larger contribution
NixOS#443671
) done by:
Niklas Hambüchen <[email protected]>
numinit pushed a commit to numinit/nixpkgs that referenced this pull request Dec 1, 2025
Also patch s3select for arrow-cpp 20.

See: NixOS#426609

Also:

* Build with current `fmt`

Split (from the original larger contribution
NixOS#443671
) done by:
Niklas Hambüchen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build failure: ceph

5 participants