ceph: Build with Python 3.12 by nh2 · Pull Request #455889 · NixOS/nixpkgs

nh2 · 2025-10-26T14:46:14Z

Extracted from PR

Build ceph with Python 3.12; move to pkgs/by-name #443671

Things done

Add a 👍 reaction to pull requests you find important.

Also patch s3select for arrow-cpp 20. See: NixOS#426609 Also: * Build with current `fmt` Split (from the original larger contribution NixOS#443671 ) done by: Niklas Hambüchen <[email protected]>

josh · 2025-10-26T17:21:14Z

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix

+    # Version pinned by Ceph `src/rocksdb` submodule, last update from:
+    #     https://github.com/ceph/ceph/tree/v19.2.3/src
+    rev = "9fa4990159853479a222244574ca41202e4c95c1";
+    sha256 = "sha256-g0CCiWHG/jjn/fumHGXp88PjOGuRjrKC3ETCy6ffWvQ=";


Hmm I'm getting a mismatch on this hash when building:

specified: sha256-g0CCiWHG/jjn/fumHGXp88PjOGuRjrKC3ETCy6ffWvQ= got: sha256-JJXtN8mk8fRsGzyimw4e7tN2/c1RJ4Ei6YioVIMhvl0=

Hm, very weird. I can repro what you say. I'll push a fix but am not sure how it built yesterday given this.

benaryorg

Just a cursory glance from my side, I'll take a closer look soon.

benaryorg · 2025-10-27T11:39:42Z

pkgs/by-name/ce/ceph/package.nix

-
-  # Remove once Ceph supports arrow-cpp >= 20, see:
-  # * https://tracker.ceph.com/issues/71269
-  # * https://github.com/NixOS/nixpkgs/issues/406306
-  ceph-arrow-cpp ? callPackage ./arrow-cpp-19.nix { },


Is there a stable link we can provide here to check for current compatibility, similar to the comment we have for Python?

Maybe https://github.com/ceph/ceph/commits/main/src/arrow ?

benaryorg · 2025-10-27T12:33:35Z

pkgs/by-name/ce/ceph/package.nix

+  src = fetchurl {
+    url = "https://download.ceph.com/tarballs/ceph-${version}.tar.gz";
+    hash = "sha256-zlgp28C81SZbaFJ4yvQk4ZgYz4K/aZqtcISTO8LscSU=";
+  };
+


I've run into issues with that pattern before (e.g. the authentik package); since src is used not just in ceph but also ceph-common below, overriding src using overrideAttrs will cause issues since ceph will be built with the new src, but ceph-common won't.
Is there anything we can do to improve this, like making ceph-common somehow reference the finalAttrs.src of ceph (like by making this a function taking a src and version argument and calling the function using the finalAttr values from within ceph)?

I realize this isn't being introduced by this PR (the code just moves up a few lines), so this is kind of out-of-scope for this PR, but I'd like to throw it in here so it's at least somewhere.

That is a very good point, I'd definitely want my overrideAttrs of Ceph source to also override its Python code!

Could we put the src of the C++ package into the passthru and get it that way for the Python? Edit: Hmm no, the C++ package references the ceph-pyhton-env.

Is there anything we can do to improve this, like making ceph-common somehow reference the finalAttrs.src of ceph (like by making this a function taking a src and version argument and calling the function using the finalAttr values from within ceph)?

👍🏻 I've wanted the same!

Could we put the src of the C++ package into the passthru and get it that way for the Python?

Most ways I can think of will just run into infinite recursion.
Is it possible to just add ceph-common and ceph-python-env as attrs of the derivation and change all their references to finalAttrs.{ceph-common,ceph-python-env}?
That would also allow overriding either of them using something like this:

ceph.overrideAttrs ({ ceph-python-env, ... }: { ceph-python-env = ceph-python-env.overrideAttrs { # I dunno, add some Python packages or something }; })

Completely forgot, of course patches should be inherited too in exactly the same way, otherwise we won't be able to apply patches to the Python parts via overrideAttrs for testing either.

josh · 2025-10-27T17:42:10Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 455889
Commit: e73414dce1b45db6119eb6c25ab4760734f6e0fe

`x86_64-linux`

✅ 14 packages built:

ceph (ceph-dev)
ceph-client (ceph-client.client, ceph-client.dev, ceph-client.doc, ceph-client.lib, ceph-client.man, ceph-dev.client, ceph.client)
ceph-csi
ceph.dev (ceph-dev.dev)
ceph.doc (ceph-dev.doc)
libceph (ceph.lib, libceph.client, libceph.dev, libceph.doc, libceph.lib, libceph.man)
ceph.man (ceph-dev.man)
qemu_full
qemu_full.debug
qemu_full.doc
qemu_full.ga
sambaFull (samba4Full)
sambaFull.dev (samba4Full.dev)
sambaFull.man (samba4Full.man)

`aarch64-linux`

✅ 10 packages built:

ceph (ceph-dev)
ceph-client (ceph-client.client, ceph-client.dev, ceph-client.doc, ceph-client.lib, ceph-client.man, ceph-dev.client, ceph.client)
ceph-csi
ceph.dev (ceph-dev.dev)
ceph.doc (ceph-dev.doc)
libceph (ceph.lib, libceph.client, libceph.dev, libceph.doc, libceph.lib, libceph.man)
ceph.man (ceph-dev.man)
qemu_full
qemu_full.doc
qemu_full.ga

eclairevoyant

I think this becomes imperative again since ceph is broken on master (this time due to some python deps).

thunze · 2025-11-14T19:23:55Z

pkgs/by-name/ce/ceph/package.nix

-        # Ceph does not support the following yet:
-        # * `bcrypt` > 4.0
-        # * `cryptography` > 40
-        # See:
-        # * https://github.com/NixOS/nixpkgs/pull/281858#issuecomment-1899358602
-        # * Upstream issue: https://tracker.ceph.com/issues/63529
-        #   > Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3
-        # * Moved to issue: https://tracker.ceph.com/issues/64213
-        #   > MGR modules incompatible with later PyO3 versions - PyO3 modules may only be initialized once per interpreter process


Unfortunately still seeing the bcrypt issue via ceph-mgr:

ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576

Encountered this when trying to run the ceph-single-node NixOS test via ceph.passthru.tests.ceph-single-node. This test times out due to ceph -s printing health: HEALTH_WARN instead of health: HEALTH_OK (13 mgr modules have failed dependencies).

Full logs: ceph-single-node.log

I can reproduce the same failure.

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix

pkgs/by-name/ce/ceph/package.nix

nh2 · 2025-11-21T06:12:55Z

Side note: I have a WIP branch ceph-20 for when this PR is done:

https://github.com/nh2/nixpkgs/pull/new/ceph-20 (currently at commit 5ba8628)

I won't have time to work on this PR #455889 though unitl beginning of December, so in case anybody wants to already address or test easy-to-address feedback from above, feel free to add some commits!

That builds ~~and passes tests~~, on commit 68cde06.

When I put the commit 5ba8628 on top of it that switches to Python 3.13, it has in the tests' logs ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576, and tests hang (don't terminate), so I'm removing that from my branch ceph-20.

Edit: The above must be wrong, running it again on these commits, I do get the PyO3 modules do not yet support subinterpreters already on Python 3.12 (without the Python 3.13 usage), and tests hang.

arachnist · 2025-11-26T19:04:44Z

This very well might be a PEBKAC, but activating ceph OSDs[2] failed for me[3] when trying to use this PR as an overlay[1].

I suspect this might be caused by python importlib shenanigans, as the code surrounding the failing part is:

10:try:
11:    from importlib.metadata import entry_points
12:
13:    def get_entry_points(group: str):  # type: ignore
14:        return entry_points().get(group, [])  # type: ignore
15:except ImportError:

[1]:

  nixpkgsCeph = pkgs.fetchFromGitHub {
    owner = "nixos";
    repo = "nixpkgs";
    rev = "e73414dce1b45db6119eb6c25ab4760734f6e0fe";
    sha256 = "sha256-38D40MYOuso2QuGePuaQrI7UzAf9vttMEnqA0fiCSqc=";
  };

[ ... ]

      overlays = [
        (self: super: {
          inherit (import nixpkgsCeph { system = "x86_64-linux"; }) ceph;
        })
      ];

[2]: ceph-volume lvm activate --bluestore --no-systemd 15 …

[3]:

exception caught by decorator
Traceback (most recent call last):
  File "/nix/store/01df67x6jgh8cafgv88hbidv2nsv7qbj-ceph-19.2.3/lib/python3.12/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
           ^^^^^^^^^^^
  File "/nix/store/01df67x6jgh8cafgv88hbidv2nsv7qbj-ceph-19.2.3/lib/python3.12/site-packages/ceph_volume/main.py", line 118, in main
    self.enable_plugins()
  File "/nix/store/01df67x6jgh8cafgv88hbidv2nsv7qbj-ceph-19.2.3/lib/python3.12/site-packages/ceph_volume/main.py", line 83, in enable_plugins
    plugins = _load_library_extensions()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/01df67x6jgh8cafgv88hbidv2nsv7qbj-ceph-19.2.3/lib/python3.12/site-packages/ceph_volume/main.py", line 188, in _load_library_extensions
    for ep in get_entry_points(group=group):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/01df67x6jgh8cafgv88hbidv2nsv7qbj-ceph-19.2.3/lib/python3.12/site-packages/ceph_volume/main.py", line 14, in get_entry_points
    return entry_points().get(group, [])  # type: ignore
           ^^^^^^^^^^^^^^^^^^
AttributeError: 'EntryPoints' object has no attribute 'get'
-->  AttributeError: 'EntryPoints' object has no attribute 'get'

edit:
This particular problem can be solved by adding another patch on ceph:

              (pkgs.fetchpatch {
                url = "https://github.com/ceph/ceph/commit/8c78a22d2cf69892570f635735d9735169b64a75.patch";
                hash = "sha256-QnlSGJw0lRdXoLaf58/yXr3ArAut40dSnkA3TFeFJNE=";
              })

but after that we hit problems with ceph-mgr, which we didn't investigate further (rolled back to previous versions):

ceph-mgr[29638]: 2025-11-26T20:41:49.400+0100 7f30e18bf900 -1 mgr[py] Traceback (most recent call last):
ceph-mgr[29638]:   File "/nix/store/fllm5is0r4zh6sl4qapbhf7rklhlla7k-ceph-19.2.3-lib/lib/ceph/mgr/mgr_module.py", line 30, in <module>
ceph-mgr[29638]:     from mgr_util import profile_method
ceph-mgr[29638]:   File "/nix/store/fllm5is0r4zh6sl4qapbhf7rklhlla7k-ceph-19.2.3-lib/lib/ceph/mgr/mgr_util.py", line 6, in <module>
ceph-mgr[29638]:     import bcrypt
ceph-mgr[29638]:   File "/nix/store/p8fvcvb8jv49fv0vigwlvrd4zyhlcal6-python3-3.12.11-env/lib/python3.12/site-packages/bcrypt/__init__.py", line 13, in <module>
ceph-mgr[29638]:     from ._bcrypt import (
ceph-mgr[29638]: ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576
ceph-mgr[29638]: 2025-11-26T20:41:49.401+0100 7f30e18bf900 -1 mgr[py] Class not found in module 'osd_perf_query'
ceph-mgr[29638]: 2025-11-26T20:41:49.401+0100 7f30e18bf900 -1 mgr[py] Error loading module 'osd_perf_query': (22) Invalid argument

numinit · 2025-11-29T19:17:13Z

I've gone ahead and marked @dotlambda's review comments as resolved because I've implemented them here after rebasing against master: numinit@f6d3953

I'll hold off from force-pushing this branch up to @nh2's tree until I can get the tests passing though.

@arachnist

This particular problem can be solved by adding another patch on ceph: [...] but after that we hit problems with ceph-mgr, which we didn't investigate further (rolled back to previous versions):

My ceph-python3.12 branch is currently here (and this is reproducible by running nix-build -A ceph.tests). I pulled in the latest patch you suggested and that fixed the first issue you mentioned. Trying to figure out the bcrypt issue now.

numinit · 2025-11-29T19:37:00Z

Which means we're back here:

pyca/cryptography#12080 (comment)

#380823 (comment)

ceph/ceph#62951

@benaryorg

Ideally we could just apply ceph#62951 onto the latest squid release (i.e. adding it as a fetchpatch or such in #455889), although I think we also need ceph#65959 for this to work properly.

Giving this a shot now.

numinit · 2025-11-29T21:05:01Z

Got patches applying cleanly on numinit@4d74cd2; still debugging a ModuleNotFoundError: No module named 'ceph.cryptotools' error from ceph.tests but this is at least progress.

Edit: Still some errors that cause tests to time out, but it got further with these:

ERROR:root:Module 'xmltodict' is not installed.
ceph/mgr/cephadm/ssl_cert_utils.py: from cryptography import x509 throws ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 which breaks cephadm

Wondering if we should use aur-ceph patches or proxmox but proxmox also has a lot of them.

numinit · 2025-11-30T21:51:27Z

I guess one thing I could try is applying most of the 55 entire patches that Proxmox does. It's for nearly the same version.

My guess is we'll still need a few of our own, and leaving out the Debian-specific ones, but they seemed to address some of the cases that the Arch patch did not.

nh2 · 2025-12-01T11:22:34Z

I guess one thing I could try is applying most of the 55 entire patches that Proxmox does

@numinit Does Proxmox track these patches somewhere instead of just collecting them as .patch files in a folder, e.g. to track upstream status?

It's for nearly the same version.

Note though that I have Ceph 20 already in the pipeline (see #455889 (comment))

numinit · 2025-12-01T18:14:43Z

@nh2 The patch files look like this:

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: John Mulligan <[email protected]>
Date: Thu, 17 Apr 2025 13:23:09 -0400
Subject: [PATCH] python-common/cryptotools: use one single dir for cryptotools

(Regrettably, the commit hash is zeroed... but my guess is it could be reverse engineered back into the original hashes with enough git digging).

BTW: numinit@0d2b82e

Have to clean up the commit series, but:

/nix/store/sy25b2dc6cv5lgr2q0ab0ns3y1vk2nyn-vm-test-run-basic-multi-node-ceph-cluster
/nix/store/mw178ixr5vp6d6r56494d86nkzs58mw8-vm-test-run-basic-single-node-ceph-cluster
/nix/store/ng7qzf43b7qm7xlw2i9h3haqfb5i2xl6-vm-test-run-basic-single-node-ceph-cluster-bluestore
/nix/store/w3rbrh6whfa7688524b9hanr2hmlx6qj-vm-test-run-basic-single-node-ceph-cluster-bluestore-dmcrypt

😄

numinit · 2025-12-01T18:15:32Z

Note though that I have Ceph 20 already in the pipeline (see #455889 (comment))

Yeah. I figure this work is just going to be for 25.11 since going to 20 would be breaking.

dotlambda · 2025-12-01T18:36:53Z

I figure this work is just going to be for 25.11

But ceph was fixed in #462435 (#466339) and #462437.
I don't think moving ceph to Python 3.12 should be backported.

numinit · 2025-12-01T18:38:03Z

I think this becomes imperative again since ceph is broken on master (this time due to some python deps).

So, guess it's not imperative anymore and we can go to 20?

Edit: the branch with passing tests using Proxmox patches, in case it's useful: https://github.com/numinit/nixpkgs/tree/ceph-python3.12-proxmox

nh2 and others added 2 commits October 26, 2025 14:35

ceph: Move patches to patches/ dir

87eb8dd

ceph: Build with Python 3.12

1df388b

Also patch s3select for arrow-cpp 20. See: NixOS#426609 Also: * Build with current `fmt` Split (from the original larger contribution NixOS#443671 ) done by: Niklas Hambüchen <[email protected]>

nh2 requested review from benaryorg, djds and josh October 26, 2025 14:46

nh2 mentioned this pull request Oct 26, 2025

Build ceph with Python 3.12; move to pkgs/by-name #443671

Open

nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. labels Oct 26, 2025

nix-owners bot requested review from adevress, johanot and krav October 26, 2025 14:51

josh reviewed Oct 26, 2025

View reviewed changes

benaryorg mentioned this pull request Oct 27, 2025

Ceph PyO3 tracking issue #380823

Open

benaryorg reviewed Oct 27, 2025

View reviewed changes

ceph: rocksdb: Pin Ceph's rocksdb version

e73414d

nh2 force-pushed the ceph-python3.12 branch from 47815f8 to e73414d Compare October 27, 2025 13:55

nixpkgs-ci bot added the 2.status: merge-bot eligible This PR can be merged by commenting "@NixOS/nixpkgs-merge-bot merge". label Nov 5, 2025

eclairevoyant approved these changes Nov 14, 2025

View reviewed changes

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Nov 14, 2025

thunze reviewed Nov 14, 2025

View reviewed changes

numinit mentioned this pull request Nov 14, 2025

NixOS 25.11 – Feature Freeze & Release Blockers #444721

Closed

53 tasks

dotlambda reviewed Nov 16, 2025

View reviewed changes

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix Show resolved Hide resolved

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix Show resolved Hide resolved

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix Show resolved Hide resolved

pkgs/by-name/ce/ceph/rocksdb_7_9/default.nix Show resolved Hide resolved

dotlambda reviewed Nov 16, 2025

View reviewed changes

eclairevoyant mentioned this pull request Nov 16, 2025

Build failure: ceph #462362

Closed

3 tasks

dotlambda linked an issue Nov 17, 2025 that may be closed by this pull request

Build failure: ceph #462362

Closed

3 tasks

nixpkgs-ci bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Nov 20, 2025

limwa mentioned this pull request Feb 25, 2026

Build failure: ceph #493884

Open

3 tasks

nixpkgs-ci bot removed the 2.status: merge-bot eligible This PR can be merged by commenting "@NixOS/nixpkgs-merge-bot merge". label Feb 26, 2026

Uh oh!

Conversation

nh2 commented Oct 26, 2025 • edited by numinit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benaryorg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nh2 Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josh commented Oct 27, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

Uh oh!

eclairevoyant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nh2 commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arachnist commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

numinit commented Nov 29, 2025

Uh oh!

numinit commented Nov 29, 2025

Uh oh!

numinit commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

numinit commented Nov 30, 2025

Uh oh!

nh2 commented Dec 1, 2025

Uh oh!

numinit commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

numinit commented Dec 1, 2025

Uh oh!

dotlambda commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

numinit commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

nh2 commented Oct 26, 2025 •

edited by numinit

Loading

nh2 Oct 27, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

nh2 commented Nov 21, 2025 •

edited

Loading

arachnist commented Nov 26, 2025 •

edited

Loading

numinit commented Nov 29, 2025 •

edited

Loading

numinit commented Dec 1, 2025 •

edited

Loading

dotlambda commented Dec 1, 2025 •

edited

Loading

numinit commented Dec 1, 2025 •

edited

Loading