Skip to content

Comments

libcephfs: FSCrypt userspace implementation#61137

Merged
vshankar merged 130 commits intoceph:mainfrom
chrisphoffman:wip-fscrypt
Nov 6, 2025
Merged

libcephfs: FSCrypt userspace implementation#61137
vshankar merged 130 commits intoceph:mainfrom
chrisphoffman:wip-fscrypt

Conversation

@chrisphoffman
Copy link
Contributor

@chrisphoffman chrisphoffman commented Dec 18, 2024

This PR adds fscrypt support to CephFS userspace.

Development was done in two phases. This PR will include work done in both phases.

Phase 1: Core development
Code: #52172
Functionality added:

  • Implementation of libraries that exist in kernel added (crypto, etc)
  • Sync and async IO added
  • Links and truncate cases
  • Ioctl workarounds added

Phase 2: Making true to fscrypt spec and feature hardening
Functionality added:

  • Access Semantics
  • Multiple Tests
  • Additional support for more fs ops
  • RMW workload edge cases worked out
  • Truncate issues
  • Snapshot cloning of fscrypt enabled subvolumes

Some areas reviewers should focus on:

  • Read/Write paths
  • Listing encrypted directories
  • Access Semantics
  • Fscrypt last block
  • Unit tests

What’s left:

  • Fix issues with unit tests (along the way some of the fscrypt unit tests broke)
  • Since this feature relies heavily on structs defined in the kernel, ensure builds work on Windows
  • Few minor comments
  • Teuthology testing (Fix any failures that exist)
  • Add new capability to test existing fs workunit tests on top of fscrypt enabled dirs (create trackers for issues, fix after this PR)
  • Documentation

Implement fscrypt in libcephfs and cephfs-fuse tracker: https://tracker.ceph.com/issues/63293

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@nizamial09
Copy link
Member

sorry for the noise but i accidentally added the `wip-nia-testinglabel which i was intending to add on another. removed it btw

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

chrisphoffman and others added 22 commits November 5, 2025 13:59
Signed-off-by: Christopher Hoffman <[email protected]>
fcopyfile() reads 1 MiB of data every time but when a fragment smaller
than 1 MiB is left, it still reads 1 MiB of data, causing to never meet
the condition of "off == size". This leads to an infinity loop which
continues to write until CephFS becomes full.

Resolves: rhbz#2379716
Fixes: https://tracker.ceph.com/issues/72238
Signed-off-by: Rishabh Dave <[email protected]>
Move "directory empty" logic to new method "_is_empty_directory".
Future logic will not be a one-liner, so let's keep it separate.

Resolves: rhbz#2376757
Signed-off-by: Marcus Watts <[email protected]>
(cherry picked from commit f0cf85a595876165e2c0eb2ca584e97f44971f9d)
Better check for empty direcotry.
It turns out in->dirstat contains a count of files and subdirectories
from a directory, so all we have to do is make sure that's valid.

Resolves: rhbz#2376757
Signed-off-by: Marcus Watts <[email protected]>
(cherry picked from commit ba233f2dda3cf30c67b653b065e9ed47d42cb9d6)
After each fscrypt unit test clean up after each
unit test.

Signed-off-by: Christopher Hoffman <[email protected]>
Add fscrypt dummy encryption to client. This will allow
for mounting a cephfs volume without providing any fscrypt
information. This will allow for more straightforward setup
for development and test suites.

Signed-off-by: Christopher Hoffman <[email protected]>
Fix warnings/errors in ceph API tests that are present in FSCrypt.cc

src/client/FSCrypt.cc:90:6: error: variable 'olen' set but not used [-Werror,-Wunused-but-set-variable]
   90 |         int olen = 0;
      |             ^
src/client/FSCrypt.cc:91:6: error: variable 'line' set but not used [-Werror,-Wunused-but-set-variable]
   91 |         int line = 0;
      |             ^
src/client/FSCrypt.cc:945:2: error: is this the way to do it? [-Werror,-W#warnings]
  945 | #warning is this the way to do it?

Signed-off-by: Christopher Hoffman <[email protected]>
Fix warnings/errors in ceph API tests that are present in various files
that were introduced by fscrypt feature

src/client/FSCrypt.cc:90:6: error: variable 'olen' set but not used [-Werror,-Wunused-but-set-variable]
   90 |         int olen = 0;
      |             ^
src/client/FSCrypt.cc:91:6: error: variable 'line' set but not used [-Werror,-Wunused-but-set-variable]
   91 |         int line = 0;
      |             ^
src/client/FSCrypt.cc:945:2: error: is this the way to do it? [-Werror,-W#warnings]
  945 | #warning is this the way to do it?
src/client/Client.cc:11850:2: error: read holes [-Werror,-W#warnings]
 11850 | #warning read holes
       |  ^
src/client/Client.cc:11855:2: error: implement file read here [-Werror,-W#warnings]
 11855 | #warning implement file read here
       |  ^
src/client/Inode.cc:847:2: error: need to make sure that we do not skip entire subtree somehow [-Werror,-W#warnings]
  847 | #warning need to make sure that we do not skip entire subtree somehow
      |  ^

Signed-off-by: Christopher Hoffman <[email protected]>
When calling get encryption policy via ioctl, do not display
the hex str of inbuf. This buffer is not used/uninitialized
so no need to display.

Signed-off-by: Christopher Hoffman <[email protected]>
ALL caps are now needed for when setting fscrypt policy

Signed-off-by: Christopher Hoffman <[email protected]>
The libcephfs API call add_fscrypt_key exposes an internal fscrypt
data structure. This is because a hash keyid (of the master key) is used
for calls such as remove_fscrypt_key. Instead of using this structure,
use a char array to obtain keyid.

Fixes: https://tracker.ceph.com/issues/63293
Signed-off-by: Christopher Hoffman <[email protected]>
Once a volume/filesystem is set to use fscrypt encryption, layout
cannot be changed. The configurable ec_profile, sets layout and
will fail.

Fixes: https://tracker.ceph.com/issues/73461
Signed-off-by: Christopher Hoffman <[email protected]>
A max io size can currently be up to INT_MAX. If it is greater,
then clamp the size to INT_MAX. This conflicts with fscrypt io
operations. An fscrypt, op needs to read a whole fscrypt block.
The size of fscrypt block size is 4K, INT_MAX % 4K is not equal
to 0. Therefore, get the nearest multiple of 4K to INT_MAX that
does not go over. In the fscrypt case, this value will be used
for clamping max io size.

Fixes: https://tracker.ceph.com/issues/73346
Signed-off-by: Christopher Hoffman <[email protected]>
When testing large io sizes and clamping that io, consider
fscrypt max io size. This max io size should be a multiple
of 4K (fscrypt block size), but not to exceed INT_MAX.

Signed-off-by: Christopher Hoffman <[email protected]>
The commit 154b867 reverted part of the fix seen in 2b74598.
This commit will reapply any missing changes of the fix.

Fixes: https://tracker.ceph.com/issues/73416
Signed-off-by: Christopher Hoffman <[email protected]>
Fix various python bindings and linting issues that
arose from libcephfs fscrypt testing in pipeline.

Signed-off-by: Christopher Hoffman <[email protected]>
Signed-off-by: Christopher Hoffman <[email protected]>
@chrisphoffman
Copy link
Contributor Author

/config check ok

@vshankar vshankar merged commit 3a08610 into ceph:main Nov 6, 2025
13 of 14 checks passed
@github-project-automation github-project-automation bot moved this from Reviewer approved to Done in Ceph-Dashboard Nov 6, 2025
@github-actions
Copy link

github-actions bot commented Nov 6, 2025

        This is an automated message by src/script/redmine-upkeep.py.

        I found one or more `Fixes:` tags in the commit messages in

        `git log 3a0861039186750b0a2348e29385c46b887a0cee^..3a0861039186750b0a2348e29385c46b887a0cee`

        The referenced tickets are:

        * https://tracker.ceph.com/issues/72192

ideepika pushed a commit to ideepika/ceph that referenced this pull request Nov 25, 2025
Removed ifdef for a failure we encountered during rebase against
case sensitive feature
-ceph#61137 (comment)

Add debug dout when entering WriteEncMgr::read
-ceph#61137 (comment)

Remove FILE_RD mark_caps_dirty
-ceph#61137 (comment)

Add comment to various lines
-ceph#61137 (comment)
-ceph#61137 (comment)
-ceph#61137 (comment)

During write_success mark FILE_WR as dirty
-ceph#61137 (comment)

Signed-off-by: Christopher Hoffman <[email protected]>
JoshuaGabriel pushed a commit to JoshuaGabriel/ceph that referenced this pull request Jan 7, 2026
Removed ifdef for a failure we encountered during rebase against
case sensitive feature
-ceph#61137 (comment)

Add debug dout when entering WriteEncMgr::read
-ceph#61137 (comment)

Remove FILE_RD mark_caps_dirty
-ceph#61137 (comment)

Add comment to various lines
-ceph#61137 (comment)
-ceph#61137 (comment)
-ceph#61137 (comment)

During write_success mark FILE_WR as dirty
-ceph#61137 (comment)

Signed-off-by: Christopher Hoffman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.