Ceph » rgw

Is there any plan to fix this in upcoming releases?

It does not silently corrupt objects as far as I can tell, but it does still return a 501 NotImplemented when you try to do a CopyObject with an SSE-C encrypted object -- which is quite frustrating. I'm in the process of adding support for SSE-C to the docker registry project and because of this bug it won't work on my ceph cluster :-(

Does this apply to other SSE modes as well, if it is still a problem? I've run into the same error with SSE-S3 and was curious if there was progress on this or whether to pursue FDE instead.

adam madsen wrote:

Does this apply to other SSE modes as well, if it is still a problem? I've run into the same error with SSE-S3 and was curious if there was progress on this or whether to pursue FDE instead.

this does apply to all flavors of server-side encryption. the low-level copy operation returns this not-implemented error if the source object uses any form of encryption

i believe Marcus does plan to implement this in the near- to medium-term

+1 as it makes encryption useless for applications nowadays using CopyObject API more.

+1 as we plan to force SSE-S3 encryption by adding the `x-amz-server-side-encryption: AES256` header, if not present.

Casey Bodley wrote:

If the source object of a copy operation is encrypted with SSE-C, we should be requiring the x-amz-copy-source-​server-side​-encryption​-customer-* headers necessary to decrypt it, and then apply the x-amz-server-side​-encryption​-customer-* headers (if given) to re-encrypt the target object.

For SSE-KMS, we should also respect the x-amz-server-side-encryption* headers when writing the target object.

Hi team,
Last night I tried to reproduce this issue on Ceph version 18.2.4 and encountered a segmentation fault in RGW.

I created a bucket with S3-SSE enabled using Vault, uploaded a few objects, and then tested the CopyObject API (using either rclone or aws-cli). During the test, the RGW process crashed and the service restarted automatically.

The backtrace information is shown below.

{
    "archived": "2025-10-08 00:57:26.474093",
    "backtrace": [
        "/lib64/libc.so.6(+0x3e6f0) [0x7faad21356f0]",
        "(RGWGetObj_Decompress::handle_data(ceph::buffer::v15_2_0::list&, long, long)+0x177) [0x55f8df37ac37]",
        "(RGWGetObj_BlockDecrypt::process(ceph::buffer::v15_2_0::list&, unsigned long, unsigned long)+0x9a) [0x55f8df4e27ba]",
        "(get_obj_data::flush(rgw::OwningList<rgw::AioResultEntry>&&)+0x7d8) [0x55f8df593768]",
        "(RGWRados::Object::Read::iterate(DoutPrefixProvider const*, long, long, RGWGetDataCB*, optional_yield)+0x2eb) [0x55f8df59b9ab]",
        "(RGWPutObj::get_data(long, long, ceph::buffer::v15_2_0::list&)+0x409) [0x55f8df3ca2b9]",
        "(RGWPutObj::execute(optional_yield)+0xeb0) [0x55f8df3ccb40]",
        "(rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xa72) [0x55f8df279142]",
        "(process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x1039) [0x55f8df27a9c9]",
        "/usr/bin/radosgw(+0xb67806) [0x55f8df9c3806]",
        "/usr/bin/radosgw(+0x3770d1) [0x55f8df1d30d1]",
        "make_fcontext()" 
    ],
    "ceph_version": "18.2.4",
    "crash_id": "2025-10-07T09:59:03.219942Z_b7c6d869-8227-410c-8651-1bac24e35cd1",
    "entity_name": "client.rgw.vng.STOR-CEPH-RGW-07.hrnnwf",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "9",
    "os_version_id": "9",
    "process_name": "radosgw",
    "stack_sig": "7770e86d2f41e472109b089ebc8cb2b8f7fd97a378c2728eba88d5dffe6770d7",
    "timestamp": "2025-10-07T09:59:03.219942Z",
    "utsname_hostname": "STOR-CEPH-RGW-07",
    "utsname_machine": "x86_64",
    "utsname_release": "5.15.0-94-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024" 
}

I'm not certain whether this issue is a known bug in Ceph 18.2.4.
Could you please share if you’ve encountered a similar case or have any insights on this behavior?

Hi team,

I’ve encountered the same crash issue again on Ceph v18.2.7 when a client performs copy object operations on a bucket with encryption enabled.

I tried reproducing the same scenario on v18.2.7 and v18.2.4, but the crash did not occur. In those versions, radosgw denies the request and returns HTTP 501 – Not Implemented instead.

While reviewing related issues, I found one that mentions a problem with copy-object requests where the x-amz-copy-source header is empty (see: https://tracker.ceph.com/issues/72669
).

Could you please advise whether these two issues might be related to each other when triggered by client behavior?

{
    "archived": "2026-01-27 06:40:27.484535",
    "backtrace": [
        "/lib64/libc.so.6(+0x3ebf0) [0x7ffb791abbf0]",
        "(RGWGetObj_Decompress::handle_data(ceph::buffer::v15_2_0::list&, long, long)+0x164) [0x562387a64434]",
        "(RGWGetObj_BlockDecrypt::process(ceph::buffer::v15_2_0::list&, unsigned long, unsigned long)+0x9a) [0x562387bd7c8a]",
        "(RGWGetObj_BlockDecrypt::handle_data(ceph::buffer::v15_2_0::list&, long, long)+0x1ae) [0x562387bde08e]",
        "(get_obj_data::flush(rgw::OwningList<rgw::AioResultEntry>&&)+0x7a8) [0x562387c89448]",
        "(RGWRados::Object::Read::iterate(DoutPrefixProvider const*, long, long, RGWGetDataCB*, optional_yield)+0x303) [0x562387c91793]",
        "(RGWPutObj::get_data(long, long, ceph::buffer::v15_2_0::list&)+0x40d) [0x562387ab5add]",
        "(RGWPutObj::execute(optional_yield)+0xede) [0x562387ab8b5e]",
        "(rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xab2) [0x5623879539d2]",
        "(process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x10a8) [0x56238795b118]",
        "/usr/bin/radosgw(+0xba224f) [0x5623880d324f]",
        "/usr/bin/radosgw(+0x379c81) [0x5623878aac81]",
        "make_fcontext()" 
    ],
    "ceph_version": "18.2.7",
    "crash_id": "2026-01-26T07:15:13.915970Z_fd2fa560-57c3-4d53-98b6-130d56145958",
    "entity_name": "client.rgw.vngv2.VSTOR-CEPH-MBF9-RGW-101.pjrasj",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "9",
    "os_version_id": "9",
    "process_name": "radosgw",
    "stack_sig": "5423ed44f98bb22600a915a94dc7a75ac9903b165c3d45eb70da660a20e37fba",
    "timestamp": "2026-01-26T07:15:13.915970Z",
    "utsname_hostname": "VSTOR-CEPH-RGW-101",
    "utsname_machine": "x86_64",
    "utsname_release": "5.15.0-94-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024" 
}

I have some log from radosgw for crash above

2026-01-26T07:14:50.269+0000 7fa796f6a640 -1 *** Caught signal (Segmentation fault) **
    -6> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 get crypto crypto_isal = 0
    -5> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 load crypto crypto_isal
    -4> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 add crypto crypto_isal 0x5626bf713650
    -3> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 get crypto crypto_isal = 0x5626bf713650
    -2> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 load: crypto crypto_isal loaded and registered
    -1> 2026-01-26T07:14:50.269+0000 7fa796f6a640  1 get crypto crypto_isal = 0x5626bf713650
     0> 2026-01-26T07:14:50.269+0000 7fa796f6a640 -1 *** Caught signal (Segmentation fault) **

Hey Marcus,

do you know if this will make it into the 20.2.1 release?