Project

General

Profile

Actions

Bug #68055

open

Ceph Object Gateway S3 API ListObjectVersions is not compatible with S3 when using key-marker parameter

Added by meteorgan gan over 1 year ago. Updated 23 days ago.

Status:
In Progress
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Backport:
reef squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

When performing the list-object-versions operation, the key-marker parameter in S3 is exclusive, but in RGW, it's inclusive.

I'm testing Ceph on my laptop using the image quay.io/ceph/demo with bucket versioning enabled.
I've already inserted some data
!

!

using aws s3 golang sdk to list object versions, and set key-marker to file-2

    endpoint := "http://127.0.0.1:8080" 
    region := "us-east-1" 
    accessKeyID := "demo" 
    secretAccessKey := "demo" 
    bucket := "demo" 
    keyMarker := "d5635716-0ecf-4e77-b2fa-18524be999be/file-2" 
    prefix := "d5635716-0ecf-4e77-b2fa-18524be999be/" 

    creds := credentials.NewStaticCredentialsProvider(accessKeyID, secretAccessKey, "")
    config, err := config.LoadDefaultConfig(context.TODO(),
        config.WithRegion(region),
        config.WithCredentialsProvider(creds))
    if err != nil {
        panic("configuration error, " + err.Error())
    }

    client := s3.NewFromConfig(config, func(o *s3.Options) {
        o.BaseEndpoint = aws.String(endpoint)
    })
    versions, err := client.ListObjectVersions(context.TODO(), &s3.ListObjectVersionsInput{
        Bucket:    &bucket,
        KeyMarker: &keyMarker,
        Prefix:    &prefix,
    })
    if err != nil {
        panic(err)
    }
    for _, version := range versions.Versions {
        fmt.Println(*version.Key)
    }

output:

d5635716-0ecf-4e77-b2fa-18524be999be/file-2
d5635716-0ecf-4e77-b2fa-18524be999be/file-2
d5635716-0ecf-4e77-b2fa-18524be999be/file-3
d5635716-0ecf-4e77-b2fa-18524be999be/file-3
d5635716-0ecf-4e77-b2fa-18524be999be/file4
d5635716-0ecf-4e77-b2fa-18524be999be/file4

The file-2 is included in the output.
when switching the endpoint to Minio or S3, file-2 is not included in the output


Files

clipboard-202409122250-dscvs.png (244 KB) clipboard-202409122250-dscvs.png meteorgan gan, 09/12/2024 02:50 PM
Actions #1

Updated by J. Eric Ivancich over 1 year ago

  • Tags set to good-first-issue, easy, low-hanging-fruit

The first step might be to write a test for this.

Actions #2

Updated by Samaksh Dhingra over 1 year ago

J. Eric Ivancich wrote in #note-1:

The first step might be to write a test for this.

J. Eric Ivancich wrote in #note-1:

The first step might be to write a test for this.

Hey, I would like to work on this issue. Thanks.

Actions #3

Updated by Samaksh Dhingra over 1 year ago

Samaksh Dhingra wrote in #note-2:

Hey, I would like to work on this issue. Thanks.

Actions #4

Updated by J. Eric Ivancich over 1 year ago

  • Assignee set to Samaksh Dhingra

Thank you @Samaksh Dhingra for agreeing to take on this issue to help get a sense of RGW and its development processes. Feel free to ask questions here or on slack.

Actions #5

Updated by Samaksh Dhingra over 1 year ago

  • Status changed from New to In Progress
Actions #6

Updated by Samaksh Dhingra over 1 year ago · Edited

I needed one clarification, what should be the response of the ListObjectVersions API when the key-marker in above example is chosen as 'file4':

Option 1:

{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Name": "demo",
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}

Option2:

{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Versions": [],
"Name": "demo",
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}

Option3:

{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Name": "demo",
"Versions": null,
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}

Or any other response?

Actions #7

Updated by Samaksh Dhingra over 1 year ago

Hi Team,

I've created a PR for this issue. This is my first submission so please tell me if I have done something wrong in the process.
https://github.com/ceph/ceph/pull/60068

Thanks and regards,
Samaksh

Actions #8

Updated by Casey Bodley over 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to reef squid
  • Pull request ID set to 60068
Actions #9

Updated by Casey Bodley about 1 year ago

  • Pull request ID changed from 60068 to 60280
Actions #10

Updated by Laura Flores 24 days ago

  • Assignee changed from Samaksh Dhingra to Hardika Paliwal
Actions #11

Updated by Hardika Paliwal 23 days ago · Edited

  • Status changed from Fix Under Review to In Progress

Hi all,

I see that there was an earlier PR (#60280) for this issue.

I’m going through it now to understand the current status and what’s pending.

I’ll update here once I’ve checked review comments / CI results and propose next steps.

and this is already closed. so I am generating a new PR for this issue

Thanks!

Actions #12

Updated by Hardika Paliwal 23 days ago · Edited

Hi all,

I’m taking over this issue after the earlier PR (#60280) was closed.
I’m currently reproducing the bug and investigating the underlying
marker logic, per review feedback, rather than applying post-filters.

I’ll update once I have a new PR ready.

Thanks!

Actions

Also available in: Atom PDF