Bug #68055
openCeph Object Gateway S3 API ListObjectVersions is not compatible with S3 when using key-marker parameter
0%
Description
When performing the list-object-versions operation, the key-marker parameter in S3 is exclusive, but in RGW, it's inclusive.
I'm testing Ceph on my laptop using the image quay.io/ceph/demo with bucket versioning enabled.
I've already inserted some data
!
!
using aws s3 golang sdk to list object versions, and set key-marker to file-2
endpoint := "http://127.0.0.1:8080"
region := "us-east-1"
accessKeyID := "demo"
secretAccessKey := "demo"
bucket := "demo"
keyMarker := "d5635716-0ecf-4e77-b2fa-18524be999be/file-2"
prefix := "d5635716-0ecf-4e77-b2fa-18524be999be/"
creds := credentials.NewStaticCredentialsProvider(accessKeyID, secretAccessKey, "")
config, err := config.LoadDefaultConfig(context.TODO(),
config.WithRegion(region),
config.WithCredentialsProvider(creds))
if err != nil {
panic("configuration error, " + err.Error())
}
client := s3.NewFromConfig(config, func(o *s3.Options) {
o.BaseEndpoint = aws.String(endpoint)
})
versions, err := client.ListObjectVersions(context.TODO(), &s3.ListObjectVersionsInput{
Bucket: &bucket,
KeyMarker: &keyMarker,
Prefix: &prefix,
})
if err != nil {
panic(err)
}
for _, version := range versions.Versions {
fmt.Println(*version.Key)
}
output:
d5635716-0ecf-4e77-b2fa-18524be999be/file-2 d5635716-0ecf-4e77-b2fa-18524be999be/file-2 d5635716-0ecf-4e77-b2fa-18524be999be/file-3 d5635716-0ecf-4e77-b2fa-18524be999be/file-3 d5635716-0ecf-4e77-b2fa-18524be999be/file4 d5635716-0ecf-4e77-b2fa-18524be999be/file4
The file-2 is included in the output.
when switching the endpoint to Minio or S3, file-2 is not included in the output
Files
Updated by J. Eric Ivancich over 1 year ago
- Tags set to good-first-issue, easy, low-hanging-fruit
The first step might be to write a test for this.
Updated by Samaksh Dhingra over 1 year ago
Updated by Samaksh Dhingra over 1 year ago
Samaksh Dhingra wrote in #note-2:
Hey, I would like to work on this issue. Thanks.
Updated by J. Eric Ivancich over 1 year ago
- Assignee set to Samaksh Dhingra
Thank you @Samaksh Dhingra for agreeing to take on this issue to help get a sense of RGW and its development processes. Feel free to ask questions here or on slack.
Updated by Samaksh Dhingra over 1 year ago
- Status changed from New to In Progress
Updated by Samaksh Dhingra over 1 year ago · Edited
I needed one clarification, what should be the response of the ListObjectVersions API when the key-marker in above example is chosen as 'file4':
Option 1:
{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Name": "demo",
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}
Option2:
{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Versions": [],
"Name": "demo",
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}
Option3:
{
"IsTruncated": false,
"KeyMarker": "file4",
"VersionIdMarker": "",
"Name": "demo",
"Versions": null,
"Prefix": "d5635716-0ecf-4e77-b2fa-18524be999be/",
"MaxKeys": 1000,
"EncodingType": "url"
}
Or any other response?
Updated by Samaksh Dhingra over 1 year ago
Hi Team,
I've created a PR for this issue. This is my first submission so please tell me if I have done something wrong in the process.
https://github.com/ceph/ceph/pull/60068
Thanks and regards,
Samaksh
Updated by Casey Bodley over 1 year ago
- Status changed from In Progress to Fix Under Review
- Backport set to reef squid
- Pull request ID set to 60068
Updated by Casey Bodley about 1 year ago
- Pull request ID changed from 60068 to 60280
Updated by Laura Flores 24 days ago
- Assignee changed from Samaksh Dhingra to Hardika Paliwal
Updated by Hardika Paliwal 23 days ago · Edited
- Status changed from Fix Under Review to In Progress
Hi all,
I see that there was an earlier PR (#60280) for this issue.
I’m going through it now to understand the current status and what’s pending.
I’ll update here once I’ve checked review comments / CI results and propose next steps.
and this is already closed. so I am generating a new PR for this issue
Thanks!
Updated by Hardika Paliwal 23 days ago · Edited
Hi all,
I’m taking over this issue after the earlier PR (#60280) was closed.
I’m currently reproducing the bug and investigating the underlying
marker logic, per review feedback, rather than applying post-filters.
I’ll update once I have a new PR ready.
Thanks!