Skip to content

Comments

rgw: implement GetObjectAttributes#55259

Merged
mattbenjamin merged 14 commits intoceph:mainfrom
linuxbox2:wip-object-attributes
Jan 10, 2025
Merged

rgw: implement GetObjectAttributes#55259
mattbenjamin merged 14 commits intoceph:mainfrom
linuxbox2:wip-object-attributes

Conversation

@mattbenjamin
Copy link
Contributor

@mattbenjamin mattbenjamin commented Jan 20, 2024 •
edited
Loading

Fixes: https://tracker.ceph.com/issues/64109

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@mattbenjamin mattbenjamin self-assigned this Jan 20, 2024
@mattbenjamin mattbenjamin requested a review from a team as a code owner January 20, 2024 22:44
@github-actions github-actions bot added the rgw label Jan 20, 2024
dang
dang approved these changes Jan 22, 2024
View reviewed changes
cbodley
cbodley reviewed Jan 22, 2024
View reviewed changes
cbodley
cbodley reviewed Jan 22, 2024
View reviewed changes
@github-actions
Copy link

github-actions bot commented Mar 24, 2024

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Mar 24, 2024
@github-actions
Copy link

github-actions bot commented Apr 12, 2024

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@mattbenjamin
Copy link
Contributor Author

mattbenjamin commented Apr 12, 2024

(will rebase this next week)

@github-actions github-actions bot removed the stale label Apr 12, 2024
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from dbd921f to 428d55d Compare May 11, 2024 13:42
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from 428d55d to 4e7859a Compare July 5, 2024 16:52
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch 3 times, most recently from 992c0af to 044270b Compare July 5, 2024 20:54
@mattbenjamin mattbenjamin requested review from dang July 5, 2024 20:58
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch 2 times, most recently from cfd4b60 to ec20bca Compare July 6, 2024 19:50
@mattbenjamin
Copy link
Contributor Author

mattbenjamin commented Jul 7, 2024

in-progress s3-test case ceph/s3-tests#571

@ivancich
Copy link
Member

ivancich commented Jan 3, 2025

jenkins test make check

@ivancich
Copy link
Member

ivancich commented Jan 3, 2025

jenkins test make check arm64

@ivancich
Copy link
Member

ivancich commented Jan 3, 2025

@mattbenjamin This passed QA but there are some CI issues that prevent merging at this moment.

@ivancich ivancich added passed-qa for PRs that have passed QA but there's an on-going conversation, so merging would be premature and removed needs-qa wip-eric-testing-2 for ivancich testing labels Jan 3, 2025
mattbenjamin and others added 13 commits January 7, 2025 19:16
@mattbenjamin
rgw: implement GetObjectAttributes
eb94939 
Implements the corresponding S3 operation, and
introduces a new Object::list_parts SAL interface to support it.

Includes Casey Bodley <[email protected]>:
  * use uncompressed part size
  * local variable shadowed a member variable and broke handling of
      PartNumberMarker in request and response

Fixes: https://tracker.ceph.com/issues/64109

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw_cksum: zero-initialize GetObjAttrs integral members
ed6134c 
(code review)

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw: GetObjectAttributes iam policy policy hooks
f73e83a 
Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw_cksum: don't display nil checksums in get-object-attributes (parts)
b164ad8 
Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw: defensive check for part cksum in try_sum_part_cksums
5c4d594 
If a create-multipart-upload requested additional checksum but a
part upload omits it, fail gracefully.

Possibly we can fixup the checksum uploads and remove this scenario,
but coding defensively would still be correct.

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw_cksum: permit fallback to checksum-type from create-multipart, in…
bac02fc 
… upload-part

There appear to be workloads that provide a checksum algorithm in
create-multipart-upload, but do not provide (what must be) the
corresponding algorithm when uploading the parts.  (complete-multipart-upload
has no checksum argument, the value is implicit.)

This behavior is inconsistent with at least some SDKs, but it is
possibly accepted behavior in AWS S3, and is not logically necessary,
since the originally supplied checksum type is already known.

Therefore, change the behavior of upload-part to fall back to a
checksum type that was sent with the corresponding create-multipart-upload
request, if one is present, rather than failing with InvalidRequest.

Fixes: https://tracker.ceph.com/issues/68513

Signed-off-by: Matt Benjamin <[email protected]>
(cherry picked from commit 6b487a4)
@mattbenjamin
rgw_cksum: return version_id in all cases
1be4399 
The version_id member in RGWGetObjAttrs shadows the one in
RGWGetObj.

Just assign it before conditionally returning the version-id header
in RGWGetObjAttrs_ObjStore_S3::send_response().

Signed-off-by: Matt Benjamin <[email protected]>
@cbodley @mattbenjamin
rgw: GetObjAttrs calls rgw_s3_prepare_decrypt()
56bf507 
RGWGetObj::execute() relies on the virtual get_decrypt_filter() which is
overridden by RGWGetObj_ObjStore_S3

but when called from RGWGetObjAttrs_ObjStore_S3, there is no override.
add one that just calls rgw_s3_prepare_decrypt()

Signed-off-by: Casey Bodley <[email protected]>
@cbodley @mattbenjamin
rgw/rados: get_part_obj_state() fixes accounted_size when uncompressed
cbcf0e3 
the part head objects don't have a RGW_ATTR_MANIFEST attribute, so
get_obj_state_impl() isn't able to set the correct
RGWObjState::accounted_size unless RGW_ATTR_COMPRESSION provides one

get_part_obj_state() builds a fake manifest in memory to represent the
part and updates state.size accordingly, but it hadn't corrected the
value of state.accounted_size

Signed-off-by: Casey Bodley <[email protected]>
@cbodley @mattbenjamin
rgw/rados: Object::list_parts() starts after marker
f6bc276 
Signed-off-by: Casey Bodley <[email protected]>
@cbodley @mattbenjamin
rgw: GetObjAttrs enforces MaxSize and returns correct IsTruncated
b246233 
Signed-off-by: Casey Bodley <[email protected]>
@mattbenjamin
rgw:attrs cap max_parts at 1000
2b8689e 
Suggested by Casey.  This is a relatively small value, but let's
be conservative.

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin
rgw:attrs: dump marker and max_parts only if sent
6c534ba 
Suggested by Casey.

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from bb59073 to 6c534ba Compare January 8, 2025 01:58
cbodley
cbodley reviewed Jan 9, 2025
View reviewed changes
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from 2404fa1 to d4f52a5 Compare January 9, 2025 20:20
@mattbenjamin
Copy link
Contributor Author

mattbenjamin commented Jan 9, 2025

jenkins retest this please

@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from d4f52a5 to 7c0ae60 Compare January 9, 2025 22:40
@mattbenjamin
Copy link
Contributor Author

mattbenjamin commented Jan 9, 2025 •
edited
Loading

maybe I am being fooled by the github interface;
nope, it seems to be stalled

@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from 7c0ae60 to 4b1984f Compare January 10, 2025 16:45
@mattbenjamin
rgw_attrs: fixes for 3 more iam/perms mappings
8f5f3fa 
Help from Pritha and Casey appreciated

Plus, fix 3 explicit mapping tests in test_rgw_iam_policy.cc.

Signed-off-by: Matt Benjamin <[email protected]>
@mattbenjamin mattbenjamin force-pushed the wip-object-attributes branch from 4b1984f to 8f5f3fa Compare January 10, 2025 20:13
@mattbenjamin mattbenjamin merged commit a75cf5c into ceph:main Jan 10, 2025
12 checks passed
@cbodley cbodley mentioned this pull request Apr 23, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cbodley cbodley cbodley approved these changes

@pritha-srivastava pritha-srivastava pritha-srivastava left review comments

@dang dang Awaiting requested review from dang

+1 more reviewer

@ismaelpuerto ismaelpuerto ismaelpuerto approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@mattbenjamin mattbenjamin

Labels

bug-fix build/ops feature passed-qa for PRs that have passed QA but there's an on-going conversation, so merging would be premature rgw tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@mattbenjamin @cbodley @ivancich @dang @ismaelpuerto @pritha-srivastava