Wip rgw blake2 by mattbenjamin · Pull Request #30606 · ceph/ceph

mattbenjamin · 2019-09-26T22:56:50Z

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs

cbodley

any ideas for test coverage?

can any of this stuff apply to multipart uploads? if not, we probably want to avoid doing the checksum calculations on the parts themselves. and similarly for Appendable objects

src/common/options.cc

src/rgw/rgw_op.cc

src/common/ceph_crypto.h

src/rgw/rgw_common.h

src/rgw/rgw_op.h

Fast, cryptographic hash functions suitable for block checksums. Signed-off-by: Matt Benjamin <[email protected]>

Adds new Blake2b digest format (native), a concrete type to represent digests, and static_visitor machinery to unify varying checksum computations. Fixes: https://tracker.ceph.com/issues/42080 Signed-off-by: Matt Benjamin <[email protected]>

The actual config-observer instance is static, but initialized from rgw_main. Signed-off-by: Matt Benjamin <[email protected]>

Signed-off-by: Matt Benjamin <[email protected]>

The default copy constructor is invalid (traps), but move assignment/construction is legal. Signed-off-by: Matt Benjamin <[email protected]>

In generic code, lift the update step from the template digest helper in ceph_crypto.h. Use it to remove some buffer::list::c_str() calls in RGW code. Add similar support in rgw::cksum::Digest, which can't call the template forms due to inheritance. Signed-off-by: Matt Benjamin <[email protected]>

adamemerson · 2019-10-23T18:43:25Z

src/common/ceph_crypto.h

+    d.Update((const unsigned char *)p.c_str(), p.length());
+  }
+}
+


I approve of this function.

cbodley · 2019-12-19T15:06:59Z

needs a rebase

stale · 2020-02-17T15:57:06Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

mattbenjamin · 2020-02-17T19:11:53Z

unstale

mattbenjamin · 2020-09-20T20:14:16Z

unstale

stale · 2020-11-22T08:16:46Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

mattbenjamin · 2020-11-22T19:45:58Z

unstale

github-actions · 2020-12-16T17:19:54Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

stale · 2021-07-21T14:00:18Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

mattbenjamin · 2021-07-21T14:01:59Z

let this not be stale

stale · 2022-01-09T08:27:34Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2022-07-07T17:42:11Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

mattbenjamin · 2022-07-07T17:57:04Z

oh noes!

github-actions · 2022-09-07T00:04:22Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2022-10-28T21:01:49Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

mattbenjamin added feature rgw labels Sep 26, 2019

mattbenjamin requested a review from cbodley September 26, 2019 22:56

mattbenjamin self-assigned this Sep 27, 2019

mattbenjamin requested a review from adamemerson September 27, 2019 13:48

mattbenjamin force-pushed the wip-rgw-blake2 branch from 9f226b7 to f290b2c Compare September 27, 2019 13:59

sebastian-philipp mentioned this pull request Sep 27, 2019

ceph-daemon: deploy ceph daemons with podman and systemd #30603

Merged

13 tasks

cbodley reviewed Sep 27, 2019

View reviewed changes

mattbenjamin added 14 commits October 15, 2019 15:04

common/rgw: add BLAKE2 submodule

139abea

Fast, cryptographic hash functions suitable for block checksums. Signed-off-by: Matt Benjamin <[email protected]>

rgw: add config-observer for rgw_request.h

dce72b2

The actual config-observer instance is static, but initialized from rgw_main. Signed-off-by: Matt Benjamin <[email protected]>

rgw: add rgw_request.cc

43b7f75

Signed-off-by: Matt Benjamin <[email protected]>

rgw: add rgw_cksum_type config option

7b5cf01

Signed-off-by: Matt Benjamin <[email protected]>

rgw: thread checksums through create-bucket and put-object

9571677

Signed-off-by: Matt Benjamin <[email protected]>

rgw: thread rgw_cksum through librgw/rgw_file

d465b30

Signed-off-by: Matt Benjamin <[email protected]>

rgw: add chksum to swift headers as X-Rgw-Cksum

31d87ac

Signed-off-by: Matt Benjamin <[email protected]>

rgw: thread rgw_cksum through RGWGetObj and S3 GET/HEAD

0867735

Signed-off-by: Matt Benjamin <[email protected]>

rgw: rgw::request::request_config fix linkage

f20d7e2

Signed-off-by: Matt Benjamin <[email protected]>

rgw: add cksum to radosgw-admin bucket stats

681f13c

Signed-off-by: Matt Benjamin <[email protected]>

rgw: add local rgw_cksum unit test

c03b4f8

Signed-off-by: Matt Benjamin <[email protected]>

common/ceph_crypto: forbid copy, allow move OpenSSLDigest

210ee97

The default copy constructor is invalid (traps), but move assignment/construction is legal. Signed-off-by: Matt Benjamin <[email protected]>

mattbenjamin force-pushed the wip-rgw-blake2 branch from f290b2c to c2f9b91 Compare October 16, 2019 16:31

adamemerson reviewed Oct 23, 2019

View reviewed changes

src/common/ceph_crypto.h

d.Update((const unsigned char *)p.c_str(), p.length());

}

}

Copy link

Contributor

adamemerson Oct 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve of this function.

stale bot added the stale label Feb 17, 2020

stale bot removed the stale label Feb 17, 2020

mattbenjamin mentioned this pull request Feb 24, 2020

RGW:Multisite: Verify if the synced object is identical to source #33492

Merged

3 tasks

stale bot closed this Sep 20, 2020

mattbenjamin reopened this Sep 20, 2020

stale bot removed the stale label Sep 20, 2020

stale bot added the stale label Nov 22, 2020

mattbenjamin removed the stale label Nov 22, 2020

github-actions bot added the needs-rebase label Dec 16, 2020

stale bot added the stale label Jul 21, 2021

stale bot removed the stale label Jul 21, 2021

stale bot added the stale label Jan 9, 2022

github-actions bot closed this Jul 7, 2022

mattbenjamin reopened this Jul 7, 2022

github-actions bot added build/ops common tests and removed stale labels Jul 7, 2022

djgalloway changed the base branch from master to main July 9, 2022 00:00

github-actions bot added the stale label Sep 7, 2022

github-actions bot closed this Oct 28, 2022

mattbenjamin mentioned this pull request Feb 3, 2023

rgw: add s3 checksum crc32 and sha1 #49986

Closed

14 tasks

Comments

Conversation

mattbenjamin commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamemerson Oct 23, 2019

Choose a reason for hiding this comment

Uh oh!

cbodley commented Dec 19, 2019

Uh oh!

stale bot commented Feb 17, 2020

Uh oh!

mattbenjamin commented Feb 17, 2020

Uh oh!

mattbenjamin commented Sep 20, 2020

Uh oh!

stale bot commented Nov 22, 2020

Uh oh!

mattbenjamin commented Nov 22, 2020

Uh oh!

github-actions bot commented Dec 16, 2020

Uh oh!

stale bot commented Jul 21, 2021

Uh oh!

mattbenjamin commented Jul 21, 2021

Uh oh!

stale bot commented Jan 9, 2022

Uh oh!

github-actions bot commented Jul 7, 2022

Uh oh!

mattbenjamin commented Jul 7, 2022

Uh oh!

github-actions bot commented Sep 7, 2022

Uh oh!

github-actions bot commented Oct 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mattbenjamin commented Sep 26, 2019 •

edited

Loading