-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Add image encryption/decryption code #3001
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
f64b250 to
c302932
Compare
Codecov Report
@@ Coverage Diff @@
## master #3001 +/- ##
==========================================
- Coverage 44.85% 39.76% -5.09%
==========================================
Files 113 84 -29
Lines 12326 11172 -1154
==========================================
- Hits 5529 4443 -1086
- Misses 5949 6089 +140
+ Partials 848 640 -208
Continue to review full report at Codecov.
|
5d64d62 to
0f1dd38
Compare
content/helpers.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bytes.Reader already providers a ReaderAt interface, and Size on a byte slice, so this seems unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dropped this patch now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, I 'downgraded' the parameters to the encryption function to not take a content.ReaderAt but a io.ReaderAt instead.
In a tree that has all the image encryption code I tried to then convert the io.ReaderAt to a plain io.Reader but that doesn't fit with the function for reading (layer blobs) from the content.Store since this one returns a ReaderAt interface via the content.Provider interface. I suppose we can leave it like this unless we somehow wrap that content.ReaderAt with some sort of helper that makes it looks like io.Reader interface.
images/encryption/gpgvault.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems likely that x/crypto/openpgp will be deprecated, see golang/go#30141 so really not sure about using it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think they reverted on that decision 3 days ago :)
0f1dd38 to
b3206d8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the nonce base64 encoded at this point? It seems weird to me to make CipherOptions a string map and put the decoding everywhere it is used, rather than putting the keys in explicitly, in which case Go will convert byte types from base64 automatically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CipherOptions is a map[string]string but the Nonce is a []byte. It looks like that map only carries the Nonce at the moment, so it could be converted to map[string][]byte and we probably wouldn't have to do the conversion then...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I converted the map to map[string][]byte, which resolve the issue: d647440
images/encryption/encryption.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The size should not be hard coded. Miscreant provides a GenerateKey function https://godoc.org/github.com/miscreant/miscreant-go#GenerateKey
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah again so don't use generateKey but the size should be got from somewhere, as it is dependent on the encryption function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a patch on top of this series that resolves the issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Miscreatnt provides a GenerateNonce function https://godoc.org/github.com/miscreant/miscreant-go#GenerateNonce
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, we are using NewStreamEncryptor which returns us
type StreamEncryptor struct {
// cipher.AEAD instance underlying this STREAM
a cipher.AEAD
// Nonce encoder instance which computes per-message nonces
n *nonceEncoder32
}
I am missing a function that gives me access to 'a'.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... might as well raise an issue with the miscreant project
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah sorry I was misreading the API docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we leave it as it is now ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo "invocation"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
images/encryption/config/config.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this comment is too pgp specific
images/encryption/encryption.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Golang JSON already marshalls and unmarshals bytes to base64 encoded strings, so this should not be necessary at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I kept the two functions but simplified them: c05cb18
images/encryption/encryption.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rand.Read uses io.ReadFull itself see https://golang.org/pkg/crypto/rand/#Read so you just need to use that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling miscreant's GenerateKey() now, which is good too since it checks for proper size...
|
They layer encryption part is getting into shape now, but I have a lot of concerns over the key wrapping, which is not really conforming to best practises, and also is not possible to implement in a FIPS compliant way (PGP uses non approved ciphers). I wonder if we could split this into the two parts? |
Are you concerned about the JSON that is holding the layer decryption key, nonce, and cipher algo parameters we are encrypting using JWE, PKCS7, and OpenPGP? @justincormack |
@justincormack ... or are you concerned about the usage of OpenPGP, JWE, and PKCS#7? |
|
I split off the symmetric key code to this PR now: #3047 |
estesp
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did a full pass through this PR; few comments on Golang import formatting, typo fix, and a few other comments.
images/encryption/utils.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any reason this one function shouldn't just go into your utils package (next file)? or conversely any reason the utility functions in the separate encryption/utils pkg shouldn't just end up in this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function Uint64ToStringArray in this file only has one user in images/encryption/gpg.go and could either move in there or we can merge it into images/encryption/utils/utils.go where there are more crypto related functions, though. Either way is fine by me. If you don't have a preference I'd probably move it into gpg.go.
|
Can you give us a quick up and running |
|
@crosbymichael looks like the request to break it up lost the overall initial PR description which was fairly extensive; see #2532 initial comment for demo/usage, etc. Will probably want an additional PR for usage docs when this is all ready for merge. |
547f71a to
51d2596
Compare
@crosbymichael Created a draft document! Much more details in the PR like @estesp mentioned, but here's the quick rundown: https://github.com/lumjjb/containerd/blob/encrypt-doc/docs/encryption.md |
51d2596 to
2c1e918
Compare
|
What are the next steps for this PR? Believe that all the feedback has been addressed. |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
6638292 to
69a6056
Compare
|
Build succeeded.
|
69a6056 to
063a171
Compare
|
Build succeeded.
|
We have to add golang.org/x/crypto/{cast5,openpgp} to make vndr happy.
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Add a ReaderDigester interface that we will use for calculating the digest on a layer that we are decryting while reading its plain data from a Reader. The digest will be used in the OCI Descriptor for identifying the layer. Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]> Signed-off-by: Brandon Lum <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Use a field keylen that represents the number of bytes of the key rather than the number of bits as we had it before. Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
We are only supporting adding recipients, so we can remove the Operation field from the EncryptConfig. Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Brandon Lum <[email protected]>
063a171 to
26c7edb
Compare
|
Build succeeded.
|
Signed-off-by: Brandon Lum <[email protected]>
|
Build succeeded.
|
|
Could we close this in favor of #3134, Thanks! |
|
Closing per request |
This PR adds the image encryption and decryption code. We add three 3rd party dependencies. We also add dependencies on code in
content/that implements a wrapper of the ReaderAt interface for a byte array, which we use for test cases. Beside that we define a ReaderDigest interface that we use for implementing a Reader interface for decryption that is also digesting the data being read. We use this interface for reading decrypted layers and getting the digest of the decrypted layer at the end.Stefan