Skip to content

Conversation

@stefanberger
Copy link
Contributor

This PR adds the image encryption and decryption code. We add three 3rd party dependencies. We also add dependencies on code in content/ that implements a wrapper of the ReaderAt interface for a byte array, which we use for test cases. Beside that we define a ReaderDigest interface that we use for implementing a Reader interface for decryption that is also digesting the data being read. We use this interface for reading decrypted layers and getting the digest of the decrypted layer at the end.

Stefan

@codecov-io
Copy link

codecov-io commented Feb 11, 2019

Codecov Report

Merging #3001 into master will decrease coverage by 5.08%.
The diff coverage is 34.07%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3001      +/-   ##
==========================================
- Coverage   44.85%   39.76%   -5.09%     
==========================================
  Files         113       84      -29     
  Lines       12326    11172    -1154     
==========================================
- Hits         5529     4443    -1086     
- Misses       5949     6089     +140     
+ Partials      848      640     -208
Flag Coverage Δ
#linux ?
#windows 39.76% <34.07%> (-0.48%) ⬇️
Impacted Files Coverage Δ
images/encryption/gpg.go 0% <0%> (ø)
images/encryption/gpgvault.go 0% <0%> (ø)
images/encryption/keywrap/pgp/keywrapper_gpg.go 35% <35%> (ø)
...mages/encryption/keywrap/pkcs7/keywrapper_pkcs7.go 44.61% <44.61%> (ø)
images/encryption/encryption.go 50% <50%> (ø)
images/encryption/keywrap/jwe/keywrapper_jwe.go 52.77% <52.77%> (ø)
images/encryption/blockcipher/blockcipher.go 69.44% <69.44%> (ø)
images/encryption/blockcipher/blockcipher_siv.go 79.46% <79.46%> (ø)
snapshots/native/native.go 1.79% <0%> (-41.26%) ⬇️
archive/tar.go 16.99% <0%> (-26.8%) ⬇️
... and 50 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04e7747...16eecb2. Read the comment docs.

@stefanberger stefanberger force-pushed the encryption_code.pr branch 3 times, most recently from 5d64d62 to 0f1dd38 Compare February 13, 2019 18:16
@stefanberger
Copy link
Contributor Author

@justincormack @dmcgowan

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes.Reader already providers a ReaderAt interface, and Size on a byte slice, so this seems unnecessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped this patch now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I 'downgraded' the parameters to the encryption function to not take a content.ReaderAt but a io.ReaderAt instead.

In a tree that has all the image encryption code I tried to then convert the io.ReaderAt to a plain io.Reader but that doesn't fit with the function for reading (layer blobs) from the content.Store since this one returns a ReaderAt interface via the content.Provider interface. I suppose we can leave it like this unless we somehow wrap that content.ReaderAt with some sort of helper that makes it looks like io.Reader interface.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems likely that x/crypto/openpgp will be deprecated, see golang/go#30141 so really not sure about using it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they reverted on that decision 3 days ago :)

golang/go#30141 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the nonce base64 encoded at this point? It seems weird to me to make CipherOptions a string map and put the decoding everywhere it is used, rather than putting the keys in explicitly, in which case Go will convert byte types from base64 automatically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CipherOptions is a map[string]string but the Nonce is a []byte. It looks like that map only carries the Nonce at the moment, so it could be converted to map[string][]byte and we probably wouldn't have to do the conversion then...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I converted the map to map[string][]byte, which resolve the issue: d647440

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah again so don't use generateKey but the size should be got from somewhere, as it is dependent on the encryption function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a patch on top of this series that resolves the issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@stefanberger stefanberger Feb 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, we are using NewStreamEncryptor which returns us

type StreamEncryptor struct {
	// cipher.AEAD instance underlying this STREAM
	a cipher.AEAD

	// Nonce encoder instance which computes per-message nonces
	n *nonceEncoder32
}

https://github.com/miscreant/miscreant.go/blob/master/stream.go#L25

I am missing a function that gives me access to 'a'.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... might as well raise an issue with the miscreant project

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry I was misreading the API docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we leave it as it is now ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo "invocation"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment is too pgp specific

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Golang JSON already marshalls and unmarshals bytes to base64 encoded strings, so this should not be necessary at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept the two functions but simplified them: c05cb18

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rand.Read uses io.ReadFull itself see https://golang.org/pkg/crypto/rand/#Read so you just need to use that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling miscreant's GenerateKey() now, which is good too since it checks for proper size...

stefanberger@9f28224

@justincormack
Copy link
Contributor

They layer encryption part is getting into shape now, but I have a lot of concerns over the key wrapping, which is not really conforming to best practises, and also is not possible to implement in a FIPS compliant way (PGP uses non approved ciphers). I wonder if we could split this into the two parts?

@stefanberger
Copy link
Contributor Author

stefanberger commented Feb 26, 2019

They layer encryption part is getting into shape now, but I have a lot of concerns over the key wrapping, which is not really conforming to best practises, and also is not possible to implement in a FIPS compliant way (PGP uses non approved ciphers). I wonder if we could split this into the two parts?

Are you concerned about the JSON that is holding the layer decryption key, nonce, and cipher algo parameters we are encrypting using JWE, PKCS7, and OpenPGP? @justincormack

@stefanberger
Copy link
Contributor Author

They layer encryption part is getting into shape now, but I have a lot of concerns over the key wrapping, which is not really conforming to best practises, and also is not possible to implement in a FIPS compliant way (PGP uses non approved ciphers). I wonder if we could split this into the two parts?

@justincormack ... or are you concerned about the usage of OpenPGP, JWE, and PKCS#7?

@stefanberger
Copy link
Contributor Author

I split off the symmetric key code to this PR now: #3047

Copy link
Member

@estesp estesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a full pass through this PR; few comments on Golang import formatting, typo fix, and a few other comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason this one function shouldn't just go into your utils package (next file)? or conversely any reason the utility functions in the separate encryption/utils pkg shouldn't just end up in this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function Uint64ToStringArray in this file only has one user in images/encryption/gpg.go and could either move in there or we can merge it into images/encryption/utils/utils.go where there are more crypto related functions, though. Either way is fine by me. If you don't have a preference I'd probably move it into gpg.go.

@crosbymichael
Copy link
Member

Can you give us a quick up and running .md file for using this. It will be helpful for someone like me to test this and for users on how to enable this feature?

@estesp
Copy link
Member

estesp commented Mar 22, 2019

@crosbymichael looks like the request to break it up lost the overall initial PR description which was fairly extensive; see #2532 initial comment for demo/usage, etc. Will probably want an additional PR for usage docs when this is all ready for merge.

@lumjjb
Copy link
Contributor

lumjjb commented Mar 25, 2019

Can you give us a quick up and running .md file for using this. It will be helpful for someone like me to test this and for users on how to enable this feature?

@crosbymichael Created a draft document! Much more details in the PR like @estesp mentioned, but here's the quick rundown: https://github.com/lumjjb/containerd/blob/encrypt-doc/docs/encryption.md

@lumjjb
Copy link
Contributor

lumjjb commented Apr 5, 2019

What are the next steps for this PR? Believe that all the feedback has been addressed.

@theopenlab-ci
Copy link

theopenlab-ci bot commented Jun 4, 2019

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

@lumjjb lumjjb force-pushed the encryption_code.pr branch from 6638292 to 69a6056 Compare June 4, 2019 17:43
@theopenlab-ci
Copy link

theopenlab-ci bot commented Jun 4, 2019

Build succeeded.

@lumjjb lumjjb force-pushed the encryption_code.pr branch from 69a6056 to 063a171 Compare June 4, 2019 19:38
@theopenlab-ci
Copy link

theopenlab-ci bot commented Jun 4, 2019

Build succeeded.

stefanberger and others added 17 commits June 4, 2019 15:49
We have to add golang.org/x/crypto/{cast5,openpgp} to make vndr happy.

Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Add a ReaderDigester interface that we will use for calculating
the digest on a layer that we are decryting while reading its
plain data from a Reader. The digest will be used in the OCI
Descriptor for identifying the layer.

Signed-off-by: Stefan Berger <[email protected]>
Signed-off-by: Stefan Berger <[email protected]>
Use a field keylen that represents the number of bytes of the key
rather than the number of bits as we had it before.

Signed-off-by: Stefan Berger <[email protected]>
We are only supporting adding recipients, so we can remove the Operation
field from the EncryptConfig.

Signed-off-by: Stefan Berger <[email protected]>
@lumjjb lumjjb force-pushed the encryption_code.pr branch from 063a171 to 26c7edb Compare June 4, 2019 19:50
@theopenlab-ci
Copy link

theopenlab-ci bot commented Jun 4, 2019

Build succeeded.

Signed-off-by: Brandon Lum <[email protected]>
@theopenlab-ci
Copy link

theopenlab-ci bot commented Jun 4, 2019

Build succeeded.

@lumjjb
Copy link
Contributor

lumjjb commented Jul 8, 2019

Could we close this in favor of #3134, Thanks!

@estesp
Copy link
Member

estesp commented Jul 8, 2019

Closing per request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants