Add a basic SBOM workflow (closes #1838)#1854
Merged
o1oo11oo merged 15 commits intosecureCodeBox:mainfrom Oct 5, 2023
Merged
Conversation
o1oo11oo
commented
Aug 9, 2023
32fadd6 to
2a56f71
Compare
Contributor
Author
|
Edit: irrelevant at this point because |
o1oo11oo
commented
Aug 9, 2023
2a56f71 to
c504b63
Compare
4d801b1 to
ed57477
Compare
✅ Deploy Preview for docs-securecodebox canceled.
|
J12934
reviewed
Sep 14, 2023
Member
J12934
left a comment
There was a problem hiding this comment.
very nice work 👏
just some minor comments.
tested it out locally and worked perfectly on the first try :)
This was referenced Sep 15, 2023
added 10 commits
September 18, 2023 14:09
Add a new ScanType to trivy, that allows generating CycloneDX SBOMs from container images. This is part of work to integrate an SBOM workflow into the secureCodeBox. At the moment the SBOMs are only generated and uploaded to storage, nothing else happens to them. Note that this is a different result type than the other trivy scans, this uses "sbom-cyclonedx", therefore the normal trivy parser will not run for these scans. Signed-off-by: Lukas Fischer <[email protected]>
This is a bit tricky, because now trivy can generate two different kinds of scan results, normal trivy-json results and sbom-cyclonedx results. To keep these parsers separated, an additional CycloneDX parser is added. To not overcomplicate the structure, this parser is added in the trivy scanner directory. If in the future there are multiple scanners creating SBOMs, it can be moved somewhere else. Adding a second parser for a scanner requires some Makefile trickery. Usually the docker-build, docker-export, kind-import and deploy Makefile targets are used to build and deploy a scanner. Adapting the first three targets is easy, the additional parser just needs an extra build step wich can be added as another dependency. The deploy is more tricky because it is a single helm upgrade --install command, which sets the necessary values for docker repository and tag. This is achieved by redefining the Makefile target, but it is not the nicest solution, make warns about the target getting overridden. The parser is a generic cyclonedx parser (although with currently limited functionality), but currently lives in the trivy scanner directory because it is only used here and means the global structure does not have to get changed. Because it is not trivy-specific, it is not supposed to use "trivy" in its name/repository. This means, the common-... Makefile targets cannot always be used, and if they can, they have to be used in unintended ways. By default they get used by setting the module variable to the directory name of what is built and then add the name of the scanner in the end, for example the parser for trivy ends up being built as parser-trivy from the parser directory. Now to prevent the CycloneDX parser getting named parser-cyclonedx-trivy, the docker-build-cyclonedx-parser target cannot use the common-docker-build and the export/import targets have to be a bit creative with using the module and name variables. This might get reworked to make the Makefiles nicer again. Signed-off-by: Lukas Fischer <[email protected]>
Basic functionality of a persistence hook, that sends CycloneDX SBOMs to OWASP Dependency-Track. Since hooks run for all scans, the hook first checks the scanType and then the bomFormat property to see if the rawResult is actually a CycloneDX SBOM. If it is it gets uploaded to Dependency-Track. To assign an SBOM to a project, either the project UUID or the name and version are required. The hook tries to grab the name and version from the name property of the SBOM and automatically creates missing projects. This is not the nicest way to get name and version, but there is no better property in the SBOM to read it from. It works for container scans, but probably breaks for other kinds of SBOM scans. Signed-off-by: Lukas Fischer <[email protected]>
Expand the comments in the trivy Makefile to explain more in detail why the changes are necessary to build the the CycloneDX parser as well. Signed-off-by: Lukas Fischer <[email protected]>
Make sure the persistence-dependencytrack hook can be tested automatically by providing some tests for it. Some parts of the tests might be coupled a bit tightly to the actual implementation, if the endpoint is switched to the PUT endpoint the tests will have to be adapted. The tests inject the mocked dependencies through parameters, similar to how the tests for the generic webhook work. Since fetch() is the only dependency here, this works pretty well. Signed-off-by: Lukas Fischer <[email protected]>
Add a small and very simple test for the CycloneDX parser, after all it doesn't do much. More annoying is again the Makefile structure, since the project is not set up for multiple parsers for a scanner, it also does not accommodate that for the tests. To circumvent this, the added target for the parser-cyclonedx tests needs to manually run the install-deps-js target with a different module name, so that all the dependencies get installed correctly. Signed-off-by: Lukas Fischer <[email protected]>
Keeping SBOM creation as part of the original trivy scanner created a whole bunch of problems, all related to the fact that we decided that SBOMs should get their own parser instead of reusing the existing trivy parser. Since nothing in the project structure assumes that scanners can have more than one parser, making the Makefile targets work with it was pretty inconvenient. The release process, orchestrated by a GitHub Workflow, also cannot work with multiple parsers. This extracts all SBOM related functionality from the trivy scanner to a new trivy-sbom scanner, which then works again with the usual project structure. The parser is still a generic CycloneDX parser, although adding Syft and then reusing the parser needs resturcturing again (or copy-pasting of code). Signed-off-by: Lukas Fischer <[email protected]>
MAke sure the parser properly reads an actual trivy-generated CycloneDX SBOM file. Signed-off-by: Lukas Fischer <[email protected]>
Signed-off-by: Lukas Fischer <[email protected]>
Releases and CI only partially use makefiles. To make sure the trivy-sbom scanner and the persistence-dependencytrack hook are included in all builds, they are added to the corresponding matrix lists here. Signed-off-by: Lukas Fischer <[email protected]>
added 4 commits
September 18, 2023 14:09
During CI the persistence-dependencytrack hook fails because the helm chart (or some file in there) is larger than allowed. To fix this, copy over ignored files and patterns from elastic's .helmignore. Signed-off-by: Lukas Fischer <[email protected]>
Make sure that people can more easily combine the Trivy SBOM scanner with the Dependency-Track hook by linking eachother from their documentation. Use absolute links to make sure that external links from GitHub, DockerHub and ArtifactHub also work. See secureCodeBox#1963 Signed-off-by: Lukas Fischer <[email protected]>
Instead of just crashing hard, check the response the Dependency-Track API returns for network errors and error status codes to print nicer error messages. Signed-off-by: Lukas Fischer <[email protected]>
The previous way of detecting the name and version to supply to Dependency-Track was very brittle, it already failed for image references including a hash, resulting in names like hello-world@sha256, because it would only split on ':' and then select the first and last component. This version uses a regex to as accurately as possible match the individual components of a docker image reference. The regex comes from the [official implementation] on GitHub, but is actually taken from a [pull request], which adds named capture groups and fixes an issue with domain recognition being too eager. Yes the regex looks pretty wild, yes there are tests. I don't think it makes sense to build the regex from the individual components like the docker library does it. Unfortunately this does not solve the problem of actually getting the reference from somewhere, for images it works with getting it from the name of the main component, but this part stays brittle. Annotations to the scans might be a possible solution for that. [official implementation]: https://github.com/distribution/reference [pull request]: distribution/distribution#3803 Signed-off-by: Lukas Fischer <[email protected]>
9e053bb to
67c3f4f
Compare
Make sure the telemetry service recognizes trivy-sbom-image as an official ScanType and sends correct data. Signed-off-by: Lukas Fischer <[email protected]>
J12934
approved these changes
Sep 29, 2023
Member
|
Look very good now 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is an MVP for an SBOM workflow as part of the secureCodeBox. It adds
There are some challenges regarding SBOM integrations.
Since trivy can now generate outputs in a different format (CycloneDX SBOMs instead of trivy-json), the parser needs to be able to handle this. It is cleaner and more future-proof to create a separate CycloneD SBOM parser. This needs to be integrated and built somehow, currently it is part of the trivy scanner directory, because only trivy creates SBOMs.Because of way too much trouble with one scanner with two parsers, trivy-sbom is now an individual scanner. The parser is a generic CycloneDX parser though. If Syft is also integrated in the future, we might need to change the architecture to have global parsers which can be used for multiple scanners.The parser integration requires some Makefile changes, which could be nicer, but work for the time being.SBOM-related items and configurations are named generically without trivy so that moving them somewhere else or independently using them in the future is easier. I'm unsure though if SBOM or CycloneDX is a better name, currently only CycloneDX is supported and used, but in the future we might want to generate SPDX SBOMs as well, and then we need to decide if that should be the same scanType and parser, or if that should remain completely separated.
Hooks are triggered for all scans, but the Depenendency-Track hook is only useful for scans which create CycloneDX SBOMs, all other formats or findings cannot be handled. To mitigate that, the hook first checks the scan results type for
sbom-cyclonedxand then checks that the raw results has thebomFormatproperty set toCycloneDX, a mandatory part of CycloneDX-SBOMs. All other scans are ignored.The Dependency-Track hook automatically creates projects for SBOMs it uploads. For this the API requires a project name and a version, which the hook tries to get from the
nameproperty of the main component in the CycloneDX SBOM. This works as long as only image SBOMs are used, which have a name of<repository>:<tag>, where the hook splits by:and gets the first component as name, the last as version or defaults tolatest. This probably stops working in the future when SBOMs for other targets are integrated as well, but CycloneDX SBOMs do not provide better properties to get this data, trivy SBOMs do not contain a version for the main component and syft only sets the sha256 hash.Currently unimplemented features that we might still want at some point:
tests for the parser-cyclonedxunit tests and integration tests existtests for the persistence-dependencytrack hookunit tests existsecurecodebox/parser-cyclonedx, which needs to be registered on Docker Hub first, otherwise the next release will fail.Closes #1838.
Checklist