Skip to content

Collaborate with dependency-track to use it as NVD proxy endpoint #6277

Closed
#6287
Closed
Collaborate with dependency-track to use it as NVD proxy endpoint#6277
#6287
Labels
enhancement
@schwichtgit

Description

@schwichtgit
schwichtgit
opened on Dec 11, 2023

Is your feature request related to a problem? Please describe.

We (like probably many others) used dependency-track (prior dependency-check v9+) as proxy for NVD (legacy/non API V2) and other feeds.
This greatly lowers the load on public feed endpoints and makes CI pipelines using dependency-check more robust.

Dependency-track V4.10.0+ provides NIST NVD API v2 support (with API key etc).

Describe the solution you'd like
We would like to use its NVD endpoint for dependency-check (like we used to).

dependency-check.sh Core --updateonly --nvdDatafeed=https://{{ endpoint }}/mirror/nvd

Describe alternatives you've considered
I understand that one can create a proxy with the vulnz cli and I have (as a test) been able to make the cached content from vulnz work via the http endpoint.

However, the dependency-check project collaborating with the dependency-track project would help the community significantly.

Here is what I did to research this a bit more ...

The cached (by dependency-track V4.10.0) NVD files look like

nvdcve-1.1-{{ year }}.json
nvdcve-1.1-{{ year }}.json.gz
nvdcve-1.1-{{ year }}.json.gz.ts
nvdcve-1.1-{{ year }}.meta
(...)
nvdcve-1.1-modified.json
nvdcve-1.1-modified.json.gz
nvdcve-1.1-modified.json.gz.ts
nvdcve-1.1-modified.meta

I noted that the cache.properties is missing (that I understand you potentially would like to remove as a requirement).

I tested if I can create the cache.property file out of the meta data available.

$ export p=../nist-orig; export c="${p}/cache.properties"; echo "prefix=nvdcve-1.1-" > "${c}"; for i in $(seq 2002 2023); do echo "lastModifiedDate.${i}=$(grep lastModifiedDate ${p}/nvdcve-1.1-${i}.meta|sed 's#lastModifiedDate:##g'|sed 's#:#\\:#g')" >> "${c}"; done; echo "lastModifiedDate.modified=$(grep lastModifiedDate ${p}/nvdcve-1.1-modified.meta|sed 's#lastModifiedDate:##g'|sed 's#:#\\:#g')" >> "${c}"

creating this file.

prefix=nvdcve-1.1-
lastModifiedDate.2002=2023-11-07T03\:04\:03-05\:00
lastModifiedDate.2003=2023-11-07T03\:03\:59-05\:00
lastModifiedDate.2004=2023-11-07T03\:03\:57-05\:00
lastModifiedDate.2005=2023-11-08T03\:02\:12-05\:00
lastModifiedDate.2006=2023-11-29T03\:01\:23-05\:00
lastModifiedDate.2007=2023-12-08T03\:03\:39-05\:00
lastModifiedDate.2008=2023-12-08T03\:03\:33-05\:00
lastModifiedDate.2009=2023-12-08T03\:03\:26-05\:00
lastModifiedDate.2010=2023-12-08T03\:03\:20-05\:00
lastModifiedDate.2011=2023-12-08T03\:03\:13-05\:00
lastModifiedDate.2012=2023-12-08T03\:03\:06-05\:00
lastModifiedDate.2013=2023-12-09T03\:02\:56-05\:00
lastModifiedDate.2014=2023-12-09T03\:02\:47-05\:00
lastModifiedDate.2015=2023-12-09T03\:02\:40-05\:00
lastModifiedDate.2016=2023-12-09T03\:02\:31-05\:00
lastModifiedDate.2017=2023-12-09T03\:02\:19-05\:00
lastModifiedDate.2018=2023-12-09T03\:01\:55-05\:00
lastModifiedDate.2019=2023-12-09T03\:01\:36-05\:00
lastModifiedDate.2020=2023-12-11T03\:01\:28-05\:00
lastModifiedDate.2021=2023-12-11T03\:01\:03-05\:00
lastModifiedDate.2022=2023-12-11T03\:00\:38-05\:00
lastModifiedDate.2023=2023-12-11T03\:00\:14-05\:00
lastModifiedDate.modified=2023-12-11T06\:00\:01-05\:00

but that failed with

$ dependency-check.sh Core --updateonly --nvdDatafeed=https://{{ endpoint }}/mirror/nvd

[INFO] Checking for updates
[INFO] NVD API Cache requires several updates; this could take a couple of minutes.
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2012.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2012.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2023.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2023.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2011.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2011.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2022.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2022.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2010.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2010.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2021.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2021.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2020.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2020.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2009.json.gz
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2009.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] Download Started for NVD Cache - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2008.json.gz
[ERROR] The execution of the download was interrupted
org.owasp.dependencycheck.data.update.exception.UpdateException: The execution of the download was interrupted
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDownload(NvdApiDataSource.java:261)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDatafeed(NvdApiDataSource.java:156)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:108)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
	at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
	at org.owasp.dependencycheck.App.run(App.java:172)
	at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: java.util.concurrent.ExecutionException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDownload(NvdApiDataSource.java:251)
	... 7 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2224)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1719)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1697)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4899)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3846)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3814)
	at org.owasp.dependencycheck.data.update.nvd.api.DownloadTask.call(DownloadTask.java:95)
	at org.owasp.dependencycheck.data.update.nvd.api.DownloadTask.call(DownloadTask.java:42)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
[INFO] Skipping RetireJS update since last update was within 24 hours.
[INFO] Skipping Hosted Suppressions file update since last update was within 2 hours.
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[ERROR] Error downloading NVD CVE - https://{{ endpoint }}/mirror/nvd/nvdcve-1.1-2008.json.gz Reason: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
[INFO] End database defrag (257 ms)
[ERROR] The execution of the download was interrupted
org.owasp.dependencycheck.data.update.exception.UpdateException: The execution of the download was interrupted
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDownload(NvdApiDataSource.java:261)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDatafeed(NvdApiDataSource.java:156)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:108)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
	at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
	at org.owasp.dependencycheck.App.run(App.java:172)
	at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: java.util.concurrent.ExecutionException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processDownload(NvdApiDataSource.java:251)
	... 7 common frames omitted
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "CVE_data_type" (class io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20), not marked as ignorable (7 known properties: "startIndex", "totalResults", "resultsPerPage", "format", "version", "vulnerabilities", "timestamp"])
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 2, column: 22] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["CVE_data_type"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2224)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1719)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1697)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4899)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3846)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3814)
	at org.owasp.dependencycheck.data.update.nvd.api.DownloadTask.call(DownloadTask.java:95)
	at org.owasp.dependencycheck.data.update.nvd.api.DownloadTask.call(DownloadTask.java:42)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions