NVD API returns transient 403 response with API key

**Describe the bug**
When using an API key, the NVD API has started returning a transient 403 response. It occurs in the middle of a database update, so not a key configuration issue (and when retrying the key is used successfully). Some ODC database updates do complete, but this has occurred about half of the time since upgrading to v9.0.2.

This is related to https://github.com/jeremylong/DependencyCheck/issues/6180 and https://github.com/jeremylong/DependencyCheck/issues/6149, but is still occurring with the 9.0.2 CLI.

```sh
[INFO] Running: [/bin/sh -c /usr/share/dependency-check/bin/dependency-check.sh     --updateonly --nvdApiKey "$(cat /kaniko/NVD_API_KEY)"     --retireJsForceUpdate --hostedSuppressionsForceUpdate] 
[INFO] Checking for updates
[INFO] NVD API has 231,966 records in this update
[INFO] Downloaded 10,000/231,966 (4%)
[INFO] Downloaded 20,000/231,966 (9%)
[INFO] Downloaded 30,000/231,966 (13%)
[INFO] Downloaded 40,000/231,966 (17%)
[INFO] Downloaded 50,000/231,966 (22%)
[INFO] Downloaded 60,000/231,966 (26%)
[INFO] Downloaded 70,000/231,966 (30%)
[INFO] Downloaded 80,000/231,966 (34%)
[INFO] Downloaded 90,000/231,966 (39%)
[INFO] Downloaded 100,000/231,966 (43%)
[INFO] Downloaded 110,000/231,966 (47%)
[INFO] Downloaded 120,000/231,966 (52%)
[INFO] Downloaded 130,000/231,966 (56%)
[INFO] Downloaded 140,000/231,966 (60%)
[INFO] Downloaded 150,000/231,966 (65%)
[INFO] Downloaded 160,000/231,966 (69%)
[INFO] Downloaded 170,000/231,966 (73%)
[INFO] Downloaded 180,000/231,966 (78%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:340)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
	at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
	at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
	at org.owasp.dependencycheck.App.run(App.java:172)
	at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
	at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346)
	at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:319)
	... 6 common frames omitted
[ERROR] Failed to process CVE-2011-0074
org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to retrieve id for new vulnerability for 'CVE-2011-0074'
	at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1054)
	at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability(CveDB.java:866)
	at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:87)
	at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:33)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]"; SQL statement:
DELETE FROM reference WHERE cveid = ? [50000-214]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:554)
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
	at org.h2.message.DbException.get(DbException.java:212)
	at org.h2.message.DbException.convert(DbException.java:395)
	at org.h2.command.Command.executeUpdate(Command.java:264)
	at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:209)
	at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:169)
	at org.owasp.dependencycheck.data.nvdcve.H2Functions.updateVulnerability(H2Functions.java:223)
	at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.h2.schema.FunctionAlias$JavaMethod.execute(FunctionAlias.java:495)
	at org.h2.schema.FunctionAlias$JavaMethod.getTableValue(FunctionAlias.java:363)
	at org.h2.expression.function.table.JavaTableFunction.getValue(JavaTableFunction.java:34)
	at org.h2.table.FunctionTable.getResult(FunctionTable.java:51)
	at org.h2.index.VirtualConstructedTableIndex.find(VirtualConstructedTableIndex.java:38)
	at org.h2.index.IndexCursor.find(IndexCursor.java:161)
	at org.h2.table.TableFilter.next(TableFilter.java:394)
	at org.h2.command.query.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1832)
	at org.h2.result.LazyResult.hasNext(LazyResult.java:78)
	at org.h2.result.FetchedResult.next(FetchedResult.java:34)
	at org.h2.command.query.Select.queryFlat(Select.java:728)
	at org.h2.command.query.Select.queryWithoutCache(Select.java:833)
	at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
	at org.h2.command.query.Query.query(Query.java:512)
	at org.h2.command.query.Query.query(Query.java:475)
	at org.h2.command.CommandContainer.query(CommandContainer.java:251)
	at org.h2.command.Command.executeQuery(Command.java:190)
	at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:128)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
	at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1049)
	... 7 common frames omitted
Caused by: org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]
	at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:1004)
	at org.h2.mvstore.DataUtils.readFully(DataUtils.java:470)
	at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
	at org.h2.mvstore.Chunk.readBufferForPage(Chunk.java:422)
	at org.h2.mvstore.MVStore.readPage(MVStore.java:2569)
	at org.h2.mvstore.MVMap.readPage(MVMap.java:633)
	at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1125)
	at org.h2.mvstore.Page.get(Page.java:243)
	at org.h2.mvstore.MVMap.get(MVMap.java:436)
	at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:472)
	at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:467)
	at org.h2.mvstore.db.MVPrimaryIndex.getRow(MVPrimaryIndex.java:263)
	at org.h2.mvstore.db.MVTable.getRow(MVTable.java:331)
	at org.h2.mvstore.db.MVSecondaryIndex$MVStoreCursor.get(MVSecondaryIndex.java:421)
	at org.h2.index.IndexCursor.get(IndexCursor.java:270)
	at org.h2.table.TableFilter.get(TableFilter.java:515)
	at org.h2.command.dml.Delete.update(Delete.java:59)
	at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:74)
	at org.h2.command.CommandContainer.update(CommandContainer.java:169)
	at org.h2.command.Command.executeUpdate(Command.java:252)
	... 34 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
	at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159)
	at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:814)
	at org.h2.mvstore.DataUtils.readFully(DataUtils.java:456)
	... 52 common frames omitted
```

**Version of dependency-check used**
The problem occurs using version 9.0.2 of the cli (from the `owasp/dependency-check` image)

**To Reproduce**

Run `/usr/share/dependency-check/bin/dependency-check.sh --updateonly`, but since transient it's hard to reproduce reliably.

**Expected behavior**

Database update should occur without error.



Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NVD API returns transient 403 response with API key #6195

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

NVD API returns transient 403 response with API key #6195

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions