When scanning vulnerabilities given an SBOM (SPDX) in an offline mode using osv-scanner the memory usage increases drastically.
It reaches more than 15GB RAM (and then the server kills it).
In order for the bug to be drastic, it requires relatively big SBOMs, I used 10MB SBOM (see attached)
Neither CPE nor PURL found for package: &{IsUnpackaged:false PackageName:main PackageSPDXIdentifier:Package-main-95578677-9f52-4db1-b76e-88241bbcf5b2 PackageVersion:0 PackageFileName: PackageSupplier:0xc00011e300 PackageOriginator:<nil> PackageDownloadLocation:NOASSERTION FilesAnalyzed:false IsFilesAnalyzedTagPresent:true PackageVerificationCode:<nil> PackageChecksums:[] PackageHomePage: PackageSourceInfo: PackageLicenseConcluded: PackageLicenseInfoFromFiles:[] PackageLicenseDeclared: PackageLicenseComments: PackageCopyrightText: PackageSummary: PackageDescription: PackageComment: PackageExternalReferences:[] PackageAttributionTexts:[] PrimaryPackagePurpose: ReleaseDate: BuiltDate: ValidUntilDate: Files:[] Annotations:[] hasFiles:[]}
Scanned ./output/gitlab/sbom.spdx.json file and found 5521 packages
Loaded npm local db from osv-data/osv-scanner/npm/all.zip
Loaded RubyGems local db from osv-data/osv-scanner/RubyGems/all.zip
Loaded Go local db from osv-data/osv-scanner/Go/all.zip
Loaded Ubuntu local db from osv-data/osv-scanner/Ubuntu/all.zip
Loaded crates.io local db from osv-data/osv-scanner/crates.io/all.zip
<the memory limit kills it, after using more than 10GB>
goroutine 1 gp=0xc000002380 m=7 mp=0xc0004bc008 [running]:
runtime.throw({0x29b6b97?, 0x100?})
<USERS_PATH>/golang.org/[email protected]/src/runtime/panic.go:1101 +0x48 fp=0xc0000562f8 sp=0xc0000562c8 pc=0xe899c8
runtime.(*mcache).allocLarge(0x2000?, 0xe000, 0x1)
<USERS_PATH>/golang.org/[email protected]/src/runtime/mcache.go:237 +0x17c fp=0xc000056348 sp=0xc0000562f8 pc=0xe2e59c
runtime.mallocgcLarge(0x10e45f9?, 0x0, 0x0)
<USERS_PATH>/golang.org/[email protected]/src/runtime/malloc.go:1540 +0x79 fp=0xc0000563a0 sp=0xc000056348 pc=0xe2b379
runtime.mallocgc(0xe000, 0x0, 0x0)
<USERS_PATH>/golang.org/[email protected]/src/runtime/malloc.go:1063 +0xc5 fp=0xc0000563d0 sp=0xc0000563a0 pc=0xe875e5
runtime.growslice(0xc18ffec000, 0xc18fff4000?, 0x0?, 0x2000?, 0x7fffffffffffb40c?)
<USERS_PATH>/golang.org/[email protected]/src/runtime/slice.go:264 +0x5c9 fp=0xc000056440 sp=0xc0000563d0 pc=0xe8c489
io.ReadAll({0x7fcc0fe07d30, 0xc185edf360})
<USERS_PATH>/golang.org/[email protected]/src/io/io.go:723 +0xc9 fp=0xc0000564b8 sp=0xc000056440 pc=0xecf009
github.com/google/osv-scanner/v2/internal/clients/clientimpl/localmatcher.(*ZipDB).loadZipFile(0xc0007a0fc0, 0xc0015ab970)
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/internal/clients/clientimpl/localmatcher/zip.go:157 +0x13e fp=0xc0000565a8 sp=0xc0000564b8 pc=0x1fc721e
github.com/google/osv-scanner/v2/internal/clients/clientimpl/localmatcher.(*ZipDB).load(0xc0007a0fc0, {0x2f551e8?, 0x4160ee0?})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/internal/clients/clientimpl/localmatcher/zip.go:201 +0x194 fp=0xc000056628 sp=0xc0000565a8 pc=0x1fc7754
github.com/google/osv-scanner/v2/internal/clients/clientimpl/localmatcher.NewZippedDB({0x2f551e8, 0x4160ee0}, {0xc000048648?, 0x2528a60?}, {0x2993007, 0x6}, {0xc00fabd090, 0x41}, {0x29eedaa, 0x16}, ...)
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/internal/clients/clientimpl/localmatcher/zip.go:215 +0x175 fp=0xc0000566b8 sp=0xc000056628 pc=0x1fc7915
github.com/google/osv-scanner/v2/internal/clients/clientimpl/localmatcher.(*LocalMatcher).loadDBFromCache(0xc000502140, {0x2f551e8, 0x4160ee0}, {{0x2993007?, 0xc0007a0de0?}, {0x0?, 0x60?}})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/internal/clients/clientimpl/localmatcher/localmatcher.go:104 +0x165 fp=0xc000056768 sp=0xc0000566b8 pc=0x1fc5dc5
github.com/google/osv-scanner/v2/internal/clients/clientimpl/localmatcher.(*LocalMatcher).MatchVulnerabilities(0xc000502140, {0x2f551e8, 0x4160ee0}, {0xc000c00000, 0x1591, 0x0?})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/internal/clients/clientimpl/localmatcher/localmatcher.go:72 +0x26e fp=0xc000056970 sp=0xc000056768 pc=0x1fc580e
github.com/google/osv-scanner/v2/pkg/osvscanner.makeVulnRequestWithMatcher({0xc00191e000, 0x1591, 0x0?}, {0x2f3eb40, 0xc000502140})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/pkg/osvscanner/osvscanner.go:529 +0x172 fp=0xc000056a50 sp=0xc000056970 pc=0x23b6072
github.com/google/osv-scanner/v2/pkg/osvscanner.DoScan({{{0x1, 0x0, {0x0, 0x0}}, {0xc00033fd70, 0x3, 0x3}, {0x4160ee0, 0x0, 0x0}}, ...})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/pkg/osvscanner/osvscanner.go:246 +0x56a fp=0xc000056e00 sp=0xc000056a50 pc=0x23b3d0a
github.com/google/osv-scanner/v2/cmd/osv-scanner/scan/source.action({0x4e7671db54dd8ac9?, 0x0?}, 0xc000404008, {0x2f3e960, 0xc00011e070}, {0x2f3e960, 0xc00011e078})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/cmd/osv-scanner/scan/source/command.go:128 +0x5c5 fp=0xc000057258 sp=0xc000056e00 pc=0x23be465
github.com/google/osv-scanner/v2/cmd/osv-scanner/scan/source.Command.func3({0x2f55258?, 0xc00033fd40?}, 0xc00033fd40?)
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/cmd/osv-scanner/scan/source/command.go:77 +0x2d fp=0xc0000572a0 sp=0xc000057258 pc=0x23bde6d
github.com/urfave/cli/v3.(*Command).run(0xc000404008, {0x2f55258, 0xc00033fd40}, {0xc000502100, 0x4, 0x4})
<USERS_PATH>/github.com/urfave/cli/[email protected]/command_run.go:340 +0x26c2 fp=0xc000057660 sp=0xc0000572a0 pc=0x20682a2
github.com/urfave/cli/v3.(*Command).run(0xc000404508, {0x2f55258, 0xc00033fb90}, {0xc000618fa0, 0x5, 0x5})
<USERS_PATH>/github.com/urfave/cli/[email protected]/command_run.go:278 +0x1f4e fp=0xc000057a20 sp=0xc000057660 pc=0x2067b2e
github.com/urfave/cli/v3.(*Command).run(0xc000404c88, {0x2f55258, 0xc00033f5f0}, {0xc0004a9020, 0x6, 0x6})
<USERS_PATH>/github.com/urfave/cli/[email protected]/command_run.go:278 +0x1f4e fp=0xc000057de0 sp=0xc000057a20 pc=0x2067b2e
github.com/urfave/cli/v3.(*Command).Run(...)
<USERS_PATH>/github.com/urfave/cli/[email protected]/command_run.go:94
github.com/google/osv-scanner/v2/cmd/osv-scanner/internal/cmd.Run({0xc000152000, 0x5, 0x5}, {0x2f3e960, 0xc00011e070}, {0x2f3e960, 0xc00011e078}, {0xc000677f28, 0x3, 0x3})
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/cmd/osv-scanner/internal/cmd/run.go:94 +0x57c fp=0xc000057ed8 sp=0xc000057de0 pc=0x23bfcbc
main.main()
<USERS_PATH>/github.com/google/osv-scanner/[email protected]/cmd/osv-scanner/main.go:14 +0x85 fp=0xc000057f50 sp=0xc000057ed8 pc=0x23c21c5
runtime.main()
<USERS_PATH>/golang.org/[email protected]/src/runtime/proc.go:283 +0x28b fp=0xc000057fe0 sp=0xc000057f50 pc=0xe5582b
runtime.goexit({})
<USERS_PATH>/golang.org/[email protected]/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000057fe8 sp=0xc000057fe0 pc=0xe91681
There are additional go routines, but their eitre stack is only in /golang.org/[email protected]/
AlmaLinux, Android, CRAN, Debian, GSD, Go, Hex, Mageia, NuGet, Packagist, PyPI, 'Rocky, Linux', SUSE, UVI, Wolfi, crates.io, openSUSE, Alpine, Bitnami, Chainguard, GIT, 'GitHub, Actions', Hackage, Linux, Maven, OSS-Fuzz, Pub, 'Red, Hat', RubyGems, SwiftURL, Ubuntu, '[EMPTY]', npm
When scanning vulnerabilities given an SBOM (SPDX) in an offline mode using osv-scanner the memory usage increases drastically.
It reaches more than 15GB RAM (and then the server kills it).
In order for the bug to be drastic, it requires relatively big SBOMs, I used 10MB SBOM (see attached)
Command:
$ OSV_SCANNER_LOCAL_DB_CACHE_DIRECTORY="./osv-data" osv-scanner scan -L ./output/gitlab/sbom.spdx.json --offlineTested versions:
This behavior does not occur in online mode or in older versions, such as v1.9.2.
stdout:
Stack trace:
Downloaded DBs
SBOM:
sbom.spdx.json