feat: [SCI] add BazelBuildExtractor for inter-module dependency analysis in Bazel monorepos#177
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf4ddf8659
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
anderruiz
changed the title
…BUILD.bazel Introduces Bazel as a package manager mapped to the Java language, adds BazelBuildFilePath constant, FileTypeBUILDBazel build file type, and registers SimpleProcessor for BUILD.bazel to enable BFS transitive closure computation. Includes tests for the SimpleProcessor registration and transitive closure. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…om BUILD.bazel files Parses BUILD.bazel files in Java/Bazel monorepos to extract internal module-to-module dependencies (//path/to/module refs). Filters out external Maven artifact() references and load() macro imports. Integrates with the existing BuildFileRelations framework via ProjectDeps and SimpleProcessor BFS for transitive closure computation. Validated against logs-backend sca-processor: 99.6% match rate vs bazel query ground truth. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…upport Adds BUILD.bazel to the expected extractor list, updates the Java/Gradle language expansion test to include BUILD.bazel, and updates integration test snapshots. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
bf4ddf8 to
e8a0e95
Compare
|
🎯 Code Coverage (details) 🔗 Commit SHA: e0aa860 | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8a0e95bd7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if strings.HasPrefix(strings.TrimSpace(line), "load(") { | ||
| continue |
There was a problem hiding this comment.
When a load() call is formatted over multiple lines, only the line containing load( is removed, leaving the following string label (for example "//rules/java:defs.bzl") to be treated as a ProjectDeps edge if that directory has a BUILD file. This reintroduces the build-infrastructure dependencies the comment is trying to exclude, so Bazel graphs with multiline loads can gain false module dependencies in the SBOM and GetBuildFileTrees.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. Replaced the line-by-line filter with a multiline regex (?s)load\([^)]*\) that removes entire load blocks, including those spanning multiple lines.
| internalRefRegex := cachedregexp.MustCompile(`"(//[^"]+)"`) | ||
| matches := internalRefRegex.FindAllSubmatch(filtered, -1) |
There was a problem hiding this comment.
Because this regex scans every string literal in the BUILD file, labels in non-dependency attributes such as visibility = ["//teams/foo:__pkg__"] or platform constraints are emitted as ProjectDeps whenever the referenced package has a BUILD file. Those edges are then written to the SBOM dependency graph and traversed by GetBuildFileTrees, so metadata/access-control labels can appear as Java module dependencies; please limit extraction to dependency-like attributes or otherwise filter non-dependency labels.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Won't fix. Labels in non-dependency attributes like visibility typically reference special Bazel targets (//visibility:public, //teams/foo:__pkg__) that don't have BUILD files on disk. The filesystem existence check already filters them out. Adding attribute-specific parsing would require a real Starlark parser.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b6dba78d6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
76129b0 to
6c28a6b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c28a6b5b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // reference Starlark macro/rule definition files (build infrastructure), | ||
| // not Java modules that end up in the service binary. We remove the entire | ||
| // block (including multiline loads) to avoid capturing their string labels. | ||
| loadBlockRegex := cachedregexp.MustCompile(`(?s)load\([^)]*\)`) |
There was a problem hiding this comment.
With the new multiline load stripping, any macro/function name that contains load( is treated as a load statement. For example, dd_workload(name = "svc", deps = ["//libs/common"]) leaves only the dd_work prefix before label extraction, so the valid dependency is never emitted as a ProjectDep; anchor this regexp to an actual Starlark load call instead of matching substrings inside macro names.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed. Anchored the load-stripping regex to start-of-line with (?ms)^\s*load\([^)]*\) so it only removes actual Starlark load statements and not function names that contain "load" as a substring.
| } | ||
|
|
||
| if ctx.RootDir != "" { | ||
| artifact.ProjectDeps = extractBazelInternalDeps(content, ctx.RootDir) |
There was a problem hiding this comment.
When GenerateSBOM or the CLI is pointed at a service directory inside a Bazel workspace, ctx.RootDir is that scan directory, but //... Bazel labels are workspace-root relative. Passing the scan dir here makes a label like //libs/common resolve under <service>/libs/common/BUILD*, so os.Stat filters it out and the SBOM loses inter-module edges unless the user scans from the workspace root; this should use the Bazel workspace/repo root instead.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Won't fix in this PR. The extractor follows the same contract as all other extractors (Gradle, Maven): ctx.RootDir is expected to be the repository root, which is how the CLI and GenerateSBOM are documented to be invoked on a Bazel monorepo. Workspace root traversal would add complexity for a use case outside the documented API contract.
- accept single-quoted Bazel labels in addition to double-quoted
- register SimpleProcessor for FileType("BUILD") alongside BUILD.bazel
- replace line-by-line load() filter with multiline regex to handle multiline load blocks
6c28a6b to
7f74377
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f74377c39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| if ctx.RootDir != "" { | ||
| artifact.ProjectDeps = extractBazelInternalDeps(content, ctx.RootDir) |
There was a problem hiding this comment.
When a BUILD file uses an absolute label for another target in the same package, such as libs/common/BUILD.bazel containing deps = ["//libs/common:helper"], the helper resolves that label back to the current BUILD file because it drops the target suffix. After path relativization this writes a CycloneDX dependency from libs/common/BUILD.bazel to itself, so consumers of the SBOM dependency graph can see a bogus self-edge; filter out deps whose filename matches the source BUILD file before assigning ProjectDeps.
Useful? React with 👍 / 👎.
|
|
||
| // Try BUILD.bazel first, then BUILD | ||
| for _, buildFileName := range []string{bazelBuildFilename, bazelBuildFilenamAlt} { | ||
| depPath := filepath.Join(rootDir, path, buildFileName) |
There was a problem hiding this comment.
If a malformed BUILD string like "//../sibling" is present, this filepath.Join normalizes the .. and can os.Stat a BUILD file outside the scanned repository; when that file exists, the extractor emits an out-of-scope ProjectDep that is later included in the SBOM dependency graph. Since this parser scans raw strings rather than Bazel-valid labels, guard the cleaned path with filepath.Rel/prefix checks and skip anything that escapes rootDir.
Useful? React with 👍 / 👎.
Replace the fixed 10-hop BFS limit with a size-adaptive limit: - <100 build files → 9999 hops (effectively unlimited, small repos) - 100–399 files → 10 hops - 400+ files → 5 hops (large monorepos like dd-source ~41k BUILD files)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0aa86066b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for len(queue) > 0 { | ||
| node := queue[0] | ||
| queue = queue[1:] | ||
| if node.depth >= maxHops { |
There was a problem hiding this comment.
When GetBuildFileTrees processes 100+ files, maxHopsForSize returns 10 (or 5 at 400+), so this guard stops BFS before deeper transitive edges are added. The public BuildFileRelations contract says Dependencies contains all transitively reachable build files, and all file types registered with SimpleProcessor use this path; a chain of 101 POMs/Gradle/requirements/BUILD files will silently omit hop 11+ (or hop 6+ in very large repos), breaking transitive attribution in the large monorepos this feature targets.
Useful? React with 👍 / 👎.
2 participants
Summary
Adds a new
BazelBuildExtractorthat parsesBUILD.bazelfiles in Java/Bazel monorepos to extract internal module-to-module dependencies (//path/to/modulerefs). This enables the SBOM generator to produceProjectDepsedges between modules within monorepos likelogs-backend, which is critical for service ownership tagging and transitive dependency attribution.What it does
BUILD.bazelfiles line-by-line, extracting//path/to/modulereferences fromdeps,runtime_deps, andexportsattributesload()macro imports, external Mavenartifact()references, and inline commentsBuildFileRelations: emitsProjectDepsedges consumed by the existingSimpleProcessorBFS to compute transitive closure with hop countsBazelas a package manager mapped to the Java language, withBUILD.bazelas its lockfile pathApproach
The extractor uses simple line-based parsing (no full Starlark AST) which is sufficient for dependency extraction. It:
"//path/to/module"patterns:targetsuffixes) to get module pathspath/to/module/BUILD.bazelas aProjectDepsentryTEST_DEPS inclusion
TEST_DEPSlists are intentionally included. These are compile-time test dependencies that affect the build graph and should be tracked for completeness. This is a conservative choice -- downstream consumers can filter if needed.Validation
Tested against
logs-backendsca-processormodule:bazel queryground truth (267/268 dependencies)domains/cloud-security-platform/libs/sca/sca-processor-config) is actually a false positive inbazel query(pulled in via macro expansion, not a direct file-level reference)Test plan
🤖 Generated with Claude Code