Evaluate each query against only its platform's documents#224
Conversation
|
🎯 Code Coverage (details) 🔗 Commit SHA: f3872a6 | Docs | Datadog PR Page | Give us feedback! |
d654918 to
d45bd2a
Compare
d45bd2a to
ac24841
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac248411a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case platformServerlessFW: | ||
| return []string{platformServerlessFW, platformCloudFormation} | ||
| } | ||
| return []string{key} |
There was a problem hiding this comment.
Keep Crossplane manifests in Kubernetes payloads
When scanning the normal supported platform set, Crossplane YAML is admitted by the analyzer as Kubernetes because Crossplane is not in pkg/platforms/platforms.go and the Kubernetes regex still matches apiVersion/kind. The sink then reclassifies the same file with all regexes as crossplane; this default branch buckets it only under crossplane, but no Crossplane queries are loaded, so Kubernetes rules receive an empty payload and findings for those manifests disappear. Please also bucket Crossplane documents with Kubernetes, or preserve the analyzer-selected platform for this case.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Crossplane has its own dedicated queries (assets/queries/crossplane/**) with "platform": "Crossplane", so crossplane-bucketed documents do get evaluated, by crossplane rules. Additionally, crossplane is not currently enabled so it's a no-op
ac24841 to
0533db3
Compare
0533db3 to
be6bef0
Compare
be6bef0 to
6fb32a2
Compare
…inate cross-platform payload overhead.
6fb32a2 to
f3872a6
Compare
Motivation
On large polyglot repositories the scanner builds a single OPA payload containing every parsed document across all platforms, then evaluates each rule query against the full cross-platform payload. This means, for example, that every Ansible rule rebuilds its task index over thousands of Kubernetes, Helm-rendered, CloudFormation, and Terraform documents that are irrelevant to it. On a real-world Go monorepo (49 k YAML files, ~48 k parsed documents), a single Ansible rule takes ~27 s to evaluate due to this cross-platform document scanning; with ~221 Ansible rules the total Ansible eval time exceeds 100 minutes. The same mechanism inflates eval time for CloudFormation and Kubernetes rules on large mixed repos.
Changes
Platform classification
A new
ClassifyFilefunction in the analyzer package determines a file's platform from its extension and content, mirroring the per-file detection already done by the analyzer but making it available to the parsing layer. Kind is used as the authoritative signal for unambiguous types (Helm→kubernetes, Terraform→terraform, Bicep→azureresourcemanager, Dockerfile→dockerfile, GRPC→grpc, INI/CFG→ansible); ambiguous YAML/JSON files fall back to content-based regex classification.Scan pipeline
Both the regular sink and the resolver sink now populate a
Platformfield on eachFileMetadataat parse time. Synthetic Terraform module documents inherit the platform of their parent file viaFileMetadatacloning.Rule engine
Inspectpartitions the combined document set by platform and builds one OPA payload per platform that is actually queried in that scan. Each rule query receives only its own platform's documents. Documents whose platform could not be determined are included in every platform's payload so no rule loses coverage.common-platform queries (force-enabled by the engine and not observed in production slow-rule telemetry) continue to receive the full cross-platform payload. Two small helpers,normalizePlatformKeyandselectPlatformPayload, handle the query-metadata-to-platform-key mapping (e.g.k8s↔kubernetes,Bicep↔azureresourcemanager).Author Checklist
QA Instruction
go test ./pkg/engine/... ./pkg/analyzer/... ./pkg/runner/... ./pkg/model/... ./pkg/scan/...— all packages should pass.test/fixtures/test_helmalongside a Terraform or Ansible fixture) and confirm the SARIF output is identical to the output produced bymain.Blast Radius
This change affects all scans that parse more than one platform type. Scans of a single platform are unaffected (the per-platform payload equals the full payload). The findings set is unchanged, validated by a findings diff on a 1 048-file mixed repository (Kubernetes, Helm-rendered, Ansible, Terraform, CloudFormation) showing identical results (1 016 findings, zero lost, zero gained) against the
mainbaseline.Additional Notes
Measured on a real-world monorepo (49 k YAML files):
I submit this contribution under the Apache-2.0 license.