Fix race conditions and non-deterministic finding counts and re-enable parallel file parsing#209
Merged
Merged
Conversation
…e parallel file parsing
|
🎯 Code Coverage (details) 🔗 Commit SHA: 3494c4d | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 605a1870a9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
MikaYuoadas
previously approved these changes
Jun 23, 2026
…cans deterministic)
robertohuertasm-datadog
approved these changes
Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
--x-parallelparsingwas added to speed up scans (~30%) but was disabled because the scanner returned a different number of findings on every run of the same repo. A first pass (K9CODESEC-1981) fixed two data races but instability remained, so the flag stayed off.This PR finishes that investigation. The user-visible drift turned out not to be the parallel-parsing feature at all, but a pre-existing bug in the query-evaluation phase that affects any multi-worker scan of a large repo:
DecodeQueryResultsgated result decoding with a 60s per-query wall-clock deadline, and a high-volume rule's decode loop (one HCL re-parse per result) gets starved under worker contention, blows the budget at a timing-dependent point, and — via an ineffectivebreak(go vet SA4011) — silently drops the remaining findings with no error. Result: non-deterministic,-race-clean, only at scale.With that fixed (plus the two remaining races), parallel parsing is safe to re-enable as an opt-in flag.
JIRA Ticket
Changes
pkg/engine/inspector.go— (root cause) remove the per-query wall-clock deadline on result decoding; decode all items, honoring only real scan cancellation. Fix the ineffectivebreak(was breaking theselect, not the loop —SA4011).pkg/engine/vulnerability_builder.go— remove the deaddetector.SetupLogs(...)call (shared-field write from workers; field is unused).pkg/detector/docker/docker_detect.go— deep-copy lines beforeprepareDockerFileLinesmutates them, so concurrent detect-line calls don't corrupt a shared*FileMetadatabacking array.cmd/scanner/scan.go— restore the hidden--x-parallelparsingCLI flag (defaultfalse) and wire it into the feature-flag override.pkg/featureflags/local_evaluator.go— leaveIaCEnableKicsParallelFileParsingdefaultfalse; parallel parsing is opt-in.Author Checklist
QA Instruction
--x-parallelparsing— finding count / SARIF results should be identical to the serial run, and the scan should be ~30% faster.go test -race ./pkg/detector/docker ./pkg/parser/terraform— no data races.go vet ./pkg/engine/...— theSA4011"ineffective break" warning is gone.Blast Radius
DataDog IaC Scanner only (engine / detectors / CLI). Important: the decode-timeout removal affects all multi-worker scans, not just
--x-parallelparsing. On large repos that previously hit the truncation, finding counts may increase (we stop silently dropping real findings) and become deterministic — so SARIF / baseline counts for big repos can change. The--x-parallelparsingre-enable itself is opt-in (default off), so no behavior change unless the flag is passed.Additional Notes
The two race fixes are validated by
-race; the decode fix is validated on full big repo scan: 3+ runs now produce a bit-identical result (count + similarity-ID hash) matching the serial baseline, and parallel parsing runs ~32% faster with identical findings.--x-parallelparsing)~32% faster (1.48×), findings identical across all runs (same similarity-ID hash).
I submit this contribution under the Apache-2.0 license.