Skip to content

Fix race conditions and non-deterministic finding counts and re-enable parallel file parsing#209

Merged
MarshalX merged 6 commits into
mainfrom
ilya.siamionau/K9CODESEC-2411/race-hunt
Jun 23, 2026
Merged

Fix race conditions and non-deterministic finding counts and re-enable parallel file parsing#209
MarshalX merged 6 commits into
mainfrom
ilya.siamionau/K9CODESEC-2411/race-hunt

Conversation

@MarshalX

@MarshalX MarshalX commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Motivation

--x-parallelparsing was added to speed up scans (~30%) but was disabled because the scanner returned a different number of findings on every run of the same repo. A first pass (K9CODESEC-1981) fixed two data races but instability remained, so the flag stayed off.

This PR finishes that investigation. The user-visible drift turned out not to be the parallel-parsing feature at all, but a pre-existing bug in the query-evaluation phase that affects any multi-worker scan of a large repo: DecodeQueryResults gated result decoding with a 60s per-query wall-clock deadline, and a high-volume rule's decode loop (one HCL re-parse per result) gets starved under worker contention, blows the budget at a timing-dependent point, and — via an ineffective break (go vet SA4011) — silently drops the remaining findings with no error. Result: non-deterministic, -race-clean, only at scale.

With that fixed (plus the two remaining races), parallel parsing is safe to re-enable as an opt-in flag.

JIRA Ticket

Changes

  • pkg/engine/inspector.go(root cause) remove the per-query wall-clock deadline on result decoding; decode all items, honoring only real scan cancellation. Fix the ineffective break (was breaking the select, not the loop — SA4011).
  • pkg/engine/vulnerability_builder.go — remove the dead detector.SetupLogs(...) call (shared-field write from workers; field is unused).
  • pkg/detector/docker/docker_detect.go — deep-copy lines before prepareDockerFileLines mutates them, so concurrent detect-line calls don't corrupt a shared *FileMetadata backing array.
  • cmd/scanner/scan.go — restore the hidden --x-parallelparsing CLI flag (default false) and wire it into the feature-flag override.
  • pkg/featureflags/local_evaluator.go — leave IaCEnableKicsParallelFileParsing default false; parallel parsing is opt-in.

Author Checklist

  • I have reviewed my own PR.
  • I have added or updated relevant unit tests where necessary. If no tests are added, I've explained why.
  • All new and existing tests pass.
  • I have tested my changes on staging (if applicable).
  • I have updated any relevant documentation (if applicable).

QA Instruction

  • Scan a large terraform repo (e.g. ~30k files) several times without the flag — the finding count should now be identical every run (previously it drifted).
  • Scan it again with --x-parallelparsing — finding count / SARIF results should be identical to the serial run, and the scan should be ~30% faster.
  • Run go test -race ./pkg/detector/docker ./pkg/parser/terraform — no data races.
  • Run go vet ./pkg/engine/... — the SA4011 "ineffective break" warning is gone.

Blast Radius

DataDog IaC Scanner only (engine / detectors / CLI). Important: the decode-timeout removal affects all multi-worker scans, not just --x-parallelparsing. On large repos that previously hit the truncation, finding counts may increase (we stop silently dropping real findings) and become deterministic — so SARIF / baseline counts for big repos can change. The --x-parallelparsing re-enable itself is opt-in (default off), so no behavior change unless the flag is passed.

Additional Notes

The two race fixes are validated by -race; the decode fix is validated on full big repo scan: 3+ runs now produce a bit-identical result (count + similarity-ID hash) matching the serial baseline, and parallel parsing runs ~32% faster with identical findings.

Mode Run 1 Run 2 Avg
Serial (flag off) 6m12s 6m10s 6m11s
Parallel (--x-parallelparsing) 4m09s 4m14s 4m11s

~32% faster (1.48×), findings identical across all runs (same similarity-ID hash).

I submit this contribution under the Apache-2.0 license.

@MarshalX
MarshalX requested a review from a team as a code owner June 23, 2026 10:55
@datadog-official

datadog-official Bot commented Jun 23, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 75.00%
Overall Coverage: 49.24% (+0.11%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3494c4d | Docs | Datadog PR Page | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 605a1870a9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread cmd/scanner/scan.go
MikaYuoadas
MikaYuoadas previously approved these changes Jun 23, 2026
@MarshalX
MarshalX merged commit 2c268dd into main Jun 23, 2026
19 checks passed
@MarshalX
MarshalX deleted the ilya.siamionau/K9CODESEC-2411/race-hunt branch June 23, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants