Motivation

A scan runs several worker pools, and they nest. The scanner fans out one goroutine per service, and inside each service the query-evaluation pool spins up its own GOMAXPROCS workers. There were five such pools in total (query eval, file reading, HCL body parsing, module-variable parsing, file-type detection), each independently sizing itself to roughly the core count, with no shared notion of how much CPU work was allowed to run at once.

The result: in-flight CPU-bound goroutines scaled as Nservices × GOMAXPROCS, well over 100 goroutines competing for 16 cores (on my laptop). Past a point, adding cores made scans slower, not faster (negative scaling): the pools oversubscribed the machine and the extra goroutines added scheduler churn, cache thrashing, and lock contention. A GOMAXPROCS sweep confirmed it, latency got worse going from 8 to 16 cores, and concurrent scans showed almost no throughput gain.

On top of that, the five pools were inconsistent: only the file reader clamped its worker count to [4, 64]; the module-variable pool ignored the auto-sizing helper and hardcoded 4; the file-type analyzer is I/O-bound (it reads file content) but was capped at GOMAXPROCS while the otherwise-identical file reader fanned out to 64; and all five were hand-rolled copies of the same channel + WaitGroup + closer boilerplate.

JIRA Ticket: K9CODESEC-2731

• Introduced a process-wide shared CPU budget (weighted semaphore sized to the available CPU count). Every CPU-bound pool (Rego eval, HCL parsing, module-variable parsing) acquires one slot per unit of work, so nesting no longer multiplies, at most AvailableCPUs() CPU-bound goroutines run at once regardless of how deep the pools stack.
• Unified the five worker pools behind one generic helper (utils.ForEach, errgroup-backed), replacing the duplicated fan-out/WaitGroup/closer boilerplate.
• Separated CPU-bound from I/O-bound work: I/O-bound pools (file reading, file-type detection) do not consume the CPU budget (blocking on disk shouldn't hold a CPU slot) and fan out wider than the core count via shared IOMinWorkers/IOMaxWorkers ([4, 64]) constants in workers.go. Fixes the analyzer being throttled to GOMAXPROCS.
• Removed the hardcoded 4 in module-variable parsing and made that pass deterministic (sorted iteration order).
• Single source of truth for core count (utils.AvailableCPUs()) used everywhere instead of ad-hoc GOMAXPROCS reads.
• Added unit tests for the new worker helper and CPU budget; added a benchmark harness for the GOMAXPROCS sweep.

Author Checklist

• I have reviewed my own PR.
• I have added or updated relevant unit tests where necessary. If no tests are added, I've explained why.
• All new and existing tests pass.
• I have tested my changes on staging (if applicable).
• I have updated any relevant documentation (if applicable).

QA Instruction

The change is internal (concurrency), so the observable behavior must be identical while scaling improves:

• Correctness: scan a repo before and after this PR; the findings count and SARIF output must be unchanged.
• Scaling: run the same scan at GOMAXPROCS=4, 8, 16 (e.g. GOMAXPROCS=16 datadog-iac-scanner scan -p <repo> -o <out>). Higher core counts should be the same or faster — in particular 16 cores should no longer be slower than 8 (the previous negative-scaling regression).
• Race safety: go test -race ./pkg/engine/... ./pkg/utils/ ./pkg/analyzer/ ./pkg/parser/terraform/modules/ passes clean.

Blast Radius

Datadog IaC Scanner only. The change is confined to the scanner's internal worker/concurrency layer (pkg/utils, pkg/engine, pkg/analyzer, pkg/parser/terraform/modules, pkg/engine/provider). No public API, CLI flag, rule, or output-format change. No impact on other services or documentation.

I submit this contribution under the Apache-2.0 license.