[INC-56036] avoid stack overflow in secrets scanner by splitting scan and validate#921
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit intoJun 16, 2026
Conversation
`Scanner::scan_with_options(validate_matches=true)` overflows the stack inside dd-sds on large inputs. Switch `find_secrets` back to `scan()` followed by a separate `validate_matches()` call until the recursion is fixed upstream. Refs: incident-56036
Contributor
There was a problem hiding this comment.
Pull request overview
This PR mitigates dd-sds stack overflows encountered when scanning very large repositories by changing the secrets scanning flow in crates/secrets to decouple match collection from validation.
Changes:
- Replace
Scanner::scan_with_options(validate_matches=true)withScanner::scan()followed by an explicitScanner::validate_matches()call (when validation is enabled). - Mark the supporting-rule match-pairing test as ignored, documenting the known regression until
dd-sdsis patched upstream.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
isabella-garza-datadog
approved these changes
Jun 16, 2026
isabella-garza-datadog
left a comment
isabella-garza-datadog
left a comment
Contributor
There was a problem hiding this comment.
LGTM as a short term fix while we work on resolving the root cause in dd-sds
jasonforal
approved these changes
Jun 16, 2026
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
akim.sadaoui/incident-56036/fix-stack-overflow
branch
June 16, 2026 19:09
Merged
isabella-garza-datadog
added a commit
that referenced
this pull request
Jun 22, 2026
gh-worker-dd-mergequeue-cf854d Bot
added a commit
that referenced
this pull request
Jun 29, 2026
feat(secrets): bump dd-sds to b2dca51 and revert #921 stack-overflow workaround Co-authored-by: isabella-garza-datadog <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scanner::scan_with_options(validate_matches=true)(introduced in f51ffceb [SDS] Implement SDS supporting rules) triggers a stack overflow insidedd-sdswhen scanning very large monorepos. In prod,code-workload-runner-secretshas been crashing ~40-75k tasks/day on Datadog dogfood orgs since the v0.8.5 cut on 2026-05-29.Switch
find_secretsback to the pre-v0.8.5 pattern:scan()to collect matches, thenvalidate_matches()as a separate step (which uses the bounded rayon thread pool internally).The two paths are equivalent for plain validators. They differ for the new
is_supporting_ruleHTTP match-pairing feature: cross-rule template-variable resolution (a supporting rule's match feeding$PARAMplaceholders in a primary rule's validator URL) only fires insidescan_with_options. The corresponding test is marked#[ignore]until the underlying recursion is fixed upstream indd-sds.Incident: https://app.datadoghq.com/incidents/56036
Test plan
cargo test --workspace— 487 passed, 2 ignored.dd-sourcemonorepo with the staging configuration before the fix (fatal runtime error: stack overflow, aborting).dd-sourcescan (894k files) completes in ~140s with identical results to the pre-regression v0.8.4 baseline (515 files with secrets, 1020 total).