Skip to content

Unify Datadog fingerprint across findings, reports, and storage#200

Merged
whitemerch merged 3 commits into
mainfrom
chakib.hamie/unify_finding_fingerprint
Jun 22, 2026
Merged

Unify Datadog fingerprint across findings, reports, and storage#200
whitemerch merged 3 commits into
mainfrom
chakib.hamie/unify_finding_fingerprint

Conversation

@whitemerch

@whitemerch whitemerch commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Motivation

Finding identity was split between an in-engine legacy hash, per-format logic, and in-memory deduplication keys, which made counts and SARIF partial fingerprints easy to disagree. This change makes one stable Datadog fingerprint the source of truth from summary creation through exports and storage deduplication, and removes unused legacy similarity and exclude-results plumbing.

Changes

Model and engine

Introduce GetDatadogFingerprintHash, stamp each VulnerableFile in CreateSummary, and remove SimilarityID / related counters and the old similarity computation package. The inspector and vulnerability builder no longer compute or suppress by legacy similarity IDs.

Scan and storage

Summary counters drop failed-similarity metrics; the scan client no longer builds exclude-results maps; in-memory deduplication keys findings with the same fingerprint hash used in summaries.

Configuration

Remove the legacy exclude-results option from parsing, migration comments, and legacy_config.md because suppression no longer keys off similarity IDs.

Reports and fixtures

All exporters read the stamped Fingerprint field (including CSV column rename). E2E comparison order and sample JSON under test/assets are updated for the new summary shape.

Author Checklist

  • I have reviewed my own PR.
  • I have added or updated relevant unit tests where necessary. If no tests are added, I've explained why.
  • All new and existing tests pass.
  • I have tested my changes on staging (if applicable).
  • I have updated any relevant documentation (if applicable).

QA Instruction

  1. Run go test -count=1 ./... from the repository root (full suite passes locally).
  2. Optional: make build then ./bin/datadog-iac-scanner scan on a small Terraform fixture and confirm SARIF partialFingerprints.DATADOG_FINGERPRINT matches the fingerprint in JSON summary output for the same finding.

Blast Radius

Affects the open-source CLI and library surfaces: JSON summary schema (removed similarity-related fields and counter), CSV column fingerprint instead of similarity_id, SARIF and other report formats, and in-memory deduplication granularity (aligned with the fingerprint hash rather than line plus search key).

Additional Notes

Downstream, findings with the same fingerprint hash are collapsed into one item. That makes the fingerprint definition load-bearing for what customers see after ingestion. This PR aligns the scanner, in-memory dedupe, and SARIF on one hash so behavior matches that contract; revisiting whether the hash is granular enough (for example across tricky Terraform or multi-rule scenarios) is intentionally follow-up work, not in scope here.

For integrations we care about most today, SARIF is the format that is consumed downstream. Other report outputs (JSON summary, CSV, JUnit, etc.) are updated to stay consistent with the same fingerprint field, but they are not the primary integration surface in practice.

I submit this contribution under the Apache-2.0 license.

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jun 19, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 91.14%
Overall Coverage: 49.55% (+0.18%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e580cfa | Docs | Datadog PR Page | Give us feedback!

@whitemerch
whitemerch changed the base branch from main to chakib.hamie/local_modules_finding_identity June 19, 2026 13:27
@whitemerch
whitemerch force-pushed the chakib.hamie/unify_finding_fingerprint branch from 712f7cc to 9f95ecb Compare June 19, 2026 13:30
chatgpt-codex-connector[bot]

This comment was marked as resolved.

…lint lll on module resolve and tfeval cache comments.
@whitemerch
whitemerch force-pushed the chakib.hamie/unify_finding_fingerprint branch from 9f95ecb to e580cfa Compare June 19, 2026 13:45
@whitemerch
whitemerch marked this pull request as ready for review June 19, 2026 13:50
@whitemerch
whitemerch requested a review from a team as a code owner June 19, 2026 13:50
MikaYuoadas
MikaYuoadas previously approved these changes Jun 22, 2026

@MikaYuoadas MikaYuoadas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I like that we have now a convenient way to have specific fingerprinting algorithm per platform.

Base automatically changed from chakib.hamie/local_modules_finding_identity to main June 22, 2026 07:05
@whitemerch
whitemerch dismissed MikaYuoadas’s stale review June 22, 2026 07:05

The base branch was changed.

@ChouraquiBen ChouraquiBen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No changes since Akim's latest review, only the branch rebased. Reapproving in his stead

@whitemerch

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 22, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-22 08:00:38 UTC ℹ️ Start processing command /merge


2026-06-22 08:00:43 UTC ❌ MergeQueue

This PR is already merged

@whitemerch
whitemerch merged commit 3eefc98 into main Jun 22, 2026
5 checks passed
@whitemerch
whitemerch deleted the chakib.hamie/unify_finding_fingerprint branch June 22, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants