Skip to content

Add a long-lived serve HTTP server#211

Merged
MarshalX merged 4 commits into
mainfrom
ilya.siamionau/K9CODESEC-2453/http-server
Jun 29, 2026
Merged

Add a long-lived serve HTTP server#211
MarshalX merged 4 commits into
mainfrom
ilya.siamionau/K9CODESEC-2453/http-server

Conversation

@MarshalX

@MarshalX MarshalX commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Motivation

Bring IaC scanning into the Datadog IDE extensions, the same way SAST and secret scanning already work.

Today the scanner is a one-shot CLI (requires a git remote, fetches rules from the API per scan, reads files from disk, writes SARIF) — unsuitable for an IDE. The extension needs a long-lived, pure-engine HTTP server it can drive by pushing rules + file content over HTTP, mirroring the datadog-static-analyzer-server contract so it can reuse the same binary-manager lifecycle (keep-alive ping, graceful shutdown, version discovery).

JIRA Ticket: K9CODESEC-2453

Changes

serve is purely additive. The scan CLI is unchanged, and the virtual-FS refactor is behavior-preserving (the CLI uses a DiskFS passthrough).

  • serve command + lifecycle mirroring the static-analyzer server: flags --port / --address / --keep-alive-timeout / --enable-shutdown (--use-rules-cache accepted as a no-op); GET /pingpong, GET /version & /revision, GET|POST /shutdown (204/403), keep-alive idle-exit goroutine, X-iac-scanner-server-* + CORS headers.
  • GET /ide/v1/iac/supported-files — static {patterns, strategy} map (directory vs single_file), derived from analyzer.PossibleFileTypes() with a drift test.
  • POST /ide/v1/iac/analyze — content-push scan. Request {files:[{path,content}], rules, config} (raw content); response {findings, missing_files, failed_queries}. No git, no network, no SARIF.
  • Content-push + virtual FS — new pkg/vfs (FS / DiskFS / MemFS). MemFS resolves Terraform siblings, vars, and local modules in memory and records referenced-but-unpushed paths as workspace-relative missing_files (the hybrid-escalation signal; CWD-independent, so one server can serve many workspaces). New MemorySourceProvider feeds pushed files to the engine instead of walking disk. vfs.FS is threaded through the Terraform parser/data-source/modules, the inspector, and extension detection.
  • scan.Client seamsClient.Scan(ctx) (in-memory results) plus WithFS / WithQuerySourceFactory / WithInMemoryScan options, and an in-memory initScan branch that bypasses the disk analyzer.

Author Checklist

  • I have reviewed my own PR.
  • I have added or updated relevant unit tests where necessary. If no tests are added, I've explained why.
  • All new and existing tests pass.
  • I have tested my changes on staging (if applicable).
  • I have updated any relevant documentation (if applicable).

QA Instruction

The server needs the Rego support libraries on disk; rules and file content are pushed per request.

go build -o /tmp/iac-scanner ./cmd/scanner
/tmp/iac-scanner serve --port 49200 --enable-shutdown --libraries-path ./assets/libraries
  • curl 127.0.0.1:49200/pingpong; GET /ide/v1/iac/supported-files → the {patterns,strategy} array.
  • POST /ide/v1/iac/analyze: fetch rules anonymously from GET https://api.datadoghq.com/api/v2/static-analysis/iac/rulesets/default-ruleset (base64-decode each rego_query, lowercase platform). Push a Terraform main.tf that uses var.x defined in a sibling variables.tf → expect findings, with var.x resolved (proves in-memory sibling resolution). Push a module { source = "../mod" } whose directory you don't send → expect "mod" to appear in missing_files as a workspace-relative path.
  • CLI regression: datadog-iac-scanner scan … SARIF output must be unchanged (DiskFS path is behavior-preserving).
  • Concurrency: go build -race, then fire several parallel /analyze requests — no DATA RACE in the log and the process must not exit 66.

Blast Radius

Only datadog-iac-scanner. serve is additive; the scan CLI and its SARIF output are unchanged (the vfs refactor is a DiskFS passthrough, covered by the existing suite). No backend/API changes.

Additional Notes

  • Deferred: Helm rendering is disabled in content-push mode (the Helm SDK needs a real chart directory) — pushed charts fall back to raw-YAML scanning; the $ref/include resolver isn't virtualized yet (niche OpenAPI/Ansible cross-file refs).

I submit this contribution under the Apache-2.0 license.

@datadog-official

datadog-official Bot commented Jun 23, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 69.57%
Overall Coverage: 49.52% (+0.19%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 545d161 | Docs | Datadog PR Page | Give us feedback!

Comment thread pkg/engine/provider/filesystem.go Fixed
@MarshalX
MarshalX force-pushed the ilya.siamionau/K9CODESEC-2453/http-server branch 3 times, most recently from c22bb36 to 4460e64 Compare June 25, 2026 10:51
@MarshalX
MarshalX force-pushed the ilya.siamionau/K9CODESEC-2453/http-server branch from 6299d64 to 545d161 Compare June 25, 2026 12:11
@MarshalX
MarshalX marked this pull request as ready for review June 25, 2026 12:13
@MarshalX
MarshalX requested a review from a team as a code owner June 25, 2026 12:13

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 545d1611d4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread pkg/server/analyze.go
Comment on lines +253 to +254
if len(rules) == 0 {
return fsSource, nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a real fallback rule source

When /ide/v1/iac/analyze is called without rules, this branch returns a filesystem source pointed at s.cfg.QueriesPath, whose default is ./assets/queries; I checked the repo tree and that path is absent, while FilesystemSource.localQueryDirs silently skips unreadable paths, so these requests succeed with zero loaded queries and report no findings. If omitting rules is an advertised fallback, point this at a packaged ruleset or return an error instead of running an empty scan.

Useful? React with 👍 / 👎.

@MikaYuoadas MikaYuoadas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Comment thread pkg/server/server.go
Comment on lines +155 to +156
// No WriteTimeout: a cold scan recompiles the rule corpus and can take
// several seconds until the prepared-query cache lands.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Suggestion: Why not put a generous write timeout still?
Without it a client that stops reading can hold the goroutine indefinitely.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love to, but I have no clear vision on what value to use... It could take minutes on huuuge repos. Should I just add and option for serve ... sub command?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will have a follow up PR with rules caching etc. I will not bother it here

@MarshalX
MarshalX merged commit 46e72d1 into main Jun 29, 2026
20 checks passed
@MarshalX
MarshalX deleted the ilya.siamionau/K9CODESEC-2453/http-server branch June 29, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants