Add a long-lived serve HTTP server#211
Conversation
|
🎯 Code Coverage (details) 🔗 Commit SHA: 545d161 | Docs | Datadog PR Page | Give us feedback! |
c22bb36 to
4460e64
Compare
6299d64 to
545d161
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 545d1611d4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if len(rules) == 0 { | ||
| return fsSource, nil |
There was a problem hiding this comment.
Use a real fallback rule source
When /ide/v1/iac/analyze is called without rules, this branch returns a filesystem source pointed at s.cfg.QueriesPath, whose default is ./assets/queries; I checked the repo tree and that path is absent, while FilesystemSource.localQueryDirs silently skips unreadable paths, so these requests succeed with zero loaded queries and report no findings. If omitting rules is an advertised fallback, point this at a packaged ruleset or return an error instead of running an empty scan.
Useful? React with 👍 / 👎.
| // No WriteTimeout: a cold scan recompiles the rule corpus and can take | ||
| // several seconds until the prepared-query cache lands. |
There was a problem hiding this comment.
💡 Suggestion: Why not put a generous write timeout still?
Without it a client that stops reading can hold the goroutine indefinitely.
There was a problem hiding this comment.
I would love to, but I have no clear vision on what value to use... It could take minutes on huuuge repos. Should I just add and option for serve ... sub command?
There was a problem hiding this comment.
I will have a follow up PR with rules caching etc. I will not bother it here
Motivation
Bring IaC scanning into the Datadog IDE extensions, the same way SAST and secret scanning already work.
Today the scanner is a one-shot CLI (requires a git remote, fetches rules from the API per scan, reads files from disk, writes SARIF) — unsuitable for an IDE. The extension needs a long-lived, pure-engine HTTP server it can drive by pushing rules + file content over HTTP, mirroring the
datadog-static-analyzer-servercontract so it can reuse the same binary-manager lifecycle (keep-alive ping, graceful shutdown, version discovery).JIRA Ticket: K9CODESEC-2453
Changes
serveis purely additive. ThescanCLI is unchanged, and the virtual-FS refactor is behavior-preserving (the CLI uses aDiskFSpassthrough).servecommand + lifecycle mirroring the static-analyzer server: flags--port/--address/--keep-alive-timeout/--enable-shutdown(--use-rules-cacheaccepted as a no-op);GET /ping→pong,GET /version&/revision,GET|POST /shutdown(204/403), keep-alive idle-exit goroutine,X-iac-scanner-server-*+ CORS headers.GET /ide/v1/iac/supported-files— static{patterns, strategy}map (directory vs single_file), derived fromanalyzer.PossibleFileTypes()with a drift test.POST /ide/v1/iac/analyze— content-push scan. Request{files:[{path,content}], rules, config}(raw content); response{findings, missing_files, failed_queries}. No git, no network, no SARIF.pkg/vfs(FS/DiskFS/MemFS).MemFSresolves Terraform siblings, vars, and local modules in memory and records referenced-but-unpushed paths as workspace-relativemissing_files(the hybrid-escalation signal; CWD-independent, so one server can serve many workspaces). NewMemorySourceProviderfeeds pushed files to the engine instead of walking disk.vfs.FSis threaded through the Terraform parser/data-source/modules, the inspector, and extension detection.scan.Clientseams —Client.Scan(ctx)(in-memory results) plusWithFS/WithQuerySourceFactory/WithInMemoryScanoptions, and an in-memoryinitScanbranch that bypasses the disk analyzer.Author Checklist
QA Instruction
The server needs the Rego support libraries on disk; rules and file content are pushed per request.
curl 127.0.0.1:49200/ping→pong;GET /ide/v1/iac/supported-files→ the{patterns,strategy}array.POST /ide/v1/iac/analyze: fetch rules anonymously fromGET https://api.datadoghq.com/api/v2/static-analysis/iac/rulesets/default-ruleset(base64-decode eachrego_query, lowercaseplatform). Push a Terraformmain.tfthat usesvar.xdefined in a siblingvariables.tf→ expect findings, withvar.xresolved (proves in-memory sibling resolution). Push amodule { source = "../mod" }whose directory you don't send → expect"mod"to appear inmissing_filesas a workspace-relative path.datadog-iac-scanner scan …SARIF output must be unchanged (DiskFS path is behavior-preserving).go build -race, then fire several parallel/analyzerequests — noDATA RACEin the log and the process must not exit66.Blast Radius
Only
datadog-iac-scanner.serveis additive; thescanCLI and its SARIF output are unchanged (thevfsrefactor is a DiskFS passthrough, covered by the existing suite). No backend/API changes.Additional Notes
$ref/include resolver isn't virtualized yet (niche OpenAPI/Ansible cross-file refs).I submit this contribution under the Apache-2.0 license.