[K9CODESEC-2947] Add network Terraform module resolvers#248
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ad94f2868
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: 9f3a939 | Docs | Datadog PR Page | Give us feedback! |
…nd download size limits.
b001341 to
aa665ba
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa665ba2cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…chable-host errors to dead-host cache.
Tip
This PR is best reviewed commit by commit.
Motivation
#240 added the offline resolver chain; #242 added pinned-git resolution via bare clones. Many real-world Terraform repos still reference registry modules (
registry.terraform.io/...) and other remote URLs that require a network fetch to materialize.tfcontent.This PR completes the resolver library side of remote module support. Nothing in the scan path invokes these resolvers yet, that wiring is a follow-up.
Related Ticket: K9CODESEC-2947
Changes
Go-getter resolver
Adds
GoGetterResolveron top of hashicorp/go-getter for generic remote downloads (git, HTTP, S3, etc.). Includes global and per-host concurrency limits (IAC_MODULE_FETCH_CONCURRENCY,IAC_MODULE_HOST_FETCH_CONCURRENCY), a 200 MiB scan-level download cap(don't ask me why this value), host allowlists, transient-error retries, and integration with the shared on-diskmoduleCache(content-addressed by source + resolved version).Registry HTTP layer
Adds registry discovery, version resolution, and download-URL translation for public and private registries (
TF_TOKEN_*auth). Go-getter depends on this layer, registry sources are translated toX-Terraform-GetURLs before download, which is why both land in one PR.Source routing
Pinned
git::...?ref=sources stay onBareGitResolver(go-getter explicitly defers them). Unpinnedgit::https://sources keep HTTPS transport instead of being rewritten to SSH. Registry sources with subdirectories (ns/name/provider//subdir) are classified correctly. Pinned registry versions skip network discovery when resolving the cache key, so offline cache hits work.Author Checklist
QA Instruction
CI should pass.
Impact
Additive only: new code under
pkg/parser/terraform/modules/resolver, plusgo.mod/go.sumandLICENSE-3rdparty.csvfor go-getter transitive deps. No CLI, scan, or engine behavior changes.Additional Notes
I submit this contribution under the Apache-2.0 license.