ci: sandbox govulncheck and latest-dependency smoke tests by kakkoyun · Pull Request #4598 · DataDog/dd-trace-go

kakkoyun · 2026-03-25T15:21:40Z

What does this PR do?

Runs the highest-risk dependency/toolchain CI paths inside geomys/sandboxed-step, pinned to v1.2.1's immutable SHA for the repo's SHA-pinning policy.

Concretely, this PR:

wraps blocking govulncheck execution in a gVisor sandbox
adds sandboxed contrib-module SARIF generation for GitHub Code Scanning
keeps the official golang/govulncheck-action for core SARIF analysis from chore(ci): govulncheck - use official golang/govulncheck-action for SARIF analysis #4599
wraps smoke-test go get -u / requested go-libddwaf dependency upgrade compilation in a sandbox
wraps contrib smoke tests that run go get -u before tests in a sandbox
wraps gotip testing, where CI downloads and runs the latest Go toolchain, in a sandbox
restores/cleans workspaces after sandboxed steps before any upload/token-consuming follow-up steps, preserving only required reports, coverage, or SARIF artifacts

Motivation

go get -u, go install ...@latest, gotip download, and govulncheck all intentionally pull and execute code from outside this repository. That is exactly the class of CI work where sandboxing helps: it strips most of the ambient authority that a GitHub-hosted runner otherwise gives to dependency code.

This builds on the checkout hardening already in the workflows (persist-credentials: false) by also isolating the execution environment with gVisor.

Notes on implementation

geomys/sandboxed-step is pinned by commit SHA because this repository has sha_pinning_required: true for GitHub Actions.
Steps that need artifacts after sandbox execution use persist-workspace-changes: 'true', then immediately run git reset --hard and git clean -fdx with narrow exclusions for the required output files.
Govulncheck tools are installed into ${GITHUB_WORKSPACE}/bin inside the sandbox so subsequent commands in the same sandboxed step can find them on PATH.

Validation

Local:

make lint/action
make -C .github/workflows test-apps.yml && git diff --exit-code -- .github/workflows/test-apps.yml
bash -n scripts/ci_test_contrib.sh scripts/ci_test_core.sh scripts/install_tools.sh scripts/lint.sh
git diff --check
YAML parse checks with yq

CI/manual runs:

Manual Govulncheck run passed after verifying sandbox logs, SARIF preservation, and Code Scanning upload.
Manual Smoke Tests run passed, including sandboxed updated-dependency compile and sandboxed go get -u contrib smoke jobs.
Manual Gotip Testing run passed, including sandboxed gotip download/test execution.
Latest PR GitHub checks show no failures; only external GitLab benchmark/mergegate aggregate checks were still pending at last inspection.

References

geomys/sandboxed-step
gVisor
Filippo Valsorda: Turn Dependabot Off — CI sandboxing discussion
PR #4595 — govulncheck workflow foundation
PR #4599 — official govulncheck-action SARIF analysis

Reviewer's Checklist

N/A — CI/workflow-only change, no Go code modified

codecov · 2026-03-25T15:26:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.46%. Comparing base (5fd6d9c) to head (f773ab6).

Additional details and impacted files

see 444 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-prod-us1-6 · 2026-03-25T15:26:30Z

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 61.80% (+4.01%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f773ab6 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-03-25T15:47:38Z

Benchmarks

Benchmark execution time: 2026-05-12 16:18:19

Comparing candidate commit f773ab6 in PR branch kakkoyun/govulncheck-sandboxed-step with baseline commit 5fd6d9c in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 273 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

scenario:BenchmarkOTLPTraceWriterFlush

…#4605) ## Why The `govulncheck-analysis` job (non-blocking SARIF upload) only scanned core packages. Contrib module vulnerabilities were caught by `govulncheck-tests` (blocking, sandboxed) but never appeared in the GitHub Security tab / Code Scanning. This created a gap: a contrib CVE would fail the PR check but leave no trace in the Security dashboard. This PR closes that gap by adding a parallel non-blocking `govulncheck-contribs-analysis` job that scans every contrib module and uploads results to GitHub Code Scanning. ## What - **New script**: `govulncheck-contribs-sarif.sh` — mirrors `govulncheck-contribs-v2.sh` but uses `-format sarif`. It scans each contrib module, writes a per-module SARIF file to a temp directory, then merges all runs into a single output file via `jq`. - **New job**: `govulncheck-contribs-analysis` — sets up Go, installs govulncheck, runs the script, and uploads the merged SARIF under the `govulncheck-contribs` category. - **Updated comment** in `govulncheck-analysis`: replaces the `TODO` with a reference to the new job. - **Updated `paths` filter**: adds `govulncheck-contribs-sarif.sh` so changes to the script trigger the workflow. ## Design decisions **Why merge SARIF with `jq` instead of uploading per-module files?** The `upload-sarif` action accepts a directory, but uploading N files (one per contrib module) would create N separate tool runs in Code Scanning, making the Security tab noisy. Merging runs into one SARIF file keeps results grouped under a single `govulncheck-contribs` category. **Why not sandbox this job?** The `govulncheck-contribs-analysis` job is non-blocking and informational, consistent with `govulncheck-analysis`. The sandboxed `govulncheck-tests` job provides the security boundary for blocking checks. ## Stack This PR is part of a stack: ``` main └── #4595 kakkoyun/govulncheck (base) └── #4597 kakkoyun/govulncheck-persist-credentials └── #4598 kakkoyun/govulncheck-sandboxed-step └── #4599 kakkoyun/govulncheck-action-allowlist └── this PR kakkoyun/govulncheck-contrib-sarif ``` Merge order: #4595 → #4597 → #4598 → #4599 → this PR. ## Test plan - [ ] CI passes on this branch (all three jobs green) - [ ] `govulncheck-contribs-analysis` job appears in the Actions run - [ ] SARIF upload succeeds (check `upload-sarif` step output) - [ ] GitHub Security tab shows findings under `govulncheck-contribs` category after merge to main

kakkoyun · 2026-04-23T08:58:26Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2318e49fea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-23T09:03:59Z

+            go install golang.org/x/vuln/cmd/govulncheck@latest
+            govulncheck -format sarif \


Put govulncheck on PATH before invoking it

In this sandbox go install does not make the next govulncheck command resolvable: Go installs commands to GOBIN/GOPATH/bin by default (https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies), while geomys/sandboxed-step constructs PATH from standard directories plus RUNNER_TOOL_CACHE entries only (https://github.com/geomys/sandboxed-step/blob/v1.2.1/generate-config.go#L1232-L1269). After setup-go, the GOPATH bin added on the host is therefore dropped inside the sandbox, so this line and the same pattern in govulncheck-tests/the contrib script fail with govulncheck: command not found; invoke $(go env GOPATH)/bin/govulncheck, extend PATH inside the sandbox, or use go run.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-04-23T09:03:59Z

+        uses: geomys/[email protected]
+        with:
+          run: |-


Persist the SARIF output out of the sandbox

This step writes govulncheck.sarif inside geomys/sandboxed-step, but the following upload-sarif step runs back on the host and reads govulncheck.sarif from the workspace. The action documents that workspace changes do not persist by default and its default for persist-workspace-changes is false (https://raw.githubusercontent.com/geomys/sandboxed-step/v1.2.1/README.md), so once the command succeeds the report is discarded with the sandbox overlay and Code Scanning has no SARIF file to upload; set persist-workspace-changes: 'true' for this SARIF-producing step or copy the report out another way.

Useful? React with 👍 / 👎.

…andboxing Confines govulncheck execution using gVisor to prevent supply chain attacks from exfiltrating tokens or making network calls in CI. Any malicious code in a scanned dependency runs inside the sandbox with no ambient authority. This PR will fail CI: geomys/sandboxed-step is not in the DataDog enterprise action allowlist. It exists to start the discussion for adding it. See PR description for context. Ref: https://words.filippo.io/dependabot/ Ref: https://github.com/geomys/sandboxed-step Signed-off-by: Kemal Akkoyun <[email protected]>

kakkoyun · 2026-04-29T09:46:39Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 852852bc5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-29T09:51:54Z

+        id: restore
+        if: always()
+        run: |-
+          git reset --hard HEAD


Reset to a trusted ref after the sandbox

Because persist-workspace-changes: 'true' lets the sandbox write the checkout, untrusted code can also rewrite .git/HEAD or refs before this host-side cleanup. git reset --hard HEAD then trusts that attacker-controlled ref, so the following local upload actions can run attacker-controlled .github/actions/... code with the Datadog/Codecov credentials; capture the checked-out SHA before entering the sandbox or re-checkout a fresh workspace before invoking credentialed/local actions.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 852852bc5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-29T09:56:46Z

        run: |-
-          export PATH="${{ github.workspace }}/bin:${PATH}"
-          ./scripts/ci_test_contrib.sh smoke ${{ toJson(matrix.chunk) }}
+          git reset --hard HEAD


Reset to a trusted ref after the sandbox

This go-get-u path persists the sandbox workspace, and the sandboxed-step README says following steps must treat that workspace as untrusted. When the sandboxed go get -u/tests can rewrite .git/HEAD and objects, git reset --hard HEAD resets to attacker-controlled content rather than the checked-out commit; the subsequent local ./.github/actions/dd-ci-upload and codecov-upload steps then run with Datadog/Codecov credentials. Capture the checked-out SHA before the sandbox and reset to that SHA, or use a fresh checkout before credentialed local actions.

Useful? React with 👍 / 👎.

`persist-workspace-changes: true` lets a sandboxed step write anywhere in `${GITHUB_WORKSPACE}`, including `.git/HEAD`, `.git/refs/*`, and tracked files such as `.github/actions/codecov-upload/action.yml`. The previous post-sandbox `git reset --hard HEAD` therefore trusted whatever the sandbox left behind, so a compromised dependency could backdoor the local upload actions and exfiltrate `CODECOV_TOKEN` / Datadog API keys once the host runner ran them. Replace the `reset + clean` cleanup with a three-step sequence: 1. Capture `git rev-parse HEAD` to a step output before the sandbox runs. 2. After the sandbox, tar test results and coverage files into `${RUNNER_TEMP}/sandbox-artifacts` (regular files only, so any sandbox-planted symlinks are skipped). 3. Re-checkout the captured SHA with `actions/checkout … clean: true`, then extract the artifacts back. The credentialed local actions (`./.github/actions/dd-ci-upload`, `./.github/actions/codecov-upload`, `DataDog/dd-sts-action`) keep their `steps.restore.outcome == 'success'` gate, so they only run on a freshly-checked-out workspace. Applied to: - `.github/workflows/gotip-testing.yml` (`test-tip` job) - `.github/workflows/smoke-tests.yml` (`go-get-u` job) `govulncheck.yml` `govulncheck-contribs-analysis` also uses `persist-workspace-changes`, but its only post-sandbox step is `github/codeql-action/upload-sarif` (loaded from a pinned external action, not the workspace) which only consumes the SARIF as data, so it does not need the same hardening.

darccio

Nice work!

darccio · 2026-05-12T17:49:37Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-05-12T17:49:42Z

View all feedbacks in Devflow UI.

2026-05-12 17:49:42 UTC ℹ️ Start processing command /merge

2026-05-12 17:49:47 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 15m (p90).

2026-05-12 18:03:34 UTC ℹ️ MergeQueue: This merge request was merged

kakkoyun mentioned this pull request Mar 25, 2026

chore(ci): govulncheck - use official golang/govulncheck-action for SARIF analysis #4599

Merged

1 task

kakkoyun changed the title ~~ci(govulncheck): add gVisor sandboxing via geomys/sandboxed-step~~ feat(ci): govulncheck - add gVisor sandboxing via geomys/sandboxed-step Mar 25, 2026

kakkoyun added the AI Assisted AI/LLM assistance used in this PR (partially or fully) label Mar 25, 2026

kakkoyun force-pushed the kakkoyun/govulncheck-persist-credentials branch from 34af022 to 31f8814 Compare March 26, 2026 10:07

kakkoyun force-pushed the kakkoyun/govulncheck-sandboxed-step branch from 42b3d61 to f5cf6b9 Compare March 26, 2026 10:10

kakkoyun mentioned this pull request Mar 26, 2026

chore(ci): govulncheck - extend SARIF upload to cover contrib modules #4605

Merged

4 tasks

Base automatically changed from kakkoyun/govulncheck-persist-credentials to kakkoyun/govulncheck March 27, 2026 10:53

kakkoyun force-pushed the kakkoyun/govulncheck-sandboxed-step branch from f5cf6b9 to aa79d34 Compare March 27, 2026 11:13

Base automatically changed from kakkoyun/govulncheck to main March 27, 2026 11:49

kakkoyun force-pushed the kakkoyun/govulncheck-sandboxed-step branch 3 times, most recently from a21d539 to 2318e49 Compare April 23, 2026 08:57

chatgpt-codex-connector Bot reviewed Apr 23, 2026

View reviewed changes

kakkoyun added 2 commits April 29, 2026 09:26

ci: sandbox latest-dependency smoke tests

2ab67a7

kakkoyun force-pushed the kakkoyun/govulncheck-sandboxed-step branch from 643b259 to 2ab67a7 Compare April 29, 2026 07:55

kakkoyun added 3 commits April 29, 2026 10:00

ci: pin sandboxed-step action

555d366

ci: make sandboxed govulncheck tools available

b95ea0a

ci: preserve sandboxed govulncheck SARIF

852852b

kakkoyun changed the title ~~feat(ci): govulncheck - add gVisor sandboxing via geomys/sandboxed-step~~ ci: sandbox govulncheck and latest-dependency smoke tests Apr 29, 2026

chatgpt-codex-connector Bot reviewed Apr 29, 2026

View reviewed changes

kakkoyun marked this pull request as ready for review April 29, 2026 09:52

kakkoyun requested a review from a team as a code owner April 29, 2026 09:52

chatgpt-codex-connector Bot reviewed Apr 29, 2026

View reviewed changes

darccio approved these changes Apr 29, 2026

View reviewed changes

Merge branch 'main' into kakkoyun/govulncheck-sandboxed-step

ac4756f

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed and removed mergequeue-status: waiting labels Apr 30, 2026

Merge branch 'main' into kakkoyun/govulncheck-sandboxed-step

e2992bb

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed and removed mergequeue-status: removed mergequeue-status: waiting labels May 12, 2026

Merge branch 'main' into kakkoyun/govulncheck-sandboxed-step

f773ab6

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: removed mergequeue-status: queued labels May 12, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 52b926a into main May 12, 2026
164 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the kakkoyun/govulncheck-sandboxed-step branch May 12, 2026 18:03

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: done and removed mergequeue-status: in_progress labels May 12, 2026

		go install golang.org/x/vuln/cmd/govulncheck@latest
		govulncheck -format sarif \

Uh oh!

Conversation

kakkoyun commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Notes on implementation

Validation

References

Reviewer's Checklist

Uh oh!

codecov Bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-prod-us1-6 Bot commented Mar 25, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Known flaky benchmarks

Uh oh!

kakkoyun commented Apr 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

kakkoyun commented Apr 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

darccio left a comment

Choose a reason for hiding this comment

Uh oh!

darccio commented May 12, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kakkoyun commented Mar 25, 2026 •

edited

Loading

codecov Bot commented Mar 25, 2026 •

edited

Loading

datadog-prod-us1-6 Bot commented Mar 25, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

pr-commenter Bot commented Mar 25, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented May 12, 2026 •

edited

Loading