Add test-rules command and CI workflow#191
Conversation
|
🎯 Code Coverage (details) 🔗 Commit SHA: 65c6531 | Docs | Datadog PR Page | Give us feedback! |
b3e4410 to
fc498c2
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
4b420ec to
b01db3b
Compare
b01db3b to
7e686fd
Compare
| # Runs the scanner's engine against the rule fixtures stored in the Datadog | ||
| # backend. This catches engine regressions that break existing rule tests. | ||
| # | ||
| # Requires DD_API_KEY and DD_APP_KEY repository secrets (and optionally |
There was a problem hiding this comment.
Remove the DD_API_KEY and DD_APP_KEY and DD_SITE. They are not required for the default rules and avoids problems with forks and external PRs.
There was a problem hiding this comment.
I always confuse the config file and the rules aahhh
| module_support | ||
|
|
||
| # local scanner binary | ||
| /scanner |
There was a problem hiding this comment.
It prevents accidentally committing a locally-built binary. I ran go build ./cmd/scanner locally which dropped scanner in the root, and ended up committing it by accident
| platformStr := rule.Platform | ||
| resourcePlatforms := testResourcePlatformsFor(platformStr) |
There was a problem hiding this comment.
| platformStr := rule.Platform | |
| resourcePlatforms := testResourcePlatformsFor(platformStr) | |
| resourcePlatforms := testResourcePlatformsFor(rule.Platform) |
Also, are these supposed to be the platforms in the rule, or the platforms specified in the command line? Because my original test code uses the command line, and the validateResultShape function then uses resourcePlatforms to ...
I'm not sure that resourcePlatforms is needed at all! It only checks that the rule's platform is in resourcePlatforms and, if so, it does extra checks. It doesn't fail anything if it's not there. But the platform is going to be in resourcePlatforms whether we use the command line or the rule's platform, so it's irrelevant. Can you have a look?
There was a problem hiding this comment.
I removed resourcePlatforms entirely safely
| // testResourcePlatformsFor returns the set of platforms that require resourceType/resourceName. | ||
| // Mirrors the logic used in the default-rules test tool. | ||
| func testResourcePlatformsFor(platform string) map[string]struct{} { | ||
| resourcePlatforms := map[string]struct{}{ |
There was a problem hiding this comment.
Removed it, now it will directly look at platformsRequiringResourceKeys
| @@ -0,0 +1,518 @@ | |||
| package main | |||
There was a problem hiding this comment.
Suggestion (not necessarily for this PR): put the code that runs tests in a separate package, and then we can import it from the default-rules repo and use it there without having two copies of the same test code.
There was a problem hiding this comment.
I will follow up in another PR
2 participants
Motivation
When the scanner engine changes, there is no automated way to verify that existing rules still produce the expected findings. The rule test fixtures live in the default-rules repository and are now also stored in the backend API (alongside the rules themselves). This PR wires up the scanner to fetch and run those tests, so every PR gets a signal that the engine has not regressed against the full default rule suite.
Changes
API client
The
Clientinterface gains aGetDefaultRulesetWithTestsmethod that calls the same endpoint asGetDefaultRulesetbut withinclude_tests=true. Three new types mirror the backend model:RuleTests,TestFile, andTestFinding. TheRulestruct gains aTests *RuleTestsfield that is populated when tests are requested.test-rules command
A new
test-rulessubcommand fetches the default ruleset (with tests) from the Datadog API, materialises each rule's fixture files into a temporary directory, and runs the scanner engine against them. Positive fixtures are compared against the stored expected findings; negative fixtures must produce no findings. The command exits non-zero if any rule fails, prints a per-rule pass/fail summary, and accepts--typeand--ruleflags to scope the run to a subset of platforms or rules.The rule is converted to a
QueryMetadatavia a JSON:API round-trip through the scanner's owndatadog.Ruletype, meaning the test exercises the exact same rule representation the scanner uses in production.CI workflow
.github/workflows/test-rules.ymlruns the command on every pull request targetingmainand on every push tomain. It requiresDD_API_KEYandDD_APP_KEYrepository secrets. Fork pull requests will pass vacuously since secrets are not available in that context.QA Instruction
CI should pass
Impact
Adds a new CLI subcommand (
test-rules) and a CI workflow. No changes to scan behaviour, output formats, or existing commands.