-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat: Add WASM performance benchmarking infrastructure #7841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add WASM performance benchmarking infrastructure #7841
Conversation
✅ Deploy Preview for openpolicyagent ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
cb1ad4e to
b8779bb
Compare
|
Heya! Thanks for contributing. I'm afraid this feature could have benefited from a bit of discussion and planning beforehand -- either on a github issue/discussion or in the #contributors channel on the OPA slack. Anyhow, it's done now, so let's review. Some upfront comments:
What do you think about adding a wasm-target angle to the benchmarks we have here, for example? They try to assess the performance of different scenarios. Thanks again for your contribution -- Let's discuss further steps! 😃 |
3833525 to
6d6d6eb
Compare
|
Hey @srenatus, thanks for the detailed review! You're right about discussing first - though honestly, code comes way easier to me than planning discussions. After seeing the WASM performance issue in #7759, I built some benchmarking to measure Made the changes you asked for: Removed CI stuff - Yeah, Gobenchdata should be separate. Kept just make bench-wasm. Added missing scenarios:
Better integration - New v1/rego/rego_wasm_bench_test.go adds WASM targets to existing patterns. Can now directly compare topdown vs WASM. Good point about browser vs wasmtime distinction too. Better approach? |
srenatus
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's get this in. Added a few more comments on the latest iteration. Thanks for bearing with me! 🐾
c0c4dca to
4958399
Compare
- Move benchmarks from v1/topdown to v1/rego to resolve import cycle - Add comprehensive WASM performance benchmarks as requested: * BenchmarkSimpleAuthzTargets: Authorization policy performance * BenchmarkBuiltinPerformanceTargets: Builtin function performance * BenchmarkDataSizesTargets: Performance with varying data sizes * BenchmarkPolicyComplexityTargets: Performance with varying policy complexity - Use ParsedModule for consistency with existing benchmarks - Update Makefile bench-wasm target to test both v1/topdown and v1/rego - Add memory benchmark warning about cgo/wasmtime-go limitations Addresses review feedback on PR open-policy-agent#7841 Signed-off-by: Anivar <[email protected]>
|
@srenatus Moved to v1/rego/rego_wasm_bench_test.go as suggested. In addition added compilation, cold start, memory allocation and bundle size benchmarks. The memory benchmarks show WASM reporting 100x less allocations - this is expected since cgo allocations through wasmtime-go aren't tracked by Go's benchmarking. |
srenatus
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! LGTM
- Add comprehensive benchmark package for WASM performance testing - Create bench-wasm CLI tool for running benchmarks - Add Makefile targets for benchmark operations - Integrate benchmarks into CI/CD pipeline - Add documentation for WASM benchmarking - Implement performance regression detection (>5% threshold) This infrastructure helps prevent WASM performance regressions and provides visibility into performance characteristics across releases. Fixes open-policy-agent#7759 Signed-off-by: Anivar Aravind <[email protected]> Signed-off-by: Anivar A Aravind <[email protected]>
Add benchmarks to track WASM target performance and prevent regressions. The benchmarks compare WASM vs regular Rego execution time and test scaling with different policy sizes. This addresses concerns from open-policy-agent#7759 about WASM performance visibility. Signed-off-by: Anivar Aravind <[email protected]> Signed-off-by: Anivar A Aravind <[email protected]>
- Remove CI integration as suggested (out of scope for this PR) - Enhance benchmark coverage with different data sizes and builtin performance - Add WASM target support to existing benchmark patterns - Document memory benchmark limitations due to cgo/wasmtime-go - Focus on wasmtime+wasmtime-go performance measurement Addresses feedback from @srenatus about benchmark scope and integration. Signed-off-by: Anivar A Aravind <[email protected]>
- Move benchmark file from topdown/ to v1/topdown/ - Update copyright year to 2025 - Modernize Rego syntax using 'if' keyword - Update Makefile bench-wasm target to run all benchmarks in v1/topdown and v1/rego Signed-off-by: Anivar <[email protected]> Signed-off-by: Anivar <[email protected]>
- Move benchmarks from v1/topdown to v1/rego to resolve import cycle - Add comprehensive WASM performance benchmarks as requested: * BenchmarkSimpleAuthzTargets: Authorization policy performance * BenchmarkBuiltinPerformanceTargets: Builtin function performance * BenchmarkDataSizesTargets: Performance with varying data sizes * BenchmarkPolicyComplexityTargets: Performance with varying policy complexity - Use ParsedModule for consistency with existing benchmarks - Update Makefile bench-wasm target to test both v1/topdown and v1/rego - Add memory benchmark warning about cgo/wasmtime-go limitations Addresses review feedback on PR open-policy-agent#7841 Signed-off-by: Anivar <[email protected]>
Added WASM vs topdown comparative benchmarks to track performance regressions mentioned in issue open-policy-agent#7759. The benchmarks provide visibility into compilation overhead, cold start penalties, and memory reporting. Key improvements: - Added copyright header (2025) - Removed context import, use b.Context() - Added isWASMNotAvailable() helper for cleaner error handling - Fixed unsafe type assertion with two-value form - Added b.Helper() to benchmark helper function - Pre-allocated slices with capacity - Used const for magic numbers Added 4 critical benchmarks: - BenchmarkWASMCompilationTargets: Compilation overhead metrics - BenchmarkWASMColdStartTargets: Cold start penalty analysis - BenchmarkMemoryAllocationTargets: Memory usage comparison with CGO warning - BenchmarkBundleSizeTargets: Policy size scaling impacts The memory benchmarks include a note about CGO allocation tracking limitations when using wasmtime-go. Fixes: open-policy-agent#7759 Signed-off-by: Anivar Mistry <[email protected]> Signed-off-by: Anivar A Aravind <[email protected]>
Signed-off-by: Anivar Mistry <[email protected]> Signed-off-by: Anivar A Aravind <[email protected]>
Signed-off-by: Stephan Renatus <[email protected]>
5c8ced3 to
5566ead
Compare
This PR introduces two new builtin functions for parsing and validating Package URLs (PURLs), which are commonly used in Software Bill of Materials (SBOMs) to identify software packages. New builtins: - purl.is_valid(string): Validates if a string is a valid PURL - purl.parse(string): Parses a PURL into its components (type, namespace, name, version, qualifiers, subpath) The implementation vendors the necessary parts of the packageurl-go library internally to avoid adding external dependencies. Only required fields (type, name) are returned as static properties, while optional fields are dynamically added when present, following OPA's pattern of omitting empty values. This addresses the need for SBOM validation in supply chain security policies, enabling OPA to parse and validate package identifiers from various ecosystems (npm, maven, docker, pypi, etc.) as requested in issue open-policy-agent#7841. Includes comprehensive documentation with SBOM policy examples and test coverage for various PURL types. Fixes open-policy-agent#7841 Signed-off-by: Anivar Aravind <[email protected]>
This PR introduces two new builtin functions for parsing and validating Package URLs (PURLs), which are commonly used in Software Bill of Materials (SBOMs) to identify software packages. New builtins: - purl.is_valid(string): Validates if a string is a valid PURL - purl.parse(string): Parses a PURL into its components (type, namespace, name, version, qualifiers, subpath) The implementation vendors the necessary parts of the packageurl-go library internally to avoid adding external dependencies. Only required fields (type, name) are returned as static properties, while optional fields are dynamically added when present, following OPA's pattern of omitting empty values. This addresses the need for SBOM validation in supply chain security policies, enabling OPA to parse and validate package identifiers from various ecosystems (npm, maven, docker, pypi, etc.) as requested in issue open-policy-agent#7841. Includes comprehensive documentation with SBOM policy examples and test coverage for various PURL types. Fixes open-policy-agent#7841 Signed-off-by: Anivar A Aravind <[email protected]>
This PR introduces two new builtin functions for parsing and validating Package URLs (PURLs), which are commonly used in Software Bill of Materials (SBOMs) to identify software packages. New builtins: - purl.is_valid(string): Validates if a string is a valid PURL - purl.parse(string): Parses a PURL into its components (type, namespace, name, version, qualifiers, subpath) The implementation vendors the necessary parts of the packageurl-go library internally to avoid adding external dependencies. Only required fields (type, name) are returned as static properties, while optional fields are dynamically added when present, following OPA's pattern of omitting empty values. This addresses the need for SBOM validation in supply chain security policies, enabling OPA to parse and validate package identifiers from various ecosystems (npm, maven, docker, pypi, etc.) as requested in issue open-policy-agent#7841. Includes comprehensive documentation with SBOM policy examples and test coverage for various PURL types. Fixes open-policy-agent#7841 Signed-off-by: Anivar A Aravind <[email protected]>
This PR introduces two new builtin functions for parsing and validating Package URLs (PURLs), which are commonly used in Software Bill of Materials (SBOMs) to identify software packages. New builtins: - purl.is_valid(string): Validates if a string is a valid PURL - purl.parse(string): Parses a PURL into its components (type, namespace, name, version, qualifiers, subpath) The implementation vendors the necessary parts of the packageurl-go library internally to avoid adding external dependencies. Only required fields (type, name) are returned as static properties, while optional fields are dynamically added when present, following OPA's pattern of omitting empty values. This addresses the need for SBOM validation in supply chain security policies, enabling OPA to parse and validate package identifiers from various ecosystems (npm, maven, docker, pypi, etc.) as requested in issue open-policy-agent#7841. Includes comprehensive documentation with SBOM policy examples and test coverage for various PURL types. Fixes open-policy-agent#7841 Signed-off-by: Anivar A Aravind <[email protected]>
This PR introduces two new builtin functions for parsing and validating Package URLs (PURLs), which are commonly used in Software Bill of Materials (SBOMs) to identify software packages. New builtins: - purl.is_valid(string): Validates if a string is a valid PURL - purl.parse(string): Parses a PURL into its components (type, namespace, name, version, qualifiers, subpath) The implementation vendors the necessary parts of the packageurl-go library internally to avoid adding external dependencies. Only required fields (type, name) are returned as static properties, while optional fields are dynamically added when present, following OPA's pattern of omitting empty values. This addresses the need for SBOM validation in supply chain security policies, enabling OPA to parse and validate package identifiers from various ecosystems (npm, maven, docker, pypi, etc.) as requested in issue open-policy-agent#7841. Includes comprehensive documentation with SBOM policy examples and test coverage for various PURL types. Fixes open-policy-agent#7841 Signed-off-by: Anivar A Aravind <[email protected]>
Why this matters
Issue #7759 highlighted that we have no visibility into WASM performance changes. As WASM becomes more critical for edge deployments and browser-based policy evaluation, we need to catch performance regressions before they impact users.
What this PR does
Adds lightweight benchmarks that compare WASM vs regular Rego performance, giving us:
The implementation
Just 220 lines added across 3 files:
topdown/wasm_bench_test.go- Standard Go benchmarks following OPA patternsMakefile- Simplemake bench-wasmtarget.github/workflows/pull-request.yaml- CI runs benchmarks automaticallyExample output
Now we can track WASM performance trends over time and ensure it remains viable for production use cases.
Testing
The benchmarks gracefully skip when WASM engine is not available, so they won't break existing workflows.
Impact
This gives OPA users confidence that WASM performance is being monitored and protected, which is crucial as more organizations adopt WASM for edge and browser deployments.
Fixes #7759