benchmarks: remove SymDB benchmarks #5795
Conversation
The three SymDB benchmarks emit nested-dict result JSONs (extraction
stats, baseline-matrix RPS tiers, background-impact pre/treatment),
not the flat [{item, samples}] shape the benchmark-analyzer expects.
The converter crashes with TypeError before any candidate-vs-baseline
comparison happens, taking the entire microbenchmarks: [other] job
with it (e.g. dd-gitlab/microbenchmarks: [other] on PR #5784).
Removing them from the manifest disables CI execution but leaves the
benchmark scripts in place — they remain runnable locally for ad-hoc
measurement and are still validated by
spec/datadog/symbol_database/validate_benchmarks_spec.rb.
A proper re-enablement would reshape the scripts' JSON output to
[{item, samples}] so each measurement (e.g. rps_drop_at_99,
extraction_wall_seconds_mean, memory_overhead_kb) becomes its own
item compared per-rep across runs by the stock analyzer.
p-datadog
added
the
AI Generated
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: f46e6dc | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-05-20 21:00:09 Comparing candidate commit f46e6dc in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.
|
The script's expect_in_fork validation in validate_benchmarks_spec.rb times out on Ruby 3.0 (10s wait exhausted) — see Ruby 3.0 / build & test (standard) [4] on PR #5795. The benchmark's output JSON is also incompatible with the stock analyzer (the original reason for the disable in #5795), so the script has no current consumer. Drop it along with its entry in validate_benchmarks_spec.rb. - Deleted benchmarks/symbol_database_baseline_matrix.rb - Removed entry from spec/datadog/symbol_database/validate_benchmarks_spec.rb (the Dir glob in the all-benchmarks guard test now returns 2 files, matching the 2 remaining entries in benchmarks_to_validate)
Drop the last two SymDB benchmark scripts and their validate spec.
The scripts emit nested-dict result JSONs incompatible with the stock
benchmark-analyzer; they have no consumer until they are rewritten to
emit benchmark-ips-shaped output ([{item, samples}]). With all three
scripts gone, validate_benchmarks_spec.rb has nothing to validate.
- Deleted benchmarks/symbol_database_extraction.rb
- Deleted benchmarks/symbol_database_background_impact.rb
- Deleted spec/datadog/symbol_database/validate_benchmarks_spec.rb
- Removed the symbol_database_ entry from benchmarks/README.md
p-datadog
changed the title
This reverts commit 50fc341.
2 participants
What does this PR do?
Removes SymDB benchmarks because they are broken.
Motivation:
Master is red:
The three scripts (
symbol_database_extraction,symbol_database_background_impact,symbol_database_baseline_matrix) emit custom nested-dict result JSONs — extraction stats, RPS-drop curves by load tier, baseline vs treatment ops/sec. The stock benchmark-analyzer's converter expects the[{item: "...", samples: [<numbers>]}]shape and crashes with:That crash happens after both candidate and baseline produce their JSONs, but before any comparison — so the entire
microbenchmarks: [other]job goes red and nothing else in that batch (tracing, error_tracking, di_instrument, library_gem_loading) gets a comparison report either. PR #5784 (job 1699825413) is the most recent example.Change log entry
None.
Additional Notes:
Re-enabling these benchmarks properly is a follow-up: reshape each script's output so each tracked measurement becomes its own
itemwith per-repsamples, e.g.[ {"item": "symdb - rps_drop @ 99% load", "samples": [0.012, 0.011, 0.013, 0.012]}, {"item": "symdb - extraction_wall_seconds_mean", "samples": [0.75, 0.74, 0.75, 0.76]}, {"item": "symdb - memory_overhead_kb", "samples": [41088, 41200, 40996, 41152]} ]Once the shape matches, the stock analyzer handles the comparison and PRs like #5784 (which removes the throttle as a regression-detection check) would surface RPS drops at high load as a normal significance report.
How to test the change?
benchmarks/execution.ymlvalidate_benchmarks_spec.rbstill passes (it loads each script; it does not depend on the script being listed inexecution.yml)dd-gitlab/microbenchmarks: [other]run: candidate/baseline produce JSONs only for the four remaining benchmarks; analyzer completes without crashing