Skip to content

🤖 Update Mock Benchmark Results#6316

Merged
thomhurst merged 1 commit into
mainfrom
automated-mock-benchmarks-update
Jun 26, 2026
Merged

🤖 Update Mock Benchmark Results#6316
thomhurst merged 1 commit into
mainfrom
automated-mock-benchmarks-update

Conversation

@thomhurst

Copy link
Copy Markdown
Owner

Automated Mock Benchmark Update

This PR updates the mock benchmark documentation with the latest results from the Mock Benchmarks workflow.

Benchmarks Produced

Individual benchmark artifacts are available for download:

  • mock-benchmark-MockCreation
  • mock-benchmark-Setup
  • mock-benchmark-Invocation
  • mock-benchmark-Verification
  • mock-benchmark-Callback
  • mock-benchmark-CombinedWorkflow
  • mock-benchmark-summary (aggregated overview)

Libraries Compared

Libraries are discovered dynamically from the benchmark output. See the generated summary.json for the full list.

Changes

  • Updated mock benchmark data in docs/static/benchmarks/mocks/latest.json
  • Regenerated mock benchmark documentation in docs/docs/benchmarks/mocks/
  • Updated mock benchmark summary in docs/static/benchmarks/mocks/summary.json

Workflow Run

  • Run ID: 28214783432
  • Triggered: schedule

🤖 This PR was automatically created and will be merged automatically once CI checks pass.

@thomhurst thomhurst added automated benchmarks documentation Improvements or additions to documentation ignore-for-release labels Jun 26, 2026
@thomhurst
thomhurst merged commit 0564258 into main Jun 26, 2026
11 of 13 checks passed
@thomhurst
thomhurst deleted the automated-mock-benchmarks-update branch June 26, 2026 03:29
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Code Review — PR #6316: Update Mock Benchmark Results

This is an automated daily PR that refreshes mock benchmark documentation by updating timestamps (2026-06-25 → 2026-06-26), regenerating benchmark numbers in .md files, and syncing the source .json data files. No application logic is touched; all 15 changed files are in docs/.


Finding 1 — Schema inconsistency: "Median" field appears only in Setup category

Files: docs/static/benchmarks/mocks/Setup.json, docs/static/benchmarks/mocks/latest.json (Setup section)

The new Setup.json adds a "Median" field to every result entry (e.g., "Median": "504.8 ns"), but no other category JSON (Callback.json, CombinedWorkflow.json, Invocation.json, MockCreation.json, Verification.json) includes this field. The latest.json faithfully mirrors this inconsistency — Setup entries have Median, everything else doesn't.

Why this matters: If any tooling (the Docusaurus site, a CI comparison script, or a future consumer) iterates over benchmark results and expects a uniform schema per category, it will either silently ignore the median for non-Setup categories or crash when it's missing. This also makes the published docs misleading — readers can't compare median performance across categories.

Suggested fix: Either emit Median for all categories when it's available in BenchmarkDotNet output, or strip it from Setup so the schema stays uniform. A schema validation step in the workflow that generates this PR would catch this automatically on future runs.


Finding 2 — "Gen2" field removed inconsistently across files

Files: docs/static/benchmarks/mocks/Callback.json (and latest.json Callback section) vs. MockCreation.json (which retains "Gen2": "-")

In the Callback category, this PR removes "Gen2": "-" from all result entries. MockCreation still carries the field. The result is that two JSON files with the same stated schema now differ structurally — some objects have two GC-generation fields, others have three.

Why this matters: Same risk as above: tooling or display logic that iterates over GC fields will behave differently per category without any indication this is intentional. If Gen2 collections are always - for Callback, the cleaner fix is to emit it consistently (or omit it consistently) rather than letting BenchmarkDotNet's run-to-run output variation silently produce different schemas.


Observation — Dramatic CombinedWorkflow performance swings

The CombinedWorkflow benchmark shows very large day-over-day deltas for the proxy-based libraries:

  • Moq: 400.863 μs → 124.317 μs (−69%)
  • NSubstitute: 17.322 μs → 8.903 μs (−49%)
  • FakeItEasy: 18.466 μs → 9.180 μs (−50%)

These improvements are far outside normal benchmark noise (< 5%) and suggest the prior day's run had significant environmental interference (JIT warm-up, shared CI runner load, etc.) rather than genuine performance changes. The resulting Mermaid chart y-axis shifted from 0 --> 482 to 0 --> 150, which would visually mislead a reader comparing two days' charts.

This isn't a bug in the PR itself — it faithfully reproduces the CI output — but it's a signal that the benchmark automation workflow would benefit from a stability gate (e.g., reject or flag results where a library's mean changes by > 20% vs. the rolling 7-day average) before auto-committing. Without that, the daily-updated docs can quietly reflect noisy runs as authoritative results.


Overall the automated pipeline is working correctly and the files are internally consistent within each run. The schema divergence between categories (Median/Gen2) is the most actionable item; the rest is a suggestion for improving benchmark reliability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated benchmarks documentation Improvements or additions to documentation ignore-for-release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant