You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer TIP This summary will be updated as you push new changes.
This is an automated daily PR that refreshes mock benchmark documentation by updating timestamps (2026-06-25 → 2026-06-26), regenerating benchmark numbers in .md files, and syncing the source .json data files. No application logic is touched; all 15 changed files are in docs/.
Finding 1 — Schema inconsistency: "Median" field appears only in Setup category
The new Setup.json adds a "Median" field to every result entry (e.g., "Median": "504.8 ns"), but no other category JSON (Callback.json, CombinedWorkflow.json, Invocation.json, MockCreation.json, Verification.json) includes this field. The latest.json faithfully mirrors this inconsistency — Setup entries have Median, everything else doesn't.
Why this matters: If any tooling (the Docusaurus site, a CI comparison script, or a future consumer) iterates over benchmark results and expects a uniform schema per category, it will either silently ignore the median for non-Setup categories or crash when it's missing. This also makes the published docs misleading — readers can't compare median performance across categories.
Suggested fix: Either emit Median for all categories when it's available in BenchmarkDotNet output, or strip it from Setup so the schema stays uniform. A schema validation step in the workflow that generates this PR would catch this automatically on future runs.
Finding 2 — "Gen2" field removed inconsistently across files
In the Callback category, this PR removes "Gen2": "-" from all result entries. MockCreation still carries the field. The result is that two JSON files with the same stated schema now differ structurally — some objects have two GC-generation fields, others have three.
Why this matters: Same risk as above: tooling or display logic that iterates over GC fields will behave differently per category without any indication this is intentional. If Gen2 collections are always - for Callback, the cleaner fix is to emit it consistently (or omit it consistently) rather than letting BenchmarkDotNet's run-to-run output variation silently produce different schemas.
The CombinedWorkflow benchmark shows very large day-over-day deltas for the proxy-based libraries:
Moq: 400.863 μs → 124.317 μs (−69%)
NSubstitute: 17.322 μs → 8.903 μs (−49%)
FakeItEasy: 18.466 μs → 9.180 μs (−50%)
These improvements are far outside normal benchmark noise (< 5%) and suggest the prior day's run had significant environmental interference (JIT warm-up, shared CI runner load, etc.) rather than genuine performance changes. The resulting Mermaid chart y-axis shifted from 0 --> 482 to 0 --> 150, which would visually mislead a reader comparing two days' charts.
This isn't a bug in the PR itself — it faithfully reproduces the CI output — but it's a signal that the benchmark automation workflow would benefit from a stability gate (e.g., reject or flag results where a library's mean changes by > 20% vs. the rolling 7-day average) before auto-committing. Without that, the daily-updated docs can quietly reflect noisy runs as authoritative results.
Overall the automated pipeline is working correctly and the files are internally consistent within each run. The schema divergence between categories (Median/Gen2) is the most actionable item; the rest is a suggestion for improving benchmark reliability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Mock Benchmark Update
This PR updates the mock benchmark documentation with the latest results from the Mock Benchmarks workflow.
Benchmarks Produced
Individual benchmark artifacts are available for download:
mock-benchmark-MockCreationmock-benchmark-Setupmock-benchmark-Invocationmock-benchmark-Verificationmock-benchmark-Callbackmock-benchmark-CombinedWorkflowmock-benchmark-summary(aggregated overview)Libraries Compared
Libraries are discovered dynamically from the benchmark output. See the generated
summary.jsonfor the full list.Changes
docs/static/benchmarks/mocks/latest.jsondocs/docs/benchmarks/mocks/docs/static/benchmarks/mocks/summary.jsonWorkflow Run
🤖 This PR was automatically created and will be merged automatically once CI checks pass.