test(linter/plugins): remove non-deterministic error output in conformance tester#16723
Merged
graphite-app[bot] merged 1 commit intomainfrom Dec 11, 2025
Conversation
github-actions
bot
added
A-linter
This was referenced Dec 11, 2025
Member
Author
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Member
Author
Merge activity
|
…mance tester (#16723) Report snapshot for conformance test results contains the errors thrown during linting. In the case of `AssertionError`s generated by `assert`, they sometimes contain diff markers e.g.: ``` + 'missingClosingLinebreak' - 'missingOpeningLinebreak' ^ ``` However, these `^` only appear sometimes, and seem to do so non-deterministically. They appear and disappear randomly from run to run in some places. Looks like this is a bug in NodeJS. Remove all such lines from the report to avoid meaningless churn in the snapshot.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses non-deterministic test output in the conformance tester by removing caret (^) markers that appear randomly in AssertionError diffs. These markers are generated inconsistently by Node.js's assert module, causing unnecessary snapshot churn.
Key Changes:
- Added logic to filter out diff marker lines containing only
^in error formatting - Updated the conformance test snapshot to reflect the removal of these markers
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| apps/oxlint/conformance/src/report.ts | Added a conditional check to skip lines that contain only ^ (after trimming whitespace) when formatting error output |
| apps/oxlint/conformance/snapshot.md | Updated snapshot to remove all instances of non-deterministic ^ diff markers from AssertionError outputs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
graphite-app
bot
force-pushed
the
12-11-refactor_linter_plugins_reduce_code_nesting
branch
from
f509ef4 to
4d9e981
Compare
graphite-app
bot
force-pushed
the
12-10-test_linter_plugins_remove_non-deterministic_error_output_in_conformance_tester
branch
from
4beef87 to
ef1ef18
Compare
Base automatically changed from
12-11-refactor_linter_plugins_reduce_code_nesting
to
main
December 11, 2025 02:25
graphite-app
bot
deleted the
12-10-test_linter_plugins_remove_non-deterministic_error_output_in_conformance_tester
branch
This was referenced Dec 11, 2025
graphite-app bot
pushed a commit
that referenced
this pull request
Dec 11, 2025
Conformance tester has been suffering from non-deterministic output from NodeJS's `assert` module. Fix this by setting `process.env.FORCE_COLOR` before loading any code. This also allows leaving the ` ^` diff marker lines in output which were removed in #16723, as they should now appear consistently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Report snapshot for conformance test results contains the errors thrown during linting. In the case of
AssertionErrors generated byassert, they sometimes contain diff markers e.g.:However, these
^only appear sometimes, and seem to do so non-deterministically. They appear and disappear randomly from run to run in some places. Looks like this is a bug in NodeJS.Remove all such lines from the report to avoid meaningless churn in the snapshot.