Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
17607b3 to
506bd73
Compare
e3d0507 to
e9640be
Compare
|
Amazing work! Going to look at this on monday (or tomorow if i have time) |
f989ce1 to
110e14f
Compare
camc314
left a comment
There was a problem hiding this comment.
We should also add tests for the following scenarios:
- supressions with cutom plugins
- supression with an invalid (malformed) supression file
- combining
--fixwith--suppress--allto make sure that we end up with the correct supression count - Not sure how we can test this, but have you considered/thought about how we're going to handle paths on different machines? e.g. windows/Mac path separator differences
| self.linter.options.suppress_all, | ||
| self.linter.options.prune_suppressions, | ||
| ) | ||
| .unwrap(); |
There was a problem hiding this comment.
I don't think it's safe to unwrap here in the case that the file is invalid - we probably want to pass this as an option to runtime::new or linter::new, and handle the failure gracefully with an error message
There was a problem hiding this comment.
What do you think of this approach:
apps/oxlint: Try to load the file using thesuppression_manager. If nothing goes wrong, a Some or None is returned. If something goes wrong, then an error is generated, but the lint works as expected. This logic will replace https://github.com/oxc-project/oxc/pull/19328/changes#diff-fd61a8cebacde1fbb072652808cc144fb381a435c98434693779485de0bdf8a2R418-R434.- The data needed by the lint is drilled through the argumens from here.
apps/oxlint->LintRunner::lint_files->LintService::run->Runtime::run - The OutputFormatter based on the state the
suppression_managerhas, will output the current messages, addedOxlintSuppressionFileAction::MalformedFilestatus.
There was a problem hiding this comment.
The other approach that comes to my mind after send the previous comments is the following. In LinterRunner::lint_files try to read the file, if it's malformed then lint the files, and return an Err("oxlint-suppression.json is malformed.") instead of Ok(self)
crates/oxc_linter/src/lib.rs
Outdated
| allocator: &'a Allocator, | ||
| ) -> Vec<Message> { | ||
| self.run_with_disable_directives(path, context_sub_hosts, allocator, None).0 | ||
| //TODO Inside LintRunner::run, we should check self.options.suppression_options, to handle each setting correctly |
There was a problem hiding this comment.
| //TODO Inside LintRunner::run, we should check self.options.suppression_options, to handle each setting correctly | |
| // TODO Inside LintRunner::run, we should check self.options.suppression_options, to handle each setting correctly | |
Do we need to action this before this PR is merged?
What is the correct handling here?
There was a problem hiding this comment.
This is a legacy comment that I forget to remove. This was an early assumption that was partially incorrect.
crates/oxc_linter/src/fixer/mod.rs
Outdated
|
|
||
| #[derive(Debug, Clone, Eq, PartialEq)] | ||
| pub struct Message { | ||
| pub suppression_id: Option<SuppressionId>, |
There was a problem hiding this comment.
isn't this information available inside error (oxc_diagnostic) ? We should have the error scope/error code to use?
There was a problem hiding this comment.
This shuold help us to avoid some excess allocations cloning rule name/rule id.
There was a problem hiding this comment.
The main regressions in performance appears to be as a result of these extra allocations
There was a problem hiding this comment.
I have updated the code to use OxcCode from the message, and none of the test are broken.
There was a problem hiding this comment.
Confirmed. No more performance regression detected
| context_sub_hosts, | ||
| allocator_guard, | ||
| me.js_allocator_pool(), | ||
| &suppression_file, |
There was a problem hiding this comment.
it might be worth making this an Option so that we don't need to create the supression file structure in the commen case where it isn't being used?
Correct me If I'm wrong, but I think this test case is covered here https://github.com/oxc-project/oxc/pull/19328/changes#diff-d6f62b74c4fcbe284a4c8b21f90cdf6200f2d41ecde269fcfeefd0a4d3c71174
Noted it.
My idea was to normalize the path as POSIX format, and reading the code I think I missed it this part. We don't need a full solution to handle the differences between Windows and Linux and Mac, our routes are relatives to
This afternoon I will have a computer with Windows to be able to manually test it. If you see it's good enough @camc314, I can convert the PR in draft. Once I have all the changes and new tests, I will mark it as ready and ping you? Also this week I'm a bit busy at work, and in the weekend I'm out. |
|
I will set the PR to draft.
This test is failing because I'm filtering the messages with fix, no matter if the fix option is turned on. Probably I will add test for
I will install a VM, because the manual test is taking forever in the computer they have lend me.
I have applied all PR comments. Once the previous points are solved, I will refactor the code again, mostly to improve readibility. |
d155b8b to
4dfc17f
Compare
Two targeted changes to the bulk-suppression machinery introduced in oxc-project#19328 (oxlint-suppressions.json support). **Change 1 — Eliminate the startup clone in `concurrent_map()`** `SuppressionManager::concurrent_map()` previously constructed a `papaya::HashMap` by iterating the suppression file and cloning every key and value into it. Workers only ever *read* one entry per file, so the copy was pure overhead: ~10–15 ms sequential startup cost for 10k files / 20k suppressions. Fix: wrap `AllSuppressionsMap` in `Arc` at load time. `concurrent_map()` now calls `Arc::clone()` (a reference-count bump), making the hand-off to rayon workers zero-copy. Side effect: the `papaya` dependency can be removed from the worker path in `service/runtime.rs`; `papaya::HashMap::pin()` is replaced by a plain `FxHashMap::get`. The `&SuppressionFile::new(…)` dangling- reference pattern is also fixed while touching that code. **Change 2 — Reduce per-message `RuleName` allocations** `build_suppression_map` called `message.into()` twice per message (once for `get_mut`, once for `insert`). Replaced with `.entry().or_insert()` so the key is computed once. The filter closure in `suppress_lint_diagnostics` also called `message.into()` twice per message (two separate `.get()` calls). Replaced with `let key = RuleName::from(message)` so both lookups share one allocation. Net result: 1–2 heap allocations per message instead of 3–4. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
d3730ac to
1488445
Compare
- All JSPlugins errors are being recorded in the file. Left Typescript as it required more changes. - Add JSPlugin test - Manual tested the binary in Posthog and oxlint-react-compiler-rules - Improved errors messages, and added a label to suggest the user what command run to update the file. - Some basic message in the cli output if the file is created or updated.
- Remove SuppressionId, this information already exist. Probably the reason of the performance regression. - SuppressionManager inited `apps/oxlint`. This replace the logic if the file has been created or updated. Pending the state if a diff haven’t been found, this will avoid render the file has been updated when it isn’t. - Integration test for malformed json. The error don’t panic, the files are linted but a message is rendered with the error. - `OxcDiagnositc` is transformed to `RuleName`. Pending to use TryFrom instead of From, as it can fail. - Removed misleading TODO comment
- `—fix —suppress-all` test without file not generate a `oxlint-suppressions.json` file and fixes are apply. - `—fix —-suppress-all`test having a initial suppression file, is not behaving correctly as the fix aren’t apply. Are filtered due the suppresion file. Pending to be fixed. - Using Option instead on initiating a default SuppressionFile struct. This solve an edge case of emitting errors when the file was malformed. Pending add more test with a combination of fix commands, a test that prune if only `fix` is provided.
…ify file update if the file has been really updated - Only mentioine the file has been update once the file is write instead of detecting if it exists. - Filtering the messages with the suppression only after of this fix has been apply, not before. - Pending more integration tests handling different of fix
… avoid noise and duplicate code in multiples tests
SuppressionTester struct to handle all the logic of files in the integration tests. - Implemented Drop trait that once the test go out of scope, all the files modified are deleted or replace by their backups. - Test logic hide behind the method `test`, this will do all the need checks to ensure everything works as expected. - Builder patter. to configure if the fixture have an specific configuration of files.
Picked manually the commit cf31c84 oxc-project@cf31c84 by @wagenet. As the papaya wrapper is not necessary at all. Removed the warning import errors and printlns
If an Message isn’t scoped to a plugin and or rule name then is discarded to be tracked.
- Colocated the logic of handle the diff channel in the struct. - A MSCP in the load method and drill proped the sender. This will allow to integrate typescript suppression rules. - No unwrap trying to write the oxlint-suppression file, pending add a message of error.
- Snapshot don’t render anymore the full route only the path relative to CWD. - Asserting parsing the JSON insterad of relying on string to avoid whitespace mismatch.
Now oxlint is filtering, updating or creating a file based on the tsgo diagnostics. An fixture is available to test different scenarios. At this point: - Pruned tsgo diagnostic after it’s fixed witn `—fix`� or `—fix-suggestions` - The diff is a bit more smarter and don’t prune TSGO rules. The other way is pending to be implement. - Diagnostics aren’t send if we are on suppression mod. Otherwise, the errors are immediately reported. Next step is update the fixtures to have TS, and add more specific TS tests. Once it’s done, refactor and rearchitecture a bit the code to reduce code duplicity, hide implementations and normalize to work always with `Message`
All fixtures are executed with type aware and type check to ensure the feature works without issues. Fixed a bug that only occurrs when an user executed oxlint as following `oxlint —fix —suppress-all` AND a suppression file doesn’t exists. Oxlint was unable to create a file because it thought we were in prune mode, instead of suppression after fixing all possible linting errors.
- Added a new SuprresionDiff Struct that abstract all the diff logic, to avoid duplication code and centralized the code. - Manager contains the list of rules linted by TSGo to avoid incorrect pruning rules logic between oxlint and TSGO - The diff handles Message as an input and output. - New test that checks type errors are never recorded. - Remove an unwrap logic inside the update, and hardening the check to avoid remove the last rule in a file in case the diff message doesn’t match with the last rule.
…e semantic name to the test and folders
188de40 to
a58c1f9
Compare
AI Disclosure
I have used AI to:
Apart from these situations, the code has been written without AI assistance. As mentioned in my first PR in Oxlint, these contributions help improve Oxlint while at the same time helping me improve my Rust skills.
Description
This PR implements
bulk-suppressionsfor Oxlint. I had in mind the goals mentioned in #10549 (comment)Implementation details
It is split into sections with all the decisions that I have taken to implement this PR.
Suppression Manager
The code is located in
crates/oxc_linter/src/suppression/mod.rsand it is split into three files:tracking.rs: Contains all the code that handles the loading, saving, and updating ofoxlint-suppressions.json.mod.rs: Implements theSuppressionManager. It also acts as a middleman betweenoxlintandtsgolintandtrackingto update the information, anddiffto execute the diff operations.diff.rs: Hides all the complexity of how the diff works from the rest of the project.Filtering Diagnostics
The Suppression Manager only filters diagnostics, that have a valid
OxcCode, otherwise the message is ignored as we can't scope it to a rule or plugin.The messages are only filtered if the new count is lower or equal than the registered one; otherwise, they are displayed. Same behavior as ESLint.
Sharing and updating the data
Oxlint uses
rayon::scopeto create as many workers as there are available threads. Those workers do the heavy linting work, and this is the data that I need for filtering or creating the file.The semantics of
Send + Syncin Rust make it hard, for good reason, to share information between threads. So the quick approach is for each worker to read the whole file just to pick a subset of the data. On top of that, the feature is probably more oriented toward medium/large projects, so this approach doesn't scale.So the implemented approach is the following:
SuppressionManagerduplicates the data into aUpdate: The data is wrapped in aconcurrent hash map. This map is safe to be shared between threads and is only used in read mode.Arcinstead of duplicating it. It reduces a level of indirection. Thanks @wagenet for the suggestion.DiffManager, this struct contains all data necessary for his operations.DiffManagercalculate a diff based on the args provided and ifbulk-suppressionsfile exists:OxcDiagnosticto be reported.Once the linting is completed and we are back in the main thread, we consume the channel with the diffs sent by the workers. If we are in
updateorcreatemode and no diffs have been emitted, then we do nothing. Otherwise, we mutate the map in theSuppressionManagerwith the operations sent by the workers and save it.In a few words, I'm trading memory usage for what is, in my opinion, a simpler process. The option that I discarded was relying on
interior mutability, but that would mean ensuring that all operations are safe and that no worker panics.TSGo lint errors are the only recorded errors.
Rather than completely changing how fixable and non-fixable errors are emitted, I have attached to the existing flow. Type errors and related errors are ignored and not recorded.
CLI Args
Currently, both commands update the file, no matter the diff found. The only situation that doesn't work the same is when the file
oxlint-suppressions.jsondoes not exist. The only way to create it is by runningoxlint --suppress-all.Comments
suppressionfolder.UPDATE: The fix commands are handle.--fixor related commands do not update the file.SuppressionTestthat implementsDropto handle the cleanup of this tests.PostHog

Oxlint React Compiler Rules

Fixes