The System.Text.Json.SourceGeneration.Roslyn4.0.Tests project defines ~35 large JsonSerializerContext classes, resulting in approximately 3500 generated source files. The size of the project has had a detrimental effect on performance, both when building but especially when editing the project: on my primary dev machine the IDE essentially goes unresponsive when attempting to make any change in the project.

I ran a performance investigation and came up with the following findings:

1. There appear to be performance issues inherent in Roslyn ingesting such big numbers of generated source files. On my machine a build will max out all 20 cores of my CPU and consume 4-7 GB of memory. It can be isolated to making a large number of calls to the SourceProductionContext.AddSource method and is not related to the source generator itself.
2. Our incremental source generator implementation will regenerate every single source file on every keystroke. This is attributable to a combination of two factors:
   • Our incremental values are combining the entire Compilation object. This is explicitly called out as an anti-pattern in the incremental source generation doc, since any change will force re-evaluation of the source generator.
   • We are calling Collect() on the ClassDeclarationSyntax values provider, feeding every single JsonSerializerContext declaration to the source generator regardless of whether it's a newly updated or cached value.
3. The source generator is not honoring the CancellationToken passed by Roslyn. Fixed in Make json source generation cancellable #69332

I'm not sure if 1) could be improved, however it definitely seems like 2) is a performance bug in our source generator implementation that we should try to fix. Not sure how it could be done without some refactoring of the Parser class.

cc @layomia @ericstj @eerhardt