Repository navigation
Include System.Linq.Async from https://github.com/dotnet/reactive into corefx #31580
Description
I am very happy we have the new IAsyncEnumerable and some c# sugar to go with it in .net core 3.
I think it would be even better if it had batteries included just like IEnumerable.
👍
Reacted by Andrii Kurdiumov, dczuo, Max Katz, Cory Nelson, Steve, Sam, James Turner and Nathan Povo
Activity
stephentoub commentedon Feb 1, 2020
Thanks. We considered this previously, and currently have no plans to ship this in the platform. It is readily available for anyone who wants to use it simply by adding a nuget package reference to it.
cc: @MadsTorgersen, @jaredpar
MgSam commentedon May 30, 2020
@stephentoub What is the reasoning behind this? You guys put a ton of engineering into making
IAsyncEnumerablework. Why ship it in a crippled state without basic LINQ support from the core framework?stephentoub commentedon May 30, 2020
How is it crippled? Add a one-line package reference to your .csproj, and you're off and running. Why is that insufficient?
MgSam commentedon May 30, 2020
Any package which you guys release separately only a tiny fraction of the overall .NET community ever knows about. And many teams try to limit adding additional dependencies in as much as possible. Finally, if the history of RX is any guide these separate packages often have much lower quality standards and documentation than what is included in the BCL.
The trend recently seems like it's been to bundle more to avoid the problems from separate packages. I don't need a separate package to use Span. Why is this feature being treated differently?
stephentoub commentedon May 30, 2020
Span is a fundamental exchange type that is now plumbed through all levels of the system, enabling the next wave of performance enhancements in and on the platform. Core primitives like int, double, and string all have dependencies on it. It's integrated with the runtime itself. It needs to be part of the platform for that to work. The same cannot be said of extension methods that operate on IAsyncEnumerable.
We collectively need to get past this for the ecosystem to be successful. It does not scale to include "in the box" all possible functionality developers may want.
jkotas commentedon May 30, 2020
Improving health of .NET ecosystem is part of the .NET Foundation charter. Check out https://github.com/dotnet-foundation/projects .
MgSam commentedon May 30, 2020
And
IAsyncEnumerablewas deemed important enough to build into the language itself; an extremely high bar for any feature. There were already excellent 3rd party libraries that provided equivalent functionality without new language features. Yet a small library that is helpful to work withIAsyncEnumerableeffectively is not important enough to be in the framework?This is a noble statement, but I don't think a feature which is pretty essential to what you guys just designed and shipped qualifies as "all possible" functionality.
You didn't actually address any of the 3 issues I pointed out:
Most developers do not know about packages you ship separately. I've been closely following this community since it went open source and I myself am barely aware of the existence of a package of extension methods for IAsyncEnumerable. For a normal C# dev who doesn't follow the community at all, there is no chance they'll be aware of it.
Many of your biggest users are in locked-down enterprises or locked-down teams where what they can use is what's included in the frameworks, and that's it. This ecosystem isn't Python or JavaScript and never will be Python or JavaScript. And this point of view is actually pretty rational- when you allow individual developers to add whatever packages they feel like it quickly becomes a maintenance nightmare because no one knows how anyone else's code actually works.
Rx .NET is a low-quality library not up to the standard of anything in the BCL and this makes it difficult to use. It has every API and the kitchen sink thrown into it with little curation. It has horrible documentation- that which does exist is largely out-of-date 3rd party docs from the many prior versions (with many breaking changes in-between) strewn about the web.
Finally, with you guys shipping the ability to trim unused assemblies from an app that bundles the framework, does saving every byte from going into the BCL really matter? I can see not adding shader processing packages into the BCL; but core methods to use language features with collection types? That seems pretty extreme to leave out.
stephentoub commentedon May 30, 2020
For consuming (foreach) and producing (iterators) IAsyncEnumerable... and the corresponding interfaces and key helpers that enable that were added to the framework. The hundreds upon hundreds of extension methods that make up an async LINQ implementation were not: the existing async LINQ library met all the key needs for it, and we did not need to design/implement/test/document/ship/service/maintain/evolve a competing implementation that would actually detract from rather than enhance the ecosystem. We instead chose to work with the maintainers of that library to ensure it was properly designed and interoperable, a library we could and do recommend as the solution developers employ if they need an async LINQ implementation.
stephentoub commentedon May 30, 2020
cc: @clairernovotny for both .NET Foundation and async LINQ
clairernovotny commentedon May 30, 2020
cc @bartdesmet as one of the main Rx authors.
I'm sorry to hear that you feel that Rx's quality is low. What is it that brings you to that result?
We strive to have high test coverage, and we use code coverage tools to help guide us to this. While we don't have 100% coverage, that's not a realistic goal for any library, but what we do have is very high.
It is true that the docs are severely lacking (mostly non-existent). Rx has been a volunteer effort and there has been no budget to write robust docs. Docs are hard and expensive to write and we'd take any contributions for that.
Turnerj commentedon May 31, 2020
While I personally think
System.Linq.Asyncwould be better inside the runtime, I have a different take onto why it should be there than @MgSam.My problem isn't with having an additional dependency, its that the discoverability of this isn't obvious when non-async LINQ is available without an additional dependency (ie. is someone going to know that async LINQ is a NuGet package). It could be argued that not many people would need to use it compared to normal/sync LINQ so it doesn't need to belong in the runtime and I'd believe that at present. The problem here though is that more and more code is being encouraged to be async, especially ASP.NET Core applications and the various controllers/action methods we write. Not including async LINQ in the runtime feels like there are mixed messages where we are encouraged to be async but are only given half of the tools out of the box to do so.
In terms of having a healthy .NET ecosystem, I feel like there is already a fairly big case of where this has happened -
Newtonsoft.JsonandSystem.Text.Json. The former was effectively the defacto standard for JSON in .NET with probably every .NET library needing to do anything with JSON using it. WithSystem.Text.Json, that's a non-trivial addition to the runtime for something that was widely adopted. I wouldn't be surprised if usage ofNewtonsoft.Jsondrops dramatically in the near future (I'm moving my own libraries and projects toSystem.Text.Json, I imagine many others are too). I don't know all the reasons forSystem.Text.Jsonnor do I even think the decision was necessarily bad as if every project is basically including the same library, why have it as a library and not the runtime. This though is where I think async LINQ might be in the not-terribly-distant future - a library that may be effectively added to every project to do the same thing.I do totally understand the point of being hesitant for doing this as the runtime already has a huge API surface with probably tens of thousands of tests and any less code added would make maintaining it far easier. If integrating
System.Linq.Asyncwas a possibility, I could totally see maintainability as a roadblock for it.I think my feelings for this can be summed up as: async LINQ feels like a second class citizen compared to (normal) LINQ. If the future is async (which it seems to be), it makes sense to me that the runtime provides parity of the existing data query methods as async.
Or maybe I'm wrong with all of this - actual use of async LINQ won't grow because async operations are often relegated to file, socket or DB operations which either don't use
IAsyncEnumerableor provide their own custom ways of handling async LINQ (that's if we call a DB layer usingIQueryablewith a customToListAsyncextension method "async LINQ").oliverjanik commentedon Jun 6, 2020
Having officially MS sanctioned packages means a lot in the enterprise space. Every dependency has to be vetted and approved by security team. Every update to every dependency has to be approved as well.
.NET has always been "batteries included". Splitting ecosystem into thousands of competing packages is evolving .NET towards where node/npm is and that is a regression IMO.
Having put enormous enginneering effort into IAsyncEnumerable and having it ignored be cause it lacks basic convenience methods is a huge waste. It will go unnoticed and unused.
And what will happen is every library that uses or exposes IAsyncEnumerable with ship it's own implementation of some of these methods or it will ship a transitive dependency on the AsyncEnumerable nuget. Too many transitive nugets lead to versioning hell.
Please reconsider.
MgSam commentedon Jul 21, 2020
@stephentoub
Actions speak louder than words. Rx and Ix have been 3rd class citizens for years. For anyone who does the most basic Google search they'll find ancient documentation for Rx v1 much of which is wrong.
introtorx.com, one of the top results, was last updated in 2015.Every .NET Core release you write an enormous blog post detailing the performance and other enhancements going into .NET. When is the last time Rx or Ix was mentioned in that? You have someone from the team itself on this very thread saying they don't have the resources they need to produce a good quality library. Async Linq needs to be properly designed by Microsoft engineers and put in .NET proper or IAsyncEnumerable will remain a throw away feature few know about or use.
It's fine if you guys don't have the resources for this now, but it should at least discussed and triaged honestly instead of pretending that there is already a satisfactory solution available.
stephentoub commentedon Jan 23, 2025
#111685 has added System.Linq.AsyncEnumerable.