I'd like to explore other options for fixing this logspam other than making this bash script more complex.

In fact, this patch doesn't solve the issue because flutter_test also has a dependency_overrides for analyzer and flutter_test is a package that end-developers depend on.

In fact, this patch doesn't solve the issue because flutter_test also has a dependency_overrides for analyzer and flutter_test is a package that end-developers depend on.

I didn't realise that dependency_overrides didn't apply across the board. This means our current solution here is a non-starter.

I think we need to find a different way to decouple our analyzer dependency. Is there some way for example that we can teach "pub" that the analyzer package in the SDK is one of the ones it should consider? (Maybe the only one it should consider?)

In fact, this patch doesn't solve the issue because flutter_test also has a dependency_overrides for analyzer and flutter_test is a package that end-developers depend on.

Right. In that case though the dependency is way looser. It won't matter if there's skew in practice and whatever version of analyzer a user has (that's compatible with test) will be fine. flutter analyze is obviously a different story...

I think we need to find a different way to decouple our analyzer dependency.

@danrubel has been thinking a bit about this...

Is there some way for example that we can teach "pub" that the analyzer package in the SDK is one of the ones it should consider? (Maybe the only one it should consider?)

Not aware of any way. FWIW a bigger hammer would be to change the verbosity level in the bash script to only show pub errors (e.g., --verbosity=error). That makes the script less fiddly but obviously doesn't address the larger concerns over coupling.

I think we should remove the analyzer dependency override on flutter_test because it doesn't do what it looks like it does.

I don't mind so much the dependency in flutter_tools, but I do think it's bad that it means we end up with two versions of that package being used by one vended flutter SDK depending on what you're doing.

Why can't we just patch pub to have a way to declare that here are the packages, overriding anything that the pub server says? That would let us in the future move to inlining other packages which would be really nice. For example we could have our own local copy of quiver (with a defined version number), and anyone who depends on it would automatically get the same exact local copy of quiver, and if they had a dependency on a version that was out of range it just wouldn't resolve.

Why can't we just patch pub to have a way to declare that here are the packages, overriding anything that the pub server says?

Just to be clear, it is not the pub server that is saying "here's where you should get analyzer from" it is other packages that depend on analyzer.

Let's say I maintain package awesome_stuff. It uses analyzer, with a dependency like so:

dependencies:
  analyzer: ^0.28.1

As the maintainer of awesome_stuff, I'm promising that my package works if you use one of those allowed versions of analyzer from pub.dartlang.org.

Now let's say Jane wants to write a Flutter app that uses awesome_stuff. If her Flutter app gives awesome_stuff some other version of analyzer that was bundled with Flutter, then I—the maintainer of awesome_stuff—and Jane have no way of knowing that it will actually work.

Jane's Flutter app is no longer meeting the contract that awesome_stuff has declared. But if it doesn't end up working, Jane is going to think awesome_stuff is broken. She'll file bugs and they might not even make sense to me because she's using some unknown copy of analyzer.

For this reason, every dependency has both a version constraint and a source, a description of where the package should be found. (The source is implicitly "pub.dartlang.org" if not listed in the dependency.) To meet that dependency, a package must be within the right version and come from the right source. That ensures that packages actually get the dependencies they expect.

Closing in favor of #5991

@munificent My proposal was to change the semantics of analyzer: ^0.28.1 from "my package works if you use one of those allowed versions of analyzer from pub.dartlang.org" to "my package works if you use one of those allowed versions of analyzer", and then have pub support being told "if you are about to look up the 'analyzer' package on pub, then instead, assume that this locally cached copy is the only available copy and that it's version is X".

Assuming that the cached version and the pub version with the same number are identical, the only effect here would be that if a package depends on a version that isn't compatible with the cached one, pub would fail to find a match. This is exactly the desired outcome.

Eventually this would allow us to do this for all the packages so that the Flutter SDK would pin the exact versions of each package that it's compatible with. (This would also mean that our builds would be faster and more reliable since we wouldn't depend on the pub server.)

Assuming that the cached version and the pub version with the same number are identical

Yes, it is this assumption that I'm not comfortable with package maintainers being forced to sign up for.

if a package depends on a version that isn't compatible with the cached one, pub would fail to find a match

You could also accomplish this by adding an additional dependency on that package constrained to the range of versions Flutter allows for it. Something along those lines might be a better way to express this.

Eventually this would allow us to do this for all the packages so that the Flutter SDK would pin the exact versions of each package that it's compatible with.

Do you intend to whitelist all packages in the world for use by Flutter, or just the ones that Flutter itself also depends on? For any given package, do you intend to whitelist multiple versions of it, or just one?

Is the idea that you have a universe of packages managed by Flutter and any regular dependency on one of those gets rerouted to pull from Flutter instead of pub.dartlang.org? Is that universe the only packages you want users to be able to see, or can they also reach out to pub.dartlang.org for other packages?

Yes, it is this assumption that I'm not comfortable with package maintainers being forced to sign up for.

It's the Flutter SDK maintainers who are signing up for it, not the package maintainers.

You could also accomplish this by adding an additional dependency on that package constrained to the range of versions Flutter allows for it.

The problem with that is that it means much more work for everyone. Every time we roll the Dart SDK, we'd have to update every pubspec.yaml we have that mentions the analyzer package.

Do you intend to whitelist all packages in the world for use by Flutter, or just the ones that Flutter itself also depends on? For any given package, do you intend to whitelist multiple versions of it, or just one?

We intend to eventually package the SDK and all its dependencies as one package.

Flutter apps would be able to depend on other things beyond that.

The problem is preventing people from using (maybe indirectly via other packages' dependencies) versions of packages used by Flutter that aren't the versions shipped with Flutter, since that would be a mighty source of confusion and bugs. Ideally without us having to pin every dependency explicitly in every pubspec.yaml, since that would be very tedious and error-prone to maintain.

It's the Flutter SDK maintainers who are signing up for it, not the package maintainers.

Say Bob maintains package awesome_stuff that depends on analyzer. Jane is writing a Flutter app that uses awesome_stuff. If on Jane's machine, pub gives awesome_stuff some version of analyzer that came from Flutter and which doesn't work with awesome_stuff, Jane is going to go file a bug on Bob's issue tracker because awesome_stuff is broken.

The Flutter folks can ensure that all of the packages it uses work together, but I am assuming you are not signing up to validate that every version of every third party package works with all of your vendored dependencies.

The problem is preventing people from using (maybe indirectly via other packages' dependencies) versions of packages used by Flutter that aren't the versions shipped with Flutter, since that would be a mighty source of confusion and bugs.

Right, if an application has two copies of conceptually the same package, Bad Stuff™ happens. This is why pub resolves shared dependencies to a single version and source for a package.

From what I can tell, though, this is a problem Flutter has created and is now trying to work around. If you didn't put user-accessible packages inside Flutter that also exist and are used from pub.dartlang.org, you wouldn't have this duplication.

If Flutter needs analyzer 1.2.3, and awesome_stuff needs analyzer 1.2.3, then if they both just got it from pub.dartlang.org, there would be no problem. I'm probably missing some pieces, but I don't see what you gain by having your own source of packages within Flutter itself that is separate from but masquerades as pub.dartlang.org.

From what I can tell, though, this is a problem Flutter has created and is now trying to work around.

Guilty.

Happy to chat (maybe best offline) about the rationale but in a nutshell this reaching into the analyzer package was done primarily for performance reasons (previous iterations of flutter analyze shelled out to the analyzer and were slooooowwwww). This worked fine for quite a while. I published fresh analyzer packages that corresponded with more than a dozen SDK rolls. For a variety of reasons this has gotten trickier and can't be sustained regardless as expecting the analyzer folks to publish a stable, strictly non-API-breaking analyzer weekly would kill their velocity.

Re-reading this I worry that I'm only just further muddying the waters. Feel free to ping me if you want to chat. I bet we can come to a better understanding over VC (or beer). 🍻

Beer makes everything better! (Well, except alcoholism, I guess. :( )

I don't think this is an issue specific to analyzer. I think it stems from wanting Flutter developers to be able to get some packages (of which analyzer is but one) from within the Flutter SDK, even though those packages are also available on pub.dartlang.org and that is where other packages depend on them from.

Say Bob maintains package awesome_stuff that depends on analyzer. Jane is writing a Flutter app that uses awesome_stuff. If on Jane's machine, pub gives awesome_stuff some version of analyzer that came from Flutter and which doesn't work with awesome_stuff, Jane is going to go file a bug on Bob's issue tracker because awesome_stuff is broken.

If awesome_stuff doesn't work with that version of analyzer, why does its pubspec.yaml say that it does? This is a bug with awesome_stuff. The same bug would manifest if you were using only pub but happened to have another package (in this case flutter) which pinned analyzer to the version that is incompatible (in practice, though not according to the pubspec.yaml) with awesome_stuff.

The Flutter folks can ensure that all of the packages it uses work together, but I am assuming you are not signing up to validate that every version of every third party package works with all of your vendored dependencies.

pub already handles this for us with its versioning logic.

Right, if an application has two copies of conceptually the same package, Bad Stuff™ happens. This is why pub resolves shared dependencies to a single version and source for a package.

That's what I'm proposing here too. I'm just proposing that flutter only claim to be compatible with a single version of each package it cares about, and that we precache these packages.

From what I can tell, though, this is a problem Flutter has created and is now trying to work around. If you didn't put user-accessible packages inside Flutter that also exist and are used from pub.dartlang.org, you wouldn't have this duplication.

I don't understand what you mean here. Flutter's packages are only in Flutter's repo. To use them you use the "sdk: flutter" source in your pubspec.yaml. This is unrelated to the problem being discussed here.

I don't see what you gain by having your own source of packages within Flutter itself that is separate from but masquerades as pub.dartlang.org.

We gain the following:

• We get the benefits of pinning a specific version, without having to continually maintain multiple pubspec.yaml files and make sure we're keeping them in sync. We have found that doing this is harder than maintaining local dependencies (we do both a lot today). The benefit of pinning a specific version is that we can guarantee compatibility -- today, we often find that when people ship new versions of their packages, Flutter breaks because their package isn't as backwards compatible as they think.
• We avoid any trouble with packages that are both in the Dart SDK and in pub (e.g. analyzer, which is what started this discussion).
• It's faster. No need to hit the network to resolve these packages. Right now this costs us a lot of time, for example minutes for each bot iteration.
• It's more reliable. The pub server sometimes returns 500s, this would become a non-issue. Right now we get at least one failure a day in the bots because of this.
• It allows us to have hermetic builds. Right now, it's impossible for us to reliably reproduce problems over time because the environment may have changed (e.g. a new version of a package might be on pub, and we don't know what version we needed to show the bug that was filed 6 months ago).
• We get all these benefits without having to change any packages on pub to use a new source, not requiring that packages decide if they're for flutter or not. People can use a package on pub that happens to use a package that we depend on and it all works transparently.

There's probably other benefits besides, but those are the ones that immediately come to mind.

If awesome_stuff doesn't work with that version of analyzer, why does its pubspec.yaml say that it does?

It doesn't. If awesome_stuff says:

dependencies:
  analyzer: 1.2.3

It's claiming to work with version 1.2.3 of analyzer from pub.dartlang.org. The above form is just a convenient notation that's equivalent to:

dependencies:
  analyzer:
    hosted:
      name: analyzer
    version: 1.2.3

The source—where the package comes from—is considered a key part of its identity. Pub doesn't consider analyzer-1.2.3 from pub.dartlang.org to be the same package as analyzer-1.2.3 from some random Git repository or analyzer-1.2.3 from some random local file path, because either of those latter ones may have all sorts of differences from the canonical one on pub.dartlang.org.

It can't guarantee that the contents of those packages are the same, so it pessimistically assumes they may not be.

The only way it's safe to assume that Flutter's copy of analyzer-1.2.3 is a reliable substitute for the analyzer-1.2.3 on pub.dartlang.org that awesome_stuff declared it depends on and was tested against is if Flutter's analyzer-1.2.3 is byte-for-byte identical with the one on pub.dartlang.org.

But, if it is, that also means there's not much value in storing the package inside Flutter at that point. You may as well just download it from pub.dartlang.org.

I don't understand what you mean here. Flutter's packages are only in Flutter's repo. To use them you use the "sdk: flutter" source in your pubspec.yaml.

Yes, sorry I wasn't clear. My "user-accessible packages", I meant things like "analyzer" where users have hosted dependencies on them, but you want to override that and get them from within Flutter's SDK instead.

The "sdk: flutter" packages are totally fine because the Flutter SDK is the only way to refer to those.

We get the benefits of pinning a specific version, without having to continually maintain multiple pubspec.yaml files and make sure we're keeping them in sync.

Yes, pinning is certainly useful. That's why pub is designed around lockfiles. But you can pin a version without physically vendoring it. All you need is a specific identifier for it, which pub defines for you. "analyzer-1.2.3 from pub.dartlang.org" is a completely precise, immutable reference to a single unambiguous blob of code.

I totally get it's a drag to have to touch lots of pubspecs if you want your constellation of Flutter packages to all pin to one single version of analyzer (or other packages). I wonder if it would be easier to just hack together a little script that will just modify them all in concert?

We avoid any trouble with packages that are both in the Dart SDK and in pub

I'm not sure what you mean by "in the Dart SDK" here. The Dart SDK does physically contain its own copy of analyzer, but it's not user accessible.

It's faster. No need to hit the network to resolve these packages. Right now this costs us a lot of time, for example minutes for each bot iteration.

Well... that's because you discard your lockfiles because you want your bots to test that the package resolution works and can correctly satisfy its constraints.

If you want to test the resolution logic from scratch every time the bots run, yes, it does have an execution cost. If you don't want to do that, you could check in your lockfiles.

Or maybe just tweak your bots to keep the lockfiles around unless a commit touches a pubspec? Re-running the version solver even on commits which just change code and don't touch any pubspecs—which is the majority of them—doesn't do anything useful.

It's more reliable. The pub server sometimes returns 500s, this would become a non-issue. Right now we get at least one failure a day in the bots because of this.

😭 Have you talked to @mkustermann about this? Either way, if you can avoid testing the version solving from scratch on every single commit, it will mitigate this too.

Right now, it's impossible for us to reliably reproduce problems over time because the environment may have changed (e.g. a new version of a package might be on pub, and we don't know what version we needed to show the bug that was filed 6 months ago).

If you had checked in lockfiles it would give you this. A lockfile precisely describes exactly what you need to get the exact same versions of all transitively dependent packages at a point in time.

Lockfiles were created to provide hermetic builds.

The Dart SDK does physically contain its own copy of analyzer, but it's not user accessible.

It's very much user accessible. It's the version we use when you run flutter analyze. It's the reason for this PR. It's the source of the original issue that led to this conversation.

Lockfiles don't provide hermetic builds, they still depend on outside code. Lockfiles don't help when you're making a library or SDK, because each application that relies on Flutter has its own independent lockfile. We can't use lock files to pin our customers' dependencies. Today the only way we can do it is by pinning the version numbers in the pubspec.yaml files, which is a pain. (We've tried using tools to automate manipulation of pubspec.yaml files in the past, but we stopped. It wasn't great.) Lock files also don't have the benefit of keeping all the code in one place, which has huge reliability and performance benefits as I've discussed.

I'm not suggesting changing the semantics of pubspec.yaml version numbers. All I'm saying is that if we could have a prepopulated cache of the relevant parts of pub and prevent pub from considering other versions of those packages, our life would be better.