My spec in #8287 includes details about how conflict rules impact this optimization. But your implementation ignores them entirely, which might lead to breaking the dependency resolution (imagine that the latest version of a package adds a conflict rule forbidding only the latest release of the other one but allowing the previous patch release for instance)

regarding the time where the pool optimizer is called, I'm not sure it is the best one compared to the PRE_POOL_CREATE event triggered for plugins. If plugins are modifying the pool packages, that might break the invariants of the optimization if it is already done.
I think it would be safer to optimize the pool just before entering the solver, but after plugins have tweaked it.

My spec in #8287 includes details about how conflict rules impact this optimization. But your implementation ignores them entirely, which might lead to breaking the dependency resolution (imagine that the latest version of a package adds a conflict rule forbidding only the latest release of the other one but allowing the previous patch release for instance)

It would get a different dependency hash and not be removed from the pool. They are actually not ignored at all :)

ah sorry, I missed that part about the hash.

@Toflar idea to ensure that the PoolOptimizer has no negative effect on the dependency solving: run the solver functional tests with and without the PoolOptimizer:

• all successful tests MUST have the same outcome in both cases
• tests reporting a dependency solving error MUST do it in both cases, but the exact output might change (as that output depends on versions that the solver tried, and it will try less of them with the optimization). Fixture files must be written for the optimized pool, as that's what users will see.

What do you think ?

Should we have some way (CLI flag, env variable, magic incantation, etc...) to disable the PoolOptimizer when running composer, to help identify cases where a dependency solving error is caused by a bug of the optimization (if solving finds a solution without the PoolOptimizer, there is a bug which must then be reported and fixed in Composer) ?
Of course, users should never need to disable it. But we are all human (at least I think so) and bugs happen. Being able to quickly know that the optimizer is breaking things would help.
And any such case would then enhance the testsuite forever if we implement the idea I gave in my previous comment.

I just wanted to share results of my tests with Drupal. Before each test, I removed the vendor, web/core (Drupal core) directories, and the composer.lock file. To test, I ran composer install --profile pointing to the correct composer binary.

A not-very-complex Drupal 9.1 dev project (this repo)

Drupal 9.1 core (dev release)

I can see the time improvement is not significant but memory improvement is. This is awesome work. Thanks! 👏

I also tried this on a project with Drupal 8; this time running composer update --profile. I ran it once before the actual tests so that caches get created, etc. This project had many out-of-date dependencies so there were many packages that got updated. I didn't clear out vendor directory for each run but before each run, I restored composer.lock to the previous state.

Again, not a significant differnce in time (this could be because it actually times updating packages on disk) but a significant reduction in peak memory.

A Symfony application using the Symfony Flex plugin (needed because our Symfony applications use * as version constraint for symfony packages):

Not much changes, which I think it expected as the Symfony Flex plugin already significantly reduces the versions that are considered.

A Neos base distribution - the most complex project on my PC that isn't requiring Symfony Flex:

Similar results as Drupal 8, not a significant step in performance but a significant reduction of (peak) memory usage.

Btw, checked the installed versions after each run and all runs install the same versions (between each run the vendor packages + composer.lock file were removed). So no version resolving regressions found in these 2 applications.

Good job man ! I tested this two days ago with almost same runtime. Now this PR is a lot faster. No idea why :)
Another thing is the memory usage. Really impressive. Improvements in this area in my opinion not less important than the runtime.

I tested a different Drupal scenario than the two from @hussainweb above, and got less dramatic results, but still a very nice improvement.

My interest was for the use-case of doing a patch-level Drupal core update when all other dependencies are already up to date. This is because we're working on an initiative to add an auto-updater to Drupal, where this scenario will be run on production, including on cheap shared hosting plans with low memory limits.

Setup: begin with a Drupal installation that's on a recently outdated patch-level version (latest Drupal version is 9.0.6). Also include some non-core dependencies (lightning_media brings in a bunch).

> composer create-project drupal/recommended-project drupal 9.0.5
> cd drupal
> composer config repositories.assets composer https://asset-packagist.org
> composer require drupal/lightning_media

Test: profile an update to the latest Drupal core version. Start with a cold cache, since that's the likely scenario on shared hosting by the time this would be run.

> composer clearcache
> composer update drupal/core-recommended --with-all-dependencies --profile

Result:

While not an order of magnitude difference, that's still an excellent reduction in peak memory usage, which can make a big difference on some cheap hosting plans.

@stof regarding the test suite: We already have loads of integration tests in the InstallerTest which test from the actual composer update command to the expected lock and output. Those tests are all green. It took me a while to get them there and you can actually see I had to adjust one of them 😉 I agree that the more tests, the better. But I guess they'll come along once we stumble upon new constellations which haven't been tested yet. We can also isolate tests specifically to the PoolOptimizer, that's why I've implemented the foundation for additional tests in this PR and also provided a few basic tests.

Should we have some way (CLI flag, env variable, magic incantation, etc...) to disable the PoolOptimizer when running composer, to help identify cases where a dependency solving error is caused by a bug of the optimization (if solving finds a solution without the PoolOptimizer, there is a bug which must then be reported and fixed in Composer) ?

Yes, totally. I've implemented and documented the new --no-optimizer flag in 96dcbc9 😎

To everybody: Thanks a lot for the numbers, they look really promising which makes me very happy 😊
Just a quick note regarding the performance tests: This PR does not have any influence on downloading or unpacking etc. but is specifically designed to improve the resolving process. So there's no point in running comparisons by clearing the Composer cache first. You should run composer update first, to make sure the package information cache is primed. Then run COMPOSER_DISABLE_NETWORK=1 composer update --profile to make sure there are no requests dispatched. This should work without any problems if you primed the cache correctly in the first place.

Those tests are all green. It took me a while to get them there and you can actually see I had to adjust one of them wink I agree that the more tests, the better. But I guess they'll come along once we stumble upon new constellations which haven't been tested yet.

The fact that problem reporting is impacted by the pool optimization is precisely the reason why my proposal is that this "no-optimizer" run disables the assertion on the output in case of reported problems (keeping only an assertion that a problem is indeed reported).
Such double run would allow us to guarantee that the optimized pool yields the same outcome than the non optimized pool (and so that the optimization is safe).

The output can't be the same though. In case of an unresolvable set of dependencies, the list of problems is a lot shorter if not all possible problems are listed but just the ones of the optimized pool (which BTW also improves readability a lot :)).

The output can't be the same though. In case of an unresolvable set of dependencies, the list of problems is a lot shorter if not all possible problems are listed but just the ones of the optimized pool (which BTW also improves readability a lot :)).

I know that. That's why I'm saying that for problems, the non-optimized run must NOT assert the output, but only the fact that the run also triggers some problems (to make sure that the dependency solver is not successful when disabling optimization).

I'm not sure a no-optimizer flag is the way to go. What about controlling if it is enabled via an environment variable. Composer already does this to control some other stuff.

In any case, wouldn't this also need to be added to the install command too, since it still defaults to doing an update when there is no lock file?

Using COMPOSER_DISABLE_NETWORK=1 composer update -v --dry-run --profile --no-plugins

For ezsystems/ezplatform:2.5:

For ezsystems/ezplatform-ee:2.5, contains commercial packages so you won't be able to replicate w/o subscription:

For the record: I had a look at why Drupal is so much slower compared to all the other projects and the reason is not Composer at all. The SAT solver is super fast for them too but the way Drupal handles package meta information is very, very slow (loads of redirects and 404s on https://www.drupal.org/files/packages and https://packages.drupal.org). It easily accounts for 80-90% of the total time spent during a composer update run. I think that's a bottleneck the Drupal community has to tackle.

@mvorisek this is not helpful. Work is being tracked and scheduled here, please don't go around nagging people.

@Seldaek The comment from him might have sounded offensive to you but to be fair people not that closely related to this project have no chance to understand how much communication or agreement between you members and Toflar about a feature like this exists outside of this PR (at least this is what I read in some comments). They see just a feature of a contributor laying around for such a long time and no obvious reason.
A lack of a real (even rough) feature or release scheduling is not helpful for one looking forward having this or that feature as well.
Don't get me wrong I understand the reasons here and other priorities I just tried to give another perspective on this.

Just in case it may help, from #10200 there is a scenario that benefits greatly from this improvement (from timeout to 2s):

{
    "require-dev": {
        "alcaeus/mongo-php-adapter": "^1.2.1",
        "doctrine/data-fixtures": "^1.5.1",
        "doctrine/doctrine-bundle": "^2.4.3",
        "doctrine/mongodb-odm": "^2.2.2",
        "doctrine/mongodb-odm-bundle": "^4.3.0",
        "doctrine/orm": "^2.10.1",
        "doctrine/phpcr-bundle": "^2.3.0",
        "doctrine/phpcr-odm": "^1.5.4",
        "jackalope/jackalope-doctrine-dbal": "^1.7.1",
        "ocramius/proxy-manager": "^2.13",
        "psy/psysh": "^0.10.9",
        "symfony/symfony": "^4.4.32 || ^5.3.9",
        "theofidry/composer-inheritance-plugin": "^1.2",
        "theofidry/psysh-bundle": "^4.4",
        "wouterj/eloquent-bundle": "^2.1.1"
    }
}

(the difference is relevant only if there is no .lock & no vendor). I do not know if this may worsen other cases though.

Do you want to try https://github.com/spryker-shop/suite ? :)

composer 2.1: Memory usage: 181.61MiB (peak: 2073.42MiB), time: 22.55s

This is probably the most demanding composer project in the world, and as such most likely will show the most dramatic improvements here.

@dereuromark here you go 😉

This looks awesome, about 25% faster and 80% reduction in memory usage for a medium sized Drupal 9 site with 30 modules:

Tested with this:

$ composer self-update --preview
$ COMPOSER_DISABLE_NETWORK=1 composer update -v -n --dry-run --profile

Tested with Open Y ( 229 composer packages )

MacBook-Pro-Andrii:openy-project podarok$ composer show | wc -l
     229

composer cache warmed up

OSX host

php -v
PHP 7.4.25 (cli) (built: Oct 23 2021 18:05:05) ( NTS )

Command for testing

composer -v -n --profile create-project ymcatwincities/openy-project --no-interaction

composer --version
Composer version 2.1.11 2021-11-02 12:10:25
[174.5MiB/44.21s] Memory usage: 174.54MiB (peak: 1000.15MiB), time: 44.21s

composer --version
Composer version 2.2.0 2021-12-22 11:03:27
[143.5MiB/48.11s] Memory usage: 143.49MiB (peak: 222.83MiB), time: 48.11s

Memory improvement -
77.7% memory usage reduction peak in comparison to composer 2.1.11.
Average memory - 17.7% usage reduction

Time is a bit higher +8.8% increase

@podarok you should test without the network interfering (see #9261 (comment)) :)

Just wanted to take a minute to thank you for making composer this performant, we really appreciate it! It has been a game changer