@mfn as per your comment on my first PR - would love you to give this version a spin on your suite of 6,000+ tests.

And to answer your question: There is no need to manually do any config cache / busting. This version caches to a dedicated cache.phpunit.php file (or as specified in the phpunit.xml).

I tried it and realized I'm on 5.8 (obviously), had to replace Env::get with env()

But unfortunately, as I thought, it's not measureable with my suite. I also can only run it in a timely manner on CI (in parallel) in the cloud and we all know they don't give you the prime resources for doing that.

Sorry for bringing up hopes.

Can't we move the listener to the foundation component in Illuminate\Foundation\Testing? It feels weird to include it in the skeleton.

@driesvints I think that would probably be a good idea yea (and I mentioned this in my initial comment). I'll send the PR to laravel/framework if you are keen to move forward with this as a thing in core - just don't wanna spam you with PRs to manage.

The only bad thing with the Listener is, exactly as you said, the framework boot problem.

Not too long ago I used a dedicated listener for a Laravel project and it would not have worked together with your because I too had to boot the framework they.

Eventually I found a different solution for the same problem without using a listener.

Do you think it's possible without that?

@mfn could you shed light on why your listener would have conflicted with this one?

This listener does boot the framework - but the thing that takes the longest during booting is collating all the config. This listener caches the config on first run, meaning that all subsequent boots are ~50% quicker (as per my benchmarking).

I would think that means your listener would actually become faster because when it boots the framework it doesn't have to collate all the config on the fly. Or am I missing something?

The Listener was, as in your case, a special one which also booted the framework.

But: it just came to my mind, I'm not using it anymore and I think hardly anyone does so I guess you can just disregard my comment ;)

@mfn no worries. if you do think of any specific scenarios where this could conflict (with or without another listener) please do mention it. I'd be happy to run some tests and see if we need to make adjustments if any are needed.

I've just pushed a new commit to this. You can now specify arbitrary commands to run before the test suite, rather than locking it down to config caching only.

It makes it more generic so devs can define the commands to run before the testsuite runs for the first time.

<listeners>
    <listener class="Illuminate\Foundation\Testing\Listener">
        <arguments>
            <!-- Define artisan commands to execute before the test suite runs -->
            <string>config:cache</string>
            <string>route:cache</string>
            <string>event:cache</string>
            <string>some-3rd-party-package:cache-stuff</string>
        </arguments>
    </listener>
</listeners>

Isn't Listener a really gwneral name? I'd rather have a CacheForTestsListener or another name that describes what this is doing better

Agreed. I think PreRunListener or SetupListener or something along those lines would be better. I’ve been more focused on the implementation than those details up until now so if anyone has any other suggestions, please do share.

I also figured Taylor, if he wants this, would know what he’d want to call it as well.

...although if this is the Laravel listener, I dont think it would be the worst thing to call it Listener. Just like the TestCase is called TestCase instead of BootsFrameworkTestCase.

But I don’t have strong opinions on the naming either way TBH

Someone just suggested this could be done in a bootstrap file...and I think I was too deep down the rabbit hole to realise that is probably the better way to go compared to a listener.

Just ship a tests/bootstrap.php or bootstrap/tests.php file, reference it in the phpunit.xml bootstrap tag and then you can provide classic Laravel style documentation as guidance as well. Again it is all in "user land" so the dev is in full control if they want to take advantage of it.

pretending these comments were not terrible...it could be something like this...

<?php

use Illuminate\Contracts\Console\Kernel;

require_once __DIR__.'/../vendor/autoload.php';

/*
|--------------------------------------------------------------------------
| Bootstrap the testing environment
|--------------------------------------------------------------------------
|
| You have the option to specify console commands that will execute before your
| test suite is run. Caching config, routes, & events may improve performance
| and bring your testing environment closer to production.
|
*/

$commands = [
    'config:cache',
    'route:cache',
    'event:cache',
];

$app = require __DIR__.'/../bootstrap/app.php';

$console = tap($app->make(Kernel::class))->bootstrap();

foreach ($commands as $command) {
    $console->call($command);
}

Yeah I think that makes decent sense probably.

I've just pushed changes to implement to bootstrap file instead of the Listener.

I've commented out the route:cache command as the routes that come in the skeleton are not cacheable. Commenting it out means that the 2 tests in the skeleton still pass out of the box.

My two cents: I think the file would be better lived under tests/bootstrap.php since that's a common location in most PHP apps I've seen so far.

I also believe the current bootstrap/app.php file would be better placed as ./bootstrap.php and the cache directory as a top-level directory or underneath storage but that's a different topic :)

We've actually found in our initial feature test benchmarks that caching the routes can actually slow down the test suite. Looks like it is due the the performance of unserialize. Our initial benchmarks were against a test app with 1,000 routes. I'm going to run some more tests against different route numbers through the week. Looks like most apps, according to this Twitter poll have far less routes.

Gonna do some more digging into this throughout the week and will report back regarding route caching.

Happy to close this and re-open once I have a better overall understanding of if route caching is a worthwhile thing to include in the bootstrapping process.

Now that I think about it, this would also break tests which register routes on runtime (I think that's a common approach for testing middleware

@m1guelpf runtime defined routes continue to work if the routes are cached, e.g. the following test passes with routes cached before the test run:

public function test_runtime_routing()
{
    Route::get('my-test-route', function () {
        return 'expected';
    });

    $response = $this->get('my-test-route');

    $this->assertSame('expected', $response->content());
}

Let me know if this isn't what you are referring to though. I might have misunderstood.

@timacdonald can you move the bootstrap file to tests/bootstrap.php?

@adrianb93 what did you actually do in the file to achieve those results?

@adrianb93 what did you actually do in the file to achieve those results?

Nothing crazy. They're both apps that work with many databases. It was just moving most of the setup work to the start of the suite.

• 23-minute suite: It needed to copy a couple of database schemas and do some seeding for some critical third-party connections we need to integrate with that are subject to schema change. This was a slow setup for each test.
  • No, this integration has no API or staging environment.
  • Yes, we still develop our app as if there is in case we can switch to another the third-party or if there are improvements.
  • It's less than ideal integration work, but it is what it is.
• 3-minute suite: This is a multi-tenant database app. The bootstrap provisions a default testing tenant database before running the suite.
  • If you fake the tenant db handler, the default connection is used.
  • Provisioning time seemed reasonable in this app but there was time to be saved here.
  • It was also valuable to use register_shutdown_function in the bootstrap to delete any databases starting with testing_tenant_* in case a test didn't call deprovision on a tenant. This is cleanup work that was once done manually.

In both cases, each test starts transactions in each test setup and rolls back in the before application destroy.

These aren't the first apps I've worked on that have built up this sort of unavoidable DB cruft. The bootstrap file helped make things better for these apps.

Really appreciate so many people taking the time, here and on other forums, to give me feedback on this PR ❤️ Hope it comes in handy for you and others.

@timacdonald awesome job! Working great for me when running from the command line (+-2s faster), however having some issues getting it work when running individual test via PHPStorm interface. Anyone who has managed to get that working?

By the way, if anybody wants to run something after the suite finishes, add this to the end of the booststrap file:

// Will run once the entire test suite has finished
register_shutdown_function(function () use ($console) {
    // Figured this is a helpful example for those on a Laravel version below 5.8
    $console->call('config:clear');
});

Thanks for this @adrianb93 I'm on Laravel 5.8 and I was experimenting 419 errors after the tests.

Edit: I just realised that env variables are correctly set when I dump them during tests but this is still config.php, services.php, ... that are in use instead of config.phpunit.php.
I had to remove the variables from phpunit.xml and put them in my .env.testing file. Now it's working without having to call 'config:clear'.

however having some issues getting it work when running individual test via PHPStorm interface. Anyone who has managed to get that working?

@patrickbrouwers I'm on Ubuntu with PHPStorm 192.5728.108 and run them through docker. For me, they still work individually with no extra configuration.

@patrickbrouwers Same problem here, I can't run individual tests. I got this error :

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable.  

Same for you ? I'm investigating for a solution...

@bastien-phi yes that's the error I'm getting too. I got it to work when I specify the absolute path in the env settings for the cache files.

@patrickbrouwers I found the exact same solution right now !
Not very convenient when sharing the phpunit.xml file with teammates IMO ...

@bastien-phi I personally always gitignore the phpunit.xml and ship a phpunit.xml.dist

@patrickbrouwers I will just update tests/bootstrap.php to update environment variables with absolute path before calling the framework :

$envVars = ['APP_CONFIG_CACHE', 'APP_SERVICES_CACHE', 'APP_PACKAGES_CACHE', 'APP_ROUTES_CACHE', 'APP_EVENTS_CACHE'];
foreach ($envVars as $envVar) {
    $_ENV[$envVar] = __DIR__ . '/../' . $_ENV[$envVar];
}

@timacdonald Any other idea as APP_CONFIG_CACHE (and others) are defined by default as absolute path in Illuminate/Foundation/Application ?

@cbaconnier as per my comment here: #5050 (comment) have you set the env vars correctly in the phpunit.xml? I found most people were skipping this step and thus having the issue you are where the cached values are overriding the normal config files.

The env values need to be set differently in 5.8. Instead of <server you need to use <env e.g.

- <server name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>
+ <env name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>

@timacdonald Oh... 😅 Thanks! That was it

@cbaconnier no worries, that has tripped a few people up.

@bastien-phi not sure. Can you put your setup in here so we can try and work out why it isn't working for you? Are you calling it from inside PHPStorm as well? If so, could you try on the command line as well and see if it is any different? Would be good to work what part of your stack is causing issue for ya.

@timacdonald I created a repo from a fresh install of laravel, with the phpunit bootstrap file : laravel-phpunit-bootstrap

I updated the bootstrap file with a "patch" in order to make it work properly.

Before the patch, ./vendor/bin/phpunit --config phpunit.xml works properly but calling phpunit from PHPStorm in a single file does not work.
PHPStorm calls the command /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity and I get

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable.  

Calling this command from the project root, the tests goes well.

The difference between the two calls is that from inside PHPStorm, current working directory is the directory when the test file belongs (in that case tests/Unit), and from the command line current directory is the project root.

If calling the command from another directory (let's say tests), the command line fails too :

$ cd tests && /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable. 

As the environment variables are relative paths, the application tries to find the directory bootstrap/cache relatively to current working directory which cause the fails.

The patch updates relative paths to absolute path and everything goes well. Unfortunately, this is far from being a clean fix...

@bastien-phi I can confirm it works without the patch in vim-test and sublime-phpunit, but unfortunately I do not have a copy of PHPStorm to help debug this directly. But a google lead me to this: https://www.jetbrains.com/help/phpstorm/run-debug-configuration-phpunit.html#commandLine

If I am understanding that correctly, they are saying the CWD is always set to the project root.

By default, the field is empty and the working directory is the root of the project.

Can you check that option out (and possibly whatever else is in there), confirm you don't have a value in there, and maybe have a play and see if there is anything in there you need to change to get it working?

@timacdonald This option is kept empty and the CWD is not set to the project root but to the directory where the test class belongs :

$ /usr/bin/php /path/to/laravel-phpunit-bootstrap/vendor/phpunit/phpunit/phpunit --configuration /path/to/laravel-phpunit-bootstrap/phpunit.xml Tests\Unit\ExampleTest /path/to/laravel-phpunit-bootstrap/tests/Unit/ExampleTest.php --teamcity
current working directory : /path/to/laravel-phpunit-bootstrap/tests/Unit

In PackageManifest.php line 168:
                                                               
  The bootstrap/cache directory must be present and writable. 

It is possible to explicitly set CWD as project root for phpunit in PHPStorm with Run Configuration Templates. Be sure to remove all existing configurations first.

It works this way.

But IMO, there is still a problem because CWD has to be project root to run the tests. Before this feature, it was possible to launch tests from another location.

The problem does not comes from the bootstrap file itself but from the fact that the files are defined with relative paths in the phpunit.xml. Currently I find only 3 solutions :

1. remove the path definitions in phpunit.xml
2. update the path definitions in phpunit.xml with absolute paths
3. explicitlty update CWD in tests/bootstrap.php to set it to project root.

1 is not a good idea at all !
With 2, there is a need to ship a phpunit.xml.example, gitignore phpunit.xml and manually update the paths.
My favorite solution is 3 but I don't know if there could be side effects.

<?php

use Illuminate\Contracts\Console\Kernel;

chdir(__DIR__.'/../');

require_once __DIR__.'/../vendor/autoload.php';

/*
|--------------------------------------------------------------------------
| Bootstrap the testing environment
|--------------------------------------------------------------------------
|
| You have the option to specify console commands that will execute before your
| test suite is run. Caching config, routes, & events may improve performance
| and bring your testing environment closer to production.
|
*/

$commands = [
    'config:cache',
    'event:cache',
    // 'route:cache',
];

$app = require __DIR__.'/../bootstrap/app.php';

$console = tap($app->make(Kernel::class))->bootstrap();

foreach ($commands as $command) {
    $console->call($command);
}

Laravel 5.8 with 171 tests 5-10% improvement 👏️

Before:

Time: 53.27 seconds, Memory: 68.25 MB
Time: 48.13 seconds, Memory: 68.25 MB
Time: 51.73 seconds, Memory: 68.25 MB
Time: 52.96 seconds, Memory: 68.25 MB
Time: 48.15 seconds, Memory: 68.25 MB

After:

Time: 42.16 seconds, Memory: 70.25 MB
Time: 42.26 seconds, Memory: 70.25 MB
Time: 47.57 seconds, Memory: 70.25 MB
Time: 43.92 seconds, Memory: 70.25 MB
Time: 44.53 seconds, Memory: 70.25 MB

OK (171 tests, 1135 assertions)

🙂️👌️

nice @stevelacey! Thanks for sharing. Glad to see a little perf bonus there for ya

@timacdonald I replaced your codes and it improved ~10% of test suit which is nice, But i dont know why i'm getting General error: 1 no such table: users on the browser after i using your codes , and it gone when i clear cache folder any idea?

@foremtehan I just had this same problem, I suspect it’s because 5.8 doesn’t have the separate caches per env logic thats coming in 6.0, and thus Laravel sees cache and ignores the .env? And thus your DB_NAME is wrong, and thus no such table.

@timacdonald is there something else we should patch in?

@foremtehan @stevelacey see #5050 (comment)

You will also want to run just once right now php artisan config:clear to make sure you get the test values out of your local config cache.

p.s. @foremtehan ~10% is a nice speed improvement 🎉

@timacdonald this PR is causing some issues:

laravel/framework#29862
#5091

We'll need to figure out solutions for these fast or else it might be best to revert this PR.

#5071 does the job

@driesvints digging into this now. Will respond in the relevant threads to keep discussions together.

I had exact problem, I just comment these out in phpunit.xml and everything back to work :

<env name="APP_CONFIG_CACHE" value="bootstrap/cache/config.phpunit.php"/>
<server name="APP_SERVICES_CACHE" value="bootstrap/cache/services.php"/>
<server name="APP_PACKAGES_CACHE" value="bootstrap/cache/packages.php"/>
<server name="APP_ROUTES_CACHE" value="bootstrap/cache/routes.phpunit.php"/>
<server name="APP_EVENTS_CACHE" value="bootstrap/cache/events.phpunit.php"/>

@foremtehan that’ll mean that your testing env is being cached in your normal cache files, so you probably hit some weird bugs when using your app locally. Once laravel/framework#29890 is tagged it should be fixed and you’ll be right to restore everything