@Kononnable @pleerock @vlapo I couldn't get Travis to really work the way I wanted it. The main issue is an inability to share cache between the build stages. If I had access to a Docker registry, I'd build an image which would then cache the install, lint and build stages so we could basically focus on spending all our time in tests. As things are, the overhead of running those processes over and over kills any advantage we get from running everything in parallel. And Travis does not provide any sort of Docker cache so you have to pull each image on each run. We could spin our own and store everything in an S3 bucket, but that seems like too much of a hack.

That's not to say we can't run things in parallel. If we don't care so much about build times, we can do this quite easily:

Running like that builds take around 16-17 minutes (and can take more if there's a wait for jobs to start or especially if there are several commits at once).

I also cleaned up some of the tests and externalized config so you can set CONNECTIONS in your environment, which would be a comma-separated list that overrides the skip values in ormconfig.json. It's better for CI to be able to control such things via environment vars and the cleanup I did now allows someone to enable any (save mssql or oracle) connection and run tests locally with no bleed over.

I know I started out talking about jest, but I wanted to first see how things run in CI. To take full advantage of jest, I think we'd need to introduce some magic layer where each test creates and runs in its own db or schema. I also removed npm install sqlite3 --build-from-source because it doesn't seem necessary and takes some time.

As I see it, the options at this point are:

1. Parallel gives useful insights into how the tests are running and build times aren't that important so let's clean this up and publish it.
2. Keep the test cleanup, but don't bother with parallel execution. Instead figure out how to speed things up with jest.
3. Find a different CI tool with better support for build stages.
4. Something else?

We could spin our own and store everything in an S3 bucket, but that seems like too much of a hack.

It would be a problem on non-collaborator builds(majority of PRs) - Travis doesn't allow to use secrets in such builds(because someone might steal them).

We might get it a little bit faster by separating compilation from running tests and compile in the background while downloading docker images(https://stackoverflow.com/questions/19543139/bash-script-processing-limited-number-of-commands-in-parallel). However db often accepts connections few seconds(even minutes in case of oracle) after container is up - so we would have to retry until it is ready.

As for jest - I didn't have an opportunity to use it yet but I'm not sure if we will gain much from it if new layer needs to be added. Database creation for each test probably will be slow(often it creates new files in db container). New schema might be an option(few tests use multiple schemas, so schema with prefix would be necessary) but I think mysql, mariadb and oracle support only one schema per database.

Another option would be to make tests separated from db driver they run on(single test runs single db driver, current test checks postgres, mysql -> 2 tests, one check only postgres, another one only mysql) and run them in parallel but make sure that two tests with same driver won't be running at the same time. Not sure if this is easily achievable.

@Kononnable I tried separating compilation from execution. I wanted to run those things as separate stages in the job, but travis doesn't have good support for sharing things between stages. Running all in one process? I don't see how that would be better than what we have now.

Creating a new and empty db should be fast, but yeah I don't know for sure that approach will work. It might get complicated supporting all these different db types. I don't think it's really possible to understand without having a working model.

@havenchyk Thanks for the comments. If we split up the builds like this, I'd want to make sure sqlite is only installed when needed, otherwise it's pretty slow (and again Travis doesn't really help much). As for the style, I got TypeScript errors when trying to use includes. I'm assuming because the tsconfig target is set to es5. All that code will need another look and cleanup before considering this for merging.

@elthrasher I didn't get what's wrong with the idea to build docker image and then run it many times? Will it be too slow because of building step?

@havenchyk The trouble there is Travis. There's no local Docker cache so I would have to either build the image in each process or build it, push it to some registry (now credentials are a problem), then pull it in each process.

@elthrasher I was talking about the second approach (build, upload, use), but also have you tried using matrix? I haven't used this option, but it can solve at least the problem with sqlite. I'm checking docs at the moment, so can't suggest anything better before I finish and try this option

Yeah, a few of my attempts did something like that. The problem still is caching Docker. However, looks like there may be a way to hack around it: travis-ci/travis-ci#5358 (comment)
Probably worth a try. The alternative would be to move to some other CI provider that supports Docker caching better.

I tried a bunch more things and TravisCI is just basically hostile to what I want to do. If I want to run a few parallel jobs, each of them will have its own cache that cannot be shared. There are a few github issues in which the Travis team says this is by design and won't be changed. Some of their reasoning makes sense, but it definitely doesn't in the context of a Docker build.

What I want to do is build one version of the code in node 10 and then run several tests in parallel against that. It's not doable in TravisCI short of setting up an S3 integration or a Docker registry. The latter could be possible or we could look for another CI solution. In the mean time I'll leave this here and look at some of the jest ideas.

@Kononnable I tried separating compilation from execution. I wanted to run those things as separate stages in the job, but travis doesn't have good support for sharing things between stages. Running all in one process? I don't see how that would be better than what we have now.

@elthrasher
I was thinking that we could compile typeorm, npm install etc. while downloading docker images in the background - it would save us some time(I don't really know how much time typeorm compilation takes on Travis). It would still be running multiple times if compilation is parallel but won't take additional time.

Okay, lots of failures and a few successes here.

There's been a long-standing issue that you can't just run tests against one kind of db. For example, disable all but postgres and run the tests and you'll get failures because it can't find mysql. I've fixed that so if mysql is disabled in ormconfig.json, no tests that require it will be run.

I added an env variable CONNECTIONS that will override the skip property of ormconfig.json. That makes it easier to manage what runs and when.

I tried to do a sort of pipeline where the build happens once and then we run a bunch of parallel processes, one for each db connection. TravisCI just doesn't support this. There's no way to pass files from one stage to the next. Caches are not sharable. The only way we could do this on TravisCI is to publish a Docker image and then pull it for each parallel stage or to stage files on S3, both of which have their own complications. The real gain here would probably be fairly minimal as each stage would have to pull the Docker image, cutting into the time we saved.

I experimented with running parallel mocha processes in the same job (sort of how it would work with jest, but explicitly by db connection). This works and gives us a slight improvement in time taken, however we get six times the logs so I had to switch to a more terse logging format.

I tried running the setup stuff in parallel and saw some performance gains, but it gets complicated because travis doesn't do a great job of waiting for the setup to finish before the tests begin. If we were running 100 builds per day, I'd still optimize this, but I don't think the added complexity is worth it for typeorm.

I removed node 9, which saw end-of-life last summer in favor of node 11. Fortunately all the tests still pass. I assume this is just a chore that nobody has gotten to yet rather than a commitment to support node 9.

I tried altering the tests so that each test would create its own schema or database, which would let us convert to jest and full parallel execution (though it still remains to be seen how much improvement we'd get out of that). However, it's extremely complicated to do that for all the databases typeorm supports. I think it's doable, but should probably be its own PR if we decide we want to try it.

Thoughts? If we want to go ahead with this, I'll add some more comments.

Just reference to issue related to this changes #2807

There's been a long-standing issue that you can't just run tests against one kind of db. For example, disable all but postgres and run the tests and you'll get failures because it can't find mysql. I've fixed that so if mysql is disabled in ormconfig.json, no tests that require it will be run.

mysql is required because very-very-old tests were using it (in the times when there was only mysql support in typeorm) and we still didn't refactor those old tests

TRAVIS

if travis is a problem lets try some other CI. I have experience working only with travis and circleci. Are there better and popular/perspective alternatives?

@pleerock This PR refactors those tests so that mysql won't always be required.

It's not that Travis is ruining the project or anything. It's just that some of the suggestions made above won't work on Travis. It kills me a little that CI runs for 4-5 minutes before the tests even start, but is this really a big problem for typeorm? Probably not.

I've added my comments so this is good to go from my perspective if we like these changes. If not, happy to discuss or revise.

Tests run via ts-node only now and are much faster! I'm surprised, but we've cut over two minutes off. This should also make development much nicer as compilation isn't necessary to run tests. I think we're now getting out of mocha what we could've hoped to get out of jest, so that conversion won't be necessary with the current feature set.

Build times are still running 6-7 minutes but only 150 seconds are spent actually executing the tests. If we want to run faster, we could look at another CI, but I would merge this PR and start another for that.

This commit: 82ad52b#diff-165e041541ffbf626d08c6ac034a9c58R7 has removed almost all of the tests from execution via only. I've fixed that here.

Ah well, running only one test explained the incredible test performance. Looks like we're back to normal levels, but I still think this is worth it.

Ok, to sum up what are Pros and Cons of this PR?

Pro:
CONNECTIONS environment variable
Some tests fixed
Db drivers don't wait for each other(so tests should be faster, especially if mssql and oracle are enabled)
ts-node - faster development ( I was using tsc -w for compiling so no big difference for me)

Con:
We don't see result for each test(benchmark tests doesn't make much sense now)

I probably missed something here, so feel free to update the list. Is there any failing CI report so we can see how it looks like when CI fails?

Here's how it looks when it fails: https://travis-ci.org/typeorm/typeorm/jobs/474388884

The performance gains for running parallel seem minor, so that could be removed for more verbose output, thus keeping most of the pros without the con.

Doesn't seem like the parallel execution is getting us anything other than confusing output so I removed it. I still think there are enough good things here to consider merging and maybe think about another CI which would support parallel better - or just solve some completely different problem. 10 minutes for CI isn't that bad.

Im okay to merge it

I added circleci configuration. There are lot of useful options in circleci documentation. Sure we can improve ci process now

I've measured times on my machine:
Enabled drivers: mysql, mariadb, postgres, sqljs

configuration    this PR      master
linux1           16m2s        16m9s  
linux2           1m50s        2m32s
windows          5m49s        5m37s

Configurations where different(hardware, db engine config) so there is no point in comparing linux to windows etc. It looks like ts-node doesn't introduce performance problems on windows anymore(at least in our case). I think it's good or, we could support both ways if it is easy to implement.

I'll give circleci a look when I can. As for running both ts-node and compiled, the main downside of that is that we don't need to ever compile the tests when we use ts-node - in fact they are excluded in the tsconfig.json in this PR. It does save a few seconds.

@elthrasher I wasn't sure about this when I proposed this try, but I use ava as a test runner and there is an option there to run tests without pre-compilation, with requiring ts-node and it seemed to be faster :)

So whats a status of this ? :-)

I like the idea of using ts-node for our tests - can we get a PR that's just that?

It seems there's a bunch of changes here that kinda all got conflated & made it harder to agree on merging them all in together

We aren't using travis any longer so that part's no longer relevant.

setupConnection was dropped, too.

I'll be closing this one, though - cause there's lots of conflicts and not too much movement for over a year.