Add DB2 to TravisCI build by MMatten · Pull Request #543 · dbfit/dbfit

MMatten · 2017-02-11T21:57:20Z

Add DB2 integration tests to CI builds.

This seems to add roughly 2 minutes to the build time.

It might be possible run the container setup and DB scheme create as a background job, as per the Oracle/Docker setup, but I can't see how the method for Oracle ensure that the schema creation has completed successfully (@javornikolov any pointers welcomed).

javornikolov · 2017-02-11T22:09:37Z

It might be possible run the container setup and DB scheme create as a background job, as per the Oracle/Docker setup, but I can't see how the method for Oracle ensure that the schema creation has completed successfully (@javornikolov any pointers welcomed).

Yep, it was a bit tricky. The Oracle Docker image starts ssh server as last step https://github.com/wnameless/docker-oracle-xe-11g/blob/master/Dockerfile (i.e. if sshd is up - it means Oracle is already up). So I'm waiting for SSH daemon (remapped to port 49161) - via nc polling for an appropriate SSH response (its not enough to check that the port is open since the port remapping makes it seem like open too early).

javornikolov · 2017-02-11T22:14:22Z

.travis.yml

 language: java
 before_install:
+  - docker run --name dbfitdb2 -d -p 50000:50000 -v "`pwd`/dbfit-java/db2/src/integration-test/resources:/var/dbfit/dbfit-java/db2/src/integration-test/resources" -e DB2INST1_PASSWORD=db2inst1-pwd -e LICENSE=accept ibmcom/db2express-c:latest db2start
+  - docker exec dbfitdb2 /var/dbfit/dbfit-java/db2/src/integration-test/resources/create_db_schema.sh


Also I moved the oracle scripts below from java resources folder to test_vm/scripts - since it's not exactly a Java resource. Maybe we can do the same for db2?

Indeed. I meant to ask you about that. I've got shell and SQL scripts for this one. Do you think we should bother having shell and sql directories?

The reason I placed the .sql in separate folder alone is because the Oracle docker startup script is picking everything from the given path and running it.

javornikolov · 2017-02-11T22:20:35Z

.travis.yml


 language: java
 before_install:
+  - docker run --name dbfitdb2 -d -p 50000:50000 -v "`pwd`/dbfit-java/db2/src/integration-test/resources:/var/dbfit/dbfit-java/db2/src/integration-test/resources" -e DB2INST1_PASSWORD=db2inst1-pwd -e LICENSE=accept ibmcom/db2express-c:latest db2start


That's a pretty long line. Can we do anything about it?

I think putting these long command lines into shell scripts is probably the way to go. I could call a shell script instead in that before_install section.

But I'm not sure where the files from the Git repo are placed on the build server filesystem. Do you know?

You shouldn't rely on hard-coded paths, you only rely on having current dir in the project root.
Anyway - separate script is perhaps not worth for that. Lines can be wrapped like I did in the Oracle polling script. Also these paths seem excessively long to me (do we really need so long paths?)

MMatten · 2017-02-11T23:22:31Z

It might be possible run the container setup and DB scheme create as a background job, as per the Oracle/Docker setup, but I can't see how the method for Oracle ensure that the schema creation has completed successfully (@javornikolov any pointers welcomed).

Yep, it was a bit tricky. The Oracle Docker image starts ssh server as last step https://github.com/wnameless/docker-oracle-xe-11g/blob/master/Dockerfile (i.e. if sshd is up - it means Oracle is already up). So I'm waiting for SSH daemon (remapped to port 49161) - via nc polling for an appropriate SSH response (its not enough to check that the port is open since the port remapping makes it seem like open too early).

For Oracle, I not sure we can tell if the Oracle schema setup script has run through to completion.

For DB2, I think the details are those in the docker-entrypoint.sh (https://github.com/IMC3ofC/db2express-c.docker/blob/master/docker-entrypoint.sh) but even then I think I don't know enough about DB2 services to be able to check for something specific that would mean everything's up).

In any case, the DB2 schema setup is run with docker exec, after starting the DB2 services.

I think I could create something to poll for the existence of some of the DbFit test objects. I guess that would be the best option). But, is it worth bothering right now with this as we're only adding a couple of minutes to the process? @javornikolov What do you think?

javornikolov · 2017-02-11T23:30:16Z

I think I could create something to poll for the existence of some of the DbFit test objects. I guess that would be the best option). But, is it worth bothering right now with this as we're only adding a couple of minutes to the process? @javornikolov What do you think?

To be honest - I think you run it in background mode anyway (-d means detached mode). So I'm not sure you can guarantee DB2 schema is up on time with the current setup.

MMatten · 2017-02-11T23:39:15Z

To be honest - I think you run it in background mode anyway (-d means detached mode). So I'm not sure you can guarantee DB2 schema is up on time with the current setup.

It's proven to be reliable so far. The DB/schema/objects creation step has never failed so the container startup seems to be a very fast method of getting the services running. May be the actual docker run command is actually synchronous then in that respect.

Do you think we need to do any more with this at the moment?

javornikolov · 2017-02-11T23:41:48Z

I can see at the end of docker-entrypoint.sh it's waiting for some entries in a log file - maybe that guarantees that the db2 is up.

Do you think we need to do any more with this at the moment?

Perhaps not.

javornikolov · 2017-02-11T23:43:50Z

build.gradle


 task travisbuild(dependsOn: [
    fastbuild,
+    ':dbfit-java:db2:integrationTest',


What about moving that to the end of the list? I think mysql is the most light-weight so good to have it on first place I think.

Light weight because of the provisioning or because of the tests that we're running?

It's the total time until the first test is done. It's a speculative guess so far - I don't really know whether db2 is slower or not. Maybe the order is just fine as it is.

The DB2 tests are ready to run as soon at the before_script section has run. The complete build is taking 7 minutes to complete. I don't think moving the order will change that.

It's more about how long it takes for a failing test to fail. But anyway - I guess the difference is negligible compared to the total time with the setup before that. OK to be as is.

MMatten · 2017-02-12T10:02:41Z

I can see at the end of docker-entrypoint.sh it's waiting for some entries in a log file - maybe that guarantees that the db2 is up.

This seems to come into effect when -d is passed, as the first arg, to docker-entrypoint.sh but I'm not sure what would invoke this or how.

Having had a look at the log file though (it gets pretty big) I can see

MESSAGE : ADM7513W  Database manager has started.

which is probably something we can use. So we could get the Oracle and DB2 being pulled, extracted and started up in parallel. I'll try calling the DB/schema creation then from the before_script section.

MMatten · 2017-02-12T13:26:52Z

Having had a look at the log file though (it gets pretty big) I can see

MESSAGE : ADM7513W Database manager has started.

which is probably something we can use. So we could get the Oracle and DB2 being pulled, extracted and >started up in parallel. I'll try calling the DB/schema creation then from the before_script section.

The log exists in the container even before I start it and the log files has a previous entry with this string. I can check for a second record in the log file (from our docker run) but I'm not sure the additional complexity is actually required. The (pretty) Travis log file view becomes a bit of a mess too as the Oracle and DB2 pull/extract messages are interleaved.

I think I could leave these are they were for the time being (i.e. pull DB2, then create the DB and schema objects) then do the Oracle pull/extract stuff. What do you think?

javornikolov · 2017-02-12T13:53:24Z

Why the build has failed now?

MMatten · 2017-02-12T14:01:54Z

Why the build has failed now?

I need to correct the character escaping in the command line.

MMatten · 2017-02-12T17:06:51Z

Either I can't see the wood for the trees or there's something in the travis build script that's causing my docker command line to fail. Better options are i) a new shell script ii) go back to the original commit/PR.

MMatten · 2017-02-12T23:15:02Z

@javornikolov I've done some tidying up and tried a few new moves (in order to get DB2 pulling in parallel with the Oracle pull). Would you mind doing another review please?

javornikolov · 2017-02-12T23:10:17Z

.travis.yml


 language: java
 before_install:
+  - docker run --name dbfitdb2 -d -p 50000:50000 -v "`pwd`/test_vm/scripts/db2:/var/dbfit/test_vm/scripts/db2" -e DB2INST1_PASSWORD=db2inst1-pwd -e LICENSE=accept ibmcom/db2express-c:latest db2start &


I'm still wondering if it makes sense to have both -d and run it in background in the shell (via & suffix).

Even /var/dbfit/test_vm/scripts/db2 is quite long path. What about mounting it e.g. under /tmp?

I was trying to avoid too many changes to the shell scripts to maintain compatibility when they're being used to install DB2 via vagrant. I'll have another look at the scripts.

javornikolov · 2017-02-12T23:13:38Z

build.gradle


 task travisbuild(dependsOn: [
    fastbuild,
+    ':dbfit-java:db2:integrationTest',


It's more about how long it takes for a failing test to fail. But anyway - I guess the difference is negligible compared to the total time with the setup before that. OK to be as is.

javornikolov · 2017-02-12T23:18:20Z

@javornikolov I've done some tidying up and tried a few new moves (in order to get DB2 pulling in parallel with the Oracle pull). Would you mind doing another review please?

I already took a look. It looks pretty good to me 👍 Just a minor remark about the paths length.

MMatten · 2017-02-13T08:21:49Z

Ok. I've tidied up the path names now. I can't fully test that the vagrant VM build is still fully working because of #542

MMatten · 2017-02-13T14:02:49Z

I'm still wondering if it makes sense to have both -d and run it in background in the shell (via & suffix).

You mean perhaps we should just run everything in the foreground?

Sent from my iPhone On 12 Feb 2017, at 23:15, Yavor Nikolov <[email protected]<mailto:[email protected]>> wrote: @javornikolov commented on this pull request.

________________________________ In .travis.yml<#543 (review)>:

@@ -5,13 +5,16 @@ services:

language: java before_install: + - docker run --name dbfitdb2 -d -p 50000:50000 -v "`pwd`/test_vm/scripts/db2:/var/dbfit/test_vm/scripts/db2" -e DB2INST1_PASSWORD=db2inst1-pwd -e LICENSE=accept ibmcom/db2express-c:latest db2start & I'm still wondering if it makes sense to have both -d and run it in background in the shell (via & suffix).

________________________________ In .travis.yml<#543 (review)>:

@@ -5,13 +5,16 @@ services:

language: java before_install: + - docker run --name dbfitdb2 -d -p 50000:50000 -v "`pwd`/test_vm/scripts/db2:/var/dbfit/test_vm/scripts/db2" -e DB2INST1_PASSWORD=db2inst1-pwd -e LICENSE=accept ibmcom/db2express-c:latest db2start & Even /var/dbfit/test_vm/scripts/db2 is quite long path. What about mounting it e.g. under /tmp?

________________________________ In build.gradle<#543 (review)>:

@@ -246,6 +246,7 @@ task fastbuild(dependsOn: [

task travisbuild(dependsOn: [ fastbuild, + ':dbfit-java:db2:integrationTest', It's more about how long it takes for a failing test to fail. But anyway - I guess the difference is negligible compared to the total time with the setup before that. OK to be as is. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#543 (review)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABztFTEmU_Wls4By71u8RZZRFoFduJs3ks5rb5KQgaJpZM4L-T6O>.

MMatten · 2017-02-13T14:04:14Z

Even /var/dbfit/test_vm/scripts/db2 is quite long path. What about mounting it e.g. under /tmp?

Now mounted at `/scripts`.

…

Sent from my iPhone On 12 Feb 2017, at 23:15, Yavor Nikolov <[email protected]<mailto:[email protected]>> wrote: Even /var/dbfit/test_vm/scripts/db2 is quite long path. What about mounting it e.g. under /tmp?

MMatten · 2017-02-13T22:17:11Z

Ok. I've tidied up the path names now. I can't fully test that the vagrant VM build is still fully working because of #542

I seem to have broken the vagrant build with the shell script changes. Another push required...

MMatten · 2017-02-14T22:02:30Z

Ok. I've tidied up the path names now. I can't fully test that the vagrant VM build is still fully working because of #542

I seem to have broken the vagrant build with the shell script changes. Another push required...

Ok. The vagrant VM build is now working again with this latest push.

MMatten · 2017-02-16T22:26:02Z

@javornikolov javornikolov
I'm still wondering if it makes sense to have both -d and run it in background in the shell (via & suffix).

I'm not sure what you meant here. Just run it all sequentially and remove the checks for the services being up?

javornikolov · 2017-02-16T23:31:15Z

@javornikolov javornikolov
I'm still wondering if it makes sense to have both -d and run it in background in the shell (via & suffix).

I'm not sure what you meant here. Just run it all sequentially and remove the checks for the services being up?

@MMatten, sorry I didn't mean to to suggesting anything specific (maybe it's all fine). Just I'm noting that -d means to run the container in detached (background) mode and that & also means to run the command in background mode. So I just got puzzled about the double background mode. But it's possible that it's necessary (I'm not really aware in detail how it works).

MMatten · 2017-02-17T22:14:34Z

Just I'm noting that -d means to run the container in detached (background) mode and that & also means to run the command in background mode.

Right. I see what you mean now. I had noticed that a while ago but forgot to try it without &. May be we should use -d for the Oracle XE container too. I'll try DB2 first.

MMatten · 2017-02-17T22:29:25Z

Just I'm noting that -d means to run the container in detached (background) mode and that & also means to run the command in background mode.

Right. I see what you mean now. I had noticed that a while ago but forgot to try it without &. May be we should use -d for the Oracle XE container too. I'll try DB2 first.

It seems the difference in the behaviour, for the DB2 entry, with -d present and & removed, is that the pull and extract phase runs in the foreground, then the background service starts. It seems marginally slower (perhaps 30 seconds, perhaps less).

I'll try applying it to the Oracle container too now.

javornikolov · 2017-02-17T22:46:18Z

I think when using -d the output is suppressed and it's not quite visible what happened. That's why I ended up using & for Oracle.

MMatten · 2017-02-17T22:46:30Z

I'll try applying it to the Oracle container too now.

Another few seconds is all it seems to add, perhaps as we're only running one pull/extract at a time. I think the build log is tidier too.

What do you think of the options?

javornikolov · 2017-02-17T22:48:10Z

What do you think of the options?

What if we try both using & only without -d?

MMatten · 2017-02-17T22:51:06Z

Ok. I'll give that combination a go. I think the pull / extract messages might make a mess of the raw log (but ok when looking at the pretty version with the collapsed sections).

MMatten · 2017-02-17T23:29:08Z

Ok. I'll give that combination a go. I think the pull / extract messages might make a mess of the raw log (but ok when looking at the pretty version with the collapsed sections).

That shaves about 30 seconds off of the time and the log (even the raw log) looks fine actually. Looks good to me. Any other points you've noticed?

javornikolov · 2017-02-18T07:35:15Z

@MMatten, all looks good to me 👍 I'm merging it. Thanks a lot!

MMatten added 2 commits February 11, 2017 21:46

Refactor DB2 integration DB setup

03d1dc3

Add DB2 integration tests to TravisCI buid

b1d6b9d

javornikolov reviewed Feb 11, 2017

View reviewed changes

MMatten force-pushed the add-db2-to-travis-ci-build branch 2 times, most recently from 8ed6d37 to a27c018 Compare February 12, 2017 10:19

MMatten force-pushed the add-db2-to-travis-ci-build branch 4 times, most recently from ca973ee to 78aa7ce Compare February 12, 2017 14:50

MMatten force-pushed the add-db2-to-travis-ci-build branch from 78aa7ce to 6460844 Compare February 12, 2017 21:58

Set up Oracle and DB2 containers in parallel

ae1ab12

MMatten force-pushed the add-db2-to-travis-ci-build branch from 6460844 to ae1ab12 Compare February 12, 2017 22:07

MMatten requested a review from javornikolov February 12, 2017 22:52

MMatten force-pushed the add-db2-to-travis-ci-build branch from 73fac30 to c827a92 Compare February 12, 2017 22:56

Move DB2 setup scripts under test_vm/

9cfc43d

MMatten force-pushed the add-db2-to-travis-ci-build branch from c827a92 to 9cfc43d Compare February 12, 2017 23:04

javornikolov reviewed Feb 12, 2017

View reviewed changes

MMatten force-pushed the add-db2-to-travis-ci-build branch from 20b31de to bf73c0f Compare February 13, 2017 08:03

MMatten force-pushed the add-db2-to-travis-ci-build branch from bf73c0f to d557828 Compare February 14, 2017 22:01

MMatten mentioned this pull request Feb 15, 2017

Dev VM provisioning failure (lvm pvcreate /dev/sdb) #542

Open

MMatten force-pushed the add-db2-to-travis-ci-build branch from d557828 to 57fa3cc Compare February 17, 2017 22:18

MMatten added 2 commits February 17, 2017 23:11

Increase maximum waiting times for Oracle and DB2 services

79c3edf

Refactor DB2 setup scripts to use relative path names

190e0c1

MMatten force-pushed the add-db2-to-travis-ci-build branch from b357f0f to 1df4a72 Compare February 17, 2017 23:13

Run Docker pull for DB2 and Oracle XE in backgroud

63d1d3e

MMatten force-pushed the add-db2-to-travis-ci-build branch from 1df4a72 to 63d1d3e Compare February 17, 2017 23:27

javornikolov merged commit e353e93 into master Feb 18, 2017

javornikolov deleted the add-db2-to-travis-ci-build branch February 18, 2017 07:35

MMatten mentioned this pull request Mar 18, 2017

Add SQL Server and Informix to TravisCI environment #544

Open

javornikolov added techdebt techtask labels Jan 28, 2018

javornikolov added this to the 4.0.0 milestone Jan 28, 2018

Conversation

MMatten commented Feb 11, 2017

Uh oh!

javornikolov commented Feb 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MMatten commented Feb 11, 2017

Uh oh!

javornikolov commented Feb 11, 2017

Uh oh!

MMatten commented Feb 11, 2017

Uh oh!

javornikolov commented Feb 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MMatten commented Feb 12, 2017

Uh oh!

MMatten commented Feb 12, 2017

Uh oh!

javornikolov commented Feb 12, 2017

Uh oh!

MMatten commented Feb 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MMatten commented Feb 12, 2017

Uh oh!

MMatten commented Feb 12, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javornikolov commented Feb 12, 2017

Uh oh!

MMatten commented Feb 13, 2017

Uh oh!

MMatten commented Feb 13, 2017 via email

Uh oh!

MMatten commented Feb 13, 2017 via email

Uh oh!

MMatten commented Feb 13, 2017

Uh oh!

MMatten commented Feb 14, 2017

Uh oh!

MMatten commented Feb 16, 2017

Uh oh!

javornikolov commented Feb 16, 2017

Uh oh!

MMatten commented Feb 17, 2017

Uh oh!

MMatten commented Feb 17, 2017

Uh oh!

javornikolov commented Feb 17, 2017

Uh oh!

MMatten commented Feb 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MMatten commented Feb 12, 2017 •

edited

Loading

MMatten commented Feb 17, 2017 •

edited

Loading