DB2 installer for dev/test VM by MMatten · Pull Request #286 · dbfit/dbfit

MMatten · 2014-05-07T22:29:45Z

Chaps,

For your consideration...

As I want to get a fully functioning test VM for the Teradata updates (in case of core fixes) I've created an installer for DB2 Express-C for the VM. It's much along the lines of the Oracle shell script.

Not knowing much at all about Chef I've added the provisioning it straight to the Vagrantfile.

One known issue if that it doesn't make the JDBC driver available to the build (I copy it to dist/lib in the script but this just gets blitzed by the build) as I can't put stuff onto your S3 area (and know almost nothing about how to configure gradle anyway).

This is my first github pull request so hope I'm not breaking any etiquette!

Cheers,

Mark

javornikolov · 2014-05-08T16:45:06Z

dbfit-java/db2/src/test/resources/install_db2.sh

Why 32 bit?

I must have been looking at old IBM docs that say 32 and 64bit are required for the installer but after experimenting it seems that only the 64bit libs are needed and that's the package that will actually get installed.

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=%2Fcom.ibm.db2.luw.qb.server.doc%2Fdoc%2Fr0008865.html

javornikolov · 2014-05-08T16:59:15Z

@MMatten, what would happen if provisioning is executed twice?
E.g. for Oracle we skip the installation if it's already installed. The check is done via grep of /etc/oratab: test_vm/vendor/cookbooks/dbfit_test/recipes/oracle.rb

MMatten · 2014-05-08T23:23:32Z

Good spot on the provisioning. I've added a check for an existing instance.

On the provisioning - is it better to use Chef than a Vagrant shell provisioner anyway? I don't know the pros and cons of that, and I can't really help with Chef either though.

javornikolov · 2014-05-08T23:31:36Z

On the provisioning - is it better to use Chef than a Vagrant shell provisioner anyway? I don't know the pros and cons of that, and I can't really help with Chef either though.

I think it's good to keep things relatively uniform. Actually the Oracle recipe is pretty much shell script - it should be straightforward to convert this one in a similar shape. For reference - take a look at oracle.rb and Vagrantfile; instead of yum - try adding packages with package and the rest is just configuring the path of the shell script.

javornikolov · 2014-05-09T23:19:11Z

In case there is no DB2 package in dbfit root (means it won't be installed): the script is still installing the related rpm packages.

javornikolov · 2014-05-09T23:27:45Z

dbfit-java/db2/src/test/resources/install_db2.sh

Who will be the owner of this folder in case we create it here?

BTW: the content of dist folder is being purged when running some of the DbFit build tasks (bundle, start*).
I'm not sure we have to deal with deploying the jdbc driver at this stage. Maybe better to leave Gradle to deal with it.

I was expecting for this to be removed anyway. If you can let Gradle deploy the driver into dist/ then great. Can put a copy of the driver onto your S3 space (I think that's where the Oracle driver is pulled from into the local maven repo) and update the build or would you do using a script? redeploy-odjbc.bat is still present but I guess is dead now as no maven.

javornikolov · 2014-05-09T23:38:52Z

db2_linuxx64_expc.tar.gz - this should be added to .gitignore.

javornikolov · 2014-05-09T23:49:15Z

I got some errors on the unpacking step. But I was using another box (due to some other experiments and due to download from the official gets stuck for some reason).

vagrant-shell: INFO: found gzipped installation tarball. Unpacking...


Stderr from the command:

tar: expc/db2/linuxamd64/install/tblm_SCSIPR.so: Cannot change ownership to uid 1, gid 1: Operation not permitted
tar: expc/db2/linuxamd64/install/db2help: Cannot change ownership to uid 1, gid 1: Operation not permitted
tar: expc/db2/linuxamd64/install/locale/ja_JP.eucJP/db2istring.cat: Cannot change ownership to uid 1, gid 1: Operation not permitted
tar: expc/db2/linuxamd64/install/locale/ja_JP.eucJP/db2install.cat: Cannot change ownership to uid 1, gid 1: Operation not permitted
tar: expc/db2/linuxamd64/install/locale/ja_JP.eucJP: Cannot change ownership to uid 1, gid 1: Operation not permitted
...

benilovj · 2014-05-10T07:12:00Z

@MMatten thanks for doing this work - so with this working within the VM, do the DB2 tests pass?

MMatten · 2014-05-10T08:00:04Z

@benilovj - Jake, yes, they (the non i/z series tests) all pass.

MMatten · 2014-05-10T08:11:16Z

@javornikolov Javor - what happens if you run the tar command in a different directory, like /tmp ?
What's the ownership and mode of your /var/dbfit dir?
Are you running the script manually via sudo, or as root or as vagrant or via vagrant/Chef?

javornikolov · 2014-05-10T08:52:09Z

What's the ownership and mode of your /var/dbfit dir?

Folder is mounted via NFS. Owner is from my host OS (uid/gid are not present in the guest).

Are you running the script manually via sudo, or as root or as vagrant or via vagrant/Chef?

I ran vagrant provision.

Uncompressing to /tmp works OK. Also uncompress from vagrant user (not root) is OK. Following seems to work under /var/dbfIt too: tar --no-same-owner -xzvf ....

MMatten · 2014-05-10T14:45:46Z

@javornikolov "--no-same-owner" looks like safe option considering the troubles I've also had with NFS. I guess that's why this might have worked for me (as Windows and so no NFS). I'll update the script to use this.

javornikolov · 2014-05-10T16:43:59Z

@javornikolov "--no-same-owner" looks like safe option considering the troubles I've also had with NFS. I guess that's why this might have worked for me (as Windows and so no NFS). I'll update the script to use this.

I have some doubts though - in theory something may depend on the ownership of files in the archive (if db2 installer doesn't set them explicitly afterwards).

javornikolov · 2014-05-10T21:38:07Z

Seems db2 installer is clever enough to handle setup when the archive is untarred with --no-same-owner. Provisioning succeeded and I managed to run Db2 tests and they succeeded. Awesome!

javornikolov · 2014-05-10T21:40:14Z

When rebuilding the machine (while db2jcc4.jar is already in dist/lib) I got the following error:

cp: cannot create regular file `/var/dbfit/dist/lib/db2jcc4.jar': Permission denied
vagrant-shell: ERROR: copying DB2 JDBC driver to DbFit lib directory /var/dbfit/dist/lib

…allation

MMatten · 2014-05-10T22:45:27Z

Ok, I've just pushed to my repo for some rework to do i) use Chef instead of the shell provisioner ii) allow installation via vagrant/chef or by executing the shell script whilst logged in iii) a tear down script to remove the installation (doesn't get run at the moment, may be useful to someone), though this will currently only succeed against a fully successful installation iv) changes to the tar switches v) if installing by running the shell script package installation won't be executed if there's no DB2 installation to perform (I think the Chef version might still do this though).

Also, I removed the JDBC driver copy to dist/lib.
Would you be be able to update the gradle build to deploy this instead?

javornikolov · 2014-05-11T06:23:55Z

The .sh scripts don't have executable bit set (good to set that: chmod +x *.sh).

javornikolov · 2014-05-11T06:24:26Z

dbfit-java/db2/src/test/resources/deinstall_db2.sh

No newline at the end of file here.

javornikolov · 2014-05-11T06:58:18Z

dbfit-java/db2/src/test/resources/install_db2.sh

Alignment issue here (mixing spaces and tabs).

javornikolov · 2014-05-11T06:58:28Z

dbfit-java/db2/src/test/resources/install_db2.sh

misalignment here too

MMatten · 2014-05-11T07:09:27Z

@javornikolov Hi, I can add the missing EOLs on the last line of the shell scripts. As for the file modes - I had checked in from the Windows host. I didn't even know Git kept properties such as mode for files. Do you know a way to recommit these (without making a dummy change, committing, removing it, committing, etc) or modify the file permissions?

javornikolov · 2014-05-11T07:10:22Z

test_vm/vendor/cookbooks/dbfit_test/recipes/db2.rb

Do we need to invoke that with DOLIBS when we have them installed here via the chef package?

The in -d the Chef command means don't do the detection as it's already called in the not_if.
The -l in the Chef command means don't install the library packages as it's done via the recipe.

Ah, indeed. OK.

javornikolov · 2014-05-11T07:13:19Z

As for the file modes - I had checked in from the Windows host. I didn't even know Git kept properties such as mode for files. Do you know a way to recommit these (without making a dummy change, committing, removing it, committing, etc) or modify the file permissions?

I wasn't sure either but seems Git keeps the properties - and chmod is seen as change. Maybe you can try to change them from within the VM - It shows up e.g. like that:

diff --git a/dbfit-java/db2/src/test/resources/deinstall_db2.sh b/dbfit-java/db2/src/test/resources/deinstall_db2.sh
old mode 100644
new mode 100755

MMatten · 2014-05-11T07:53:53Z

@javornikolov I chmodded the files and git saw them as a change. For some reason mode changes don't get listed by ls though. I think this is another Win/Linux VB synced folder issue.

MMatten · 2014-05-11T07:59:57Z

@javornikolov I just cloned my fork to a new local repo on the guest and the local files didn't have the x attribute. So I've added from Linux, committed and pushed again. Should be ok now.

javornikolov · 2014-05-11T08:41:39Z

Thanks! It looks good now.
I'll rebuild my VM to make sure it all works fine end to end.

javornikolov · 2014-05-11T09:55:37Z

I'll rebuild my VM to make sure it all works fine end to end.

So gave it a try and all the provisioning (including db2) is working great for me 👍

MMatten · 2014-05-11T10:09:13Z

Great stuff. Thanks! We still wont be able to execute tests until we can get the driver into the build.
Is there anything else you need me to do on this? Will you accept it into your the main repo?

benilovj · 2014-05-11T10:24:25Z

LGTM - merging

DB2 installer for dev/test VM

benilovj · 2014-05-11T10:24:49Z

thanks @MMatten!

MMatten · 2014-05-11T10:26:55Z

It's been a pleasure! Always glad to help out with odds and ends.

MMatten added 4 commits May 7, 2014 19:31

Added DB2 installer to provisioning

4f887bd

Correct title Oracle -> DB2

d142cc2

Correct flags on rm of installation package files

fcf1584

Correctly set passwd for dftest OS user

dac3205

javornikolov reviewed May 8, 2014
View reviewed changes

Succeed early if detect an existing db2inst1 installed instance

f8fdd72

javornikolov reviewed May 9, 2014
View reviewed changes

Refine DB2 installer: use Chef and better detection for existing inst…

1968160

…allation

javornikolov reviewed May 11, 2014
View reviewed changes

dbfit-java/db2/src/test/resources/deinstall_db2.sh

Copy link
Copy Markdown

Contributor

javornikolov May 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No newline at the end of file here.

javornikolov mentioned this pull request May 11, 2014

JUnit test to invoke Db2 acceptance tests #294

Closed

javornikolov reviewed May 11, 2014
View reviewed changes

dbfit-java/db2/src/test/resources/install_db2.sh Outdated

Copy link
Copy Markdown

Contributor

javornikolov May 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alignment issue here (mixing spaces and tabs).

javornikolov reviewed May 11, 2014
View reviewed changes

dbfit-java/db2/src/test/resources/install_db2.sh Outdated

Copy link
Copy Markdown

Contributor

javornikolov May 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

misalignment here too

javornikolov reviewed May 11, 2014
View reviewed changes

Add missing last line EOLs and +x on shell scripts

93ab086

correct chmod +x on shell scripts - from Linux this time

c3d9515

benilovj added a commit that referenced this pull request May 11, 2014

Merge pull request #286 from pmsoftware/master

37942e2

DB2 installer for dev/test VM

benilovj merged commit 37942e2 into dbfit:master May 11, 2014

benilovj added the techdebt label May 11, 2014

benilovj added this to the Next release milestone May 11, 2014

This was referenced May 11, 2014

Add Db2 acceptance tests to the build #296

Merged

Release 3.0.1 #303

Merged

Conversation

MMatten commented May 7, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javornikolov commented May 8, 2014

Uh oh!

MMatten commented May 8, 2014

Uh oh!

javornikolov commented May 8, 2014

Uh oh!

javornikolov commented May 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javornikolov commented May 9, 2014

Uh oh!

javornikolov commented May 9, 2014

Uh oh!

benilovj commented May 10, 2014

Uh oh!

MMatten commented May 10, 2014

Uh oh!

MMatten commented May 10, 2014

Uh oh!

javornikolov commented May 10, 2014

Uh oh!

MMatten commented May 10, 2014

Uh oh!

javornikolov commented May 10, 2014

Uh oh!

javornikolov commented May 10, 2014

Uh oh!

javornikolov commented May 10, 2014

Uh oh!

MMatten commented May 10, 2014

Uh oh!

javornikolov commented May 11, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MMatten commented May 11, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javornikolov commented May 11, 2014

Uh oh!

MMatten commented May 11, 2014

Uh oh!

MMatten commented May 11, 2014

Uh oh!

javornikolov commented May 11, 2014

Uh oh!

javornikolov commented May 11, 2014

Uh oh!

MMatten commented May 11, 2014

Uh oh!

benilovj commented May 11, 2014

Uh oh!

benilovj commented May 11, 2014

Uh oh!

MMatten commented May 11, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone