Replace `serverspec` with `inspec` tests via. new CI matrix (inc. `semantic-release`) #262

myii · 2019-04-25T14:20:07Z

Currently, there is no CI-based testing on this formula. This PR brings in a number of the latest changes from the template-formula, in order to bring inspec testing via. a matrix of pre-salted images configured in Kitchen and run via. Travis. In it's current state, 9/10 of the platforms pass the tests. Here's a list of what remains before this can be merged:

Note that the formula itself needs to be improved to get it working with pillar.example instead of the provided test pillar (if we want to go that way). The improvements that need to be made can be seen by the commented out sections in the test pillar.

Update: Just a note about an issue with serverspec. While testing out these ideas initially, tried to hook up the existing serverspec tests but ended up hitting a weird bug where it was trying to run apt-get on opensuse -- tracked it down to this block in kitchen-verifier-serverspec.

daks

Looks good to me. Maybe squash some commits.
Are you planning to implement semantic-release too?

Very good job @myii! Thanks

myii · 2019-04-25T22:04:21Z

@daks Thanks for the review. You're right about the commits -- I was planning to do that at the end. The idea right now is that each commit is a part that may have changes depending on the review, so the PR can be modified as required. For example, I definitely want to lose the last commit as well as maybe the one before that. It's easier to make the necessary modifications while this is a fluid WIP.

I left the semantic-release commented out because it's definitely needed, whether now or very soon. It opens up the discussion for it also.

Really appreciate you taking the time to review this PR.

vutny

Awesome job, @myii !

I'm not aware of the settings for this repo, but GitHub should dismiss any previous approvals if you'd force-push any commits afterwards. So better to squash them right before submitting a PR.
And any separate commit has no real value, since it may or may not represent the finished task and actually working changes. If it cannot be safely reverted in master later, I think a single squashed commit is the right way.

daks · 2019-04-26T13:48:06Z

@vutny I think what it means is that during work (WIP PR) it's simpler to have one commit per change, it eases testing changes, reverting them if needed, etc. Then when work is finished, commits can be squashed.
I tend to use this pattern too.

myii · 2019-04-30T16:03:09Z

With 13/13 images passing and the README updated, this PR is now ready for final review.

aboe76

@myii nice work on this formula !!!!

myii · 2019-05-02T08:34:58Z

While 14/14 images being tested appears excessive, all sorts of combinations are there, besides different platforms and salt versions, as supplied by the pre-salted images:

Testing with and without installing from the upstream repo.
Testing multiple PostgreSQL versions.
Testing multiple ports.

So if there is a requirement to remove some of these images, care must be exercised so worthwhile testing isn't lost.

daks

Great job @myii

.travis.yml

* Quoting from https://kitchen.ci/docs/getting-started/kitchen-yml/: - As of test-kitchen 1.21.0, we now prefer `kitchen.yml` over `.kitchen.yml`. - This preference applies to `kitchen.local.yml` as well. - This is backward compatible so the dot versions continue to work.

* `pillar.example` will be difficult to get working at this stage * Ideally, work back towards `pillar.example` in the long run

* Fix the formula and then revert this eventually

myii · 2019-05-02T18:15:55Z

While not directly related to serverspec => inspec, adding semantic-release to this PR was very straightforward after all of the other changes made. I've already tested it in my fork, the same way we have done with all of the other deployments. This PR is ready to merge, depending on any final comments.

myii · 2019-05-03T09:41:05Z

Travis is hitting stuck builds on all three of the opensuse instances:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

Tried restarting each of them to no avail. There was nothing changed to have triggered this.

* Add new `centos-6` image * Install `9.6` from upstream repo

* Fix `opensuse-leap-15` (enable `suse` and workaround `service` bug) * Use port `5433` for `debian` and `opensuse-leap-15` * Use upstream repo version `10` for `debian`

myii · 2019-05-06T03:39:23Z

Thanks to @javierbertoli, the Travis issue has been resolved and we're back to 13/13 images passing. This is ready to merge again.

saltstack-formulas-travis · 2019-05-06T07:40:53Z

🎉 This PR is included in version 0.37.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

myii requested review from aboe76, daks, javierbertoli and vutny April 25, 2019 14:20

myii mentioned this pull request Apr 25, 2019

Use copy state instead of salt-call #256

Closed

daks approved these changes Apr 25, 2019

View reviewed changes

vutny approved these changes Apr 26, 2019

View reviewed changes

myii force-pushed the chore/add-new-ci-matrix branch 6 times, most recently from 4487e37 to 55f1cf0 Compare April 30, 2019 15:09

myii changed the title ~~WIP: Replace serverspec with inspec tests via. new CI matrix~~ Replace serverspec with inspec tests via. new CI matrix Apr 30, 2019

aboe76 approved these changes Apr 30, 2019

View reviewed changes

myii mentioned this pull request May 2, 2019

Setting postgres.port does not work on RedHat OSes #266

Open

daks approved these changes May 2, 2019

View reviewed changes

.travis.yml Show resolved Hide resolved

myii force-pushed the chore/add-new-ci-matrix branch from 314cddb to ce1a00f Compare May 2, 2019 14:49

myii added 7 commits May 2, 2019 19:08

ci(gemfile): prepare for inspec testing

157e169

ci(kitchen): use pre-salted images as used in template-formula

611ec11

ci(travis): add .travis.yml based on template-formula

6467df7

ci(pillar_from_files): use custom pillar based on pillar.example

c64d9e4

* `pillar.example` will be difficult to get working at this stage * Ideally, work back towards `pillar.example` in the long run

test(inspec): replace serverspec with inspec tests

58ac122

test(inspec): use relaxed command output match for the time being

3c53684

* Fix the formula and then revert this eventually

myii force-pushed the chore/add-new-ci-matrix branch from ce1a00f to 4cee3ca Compare May 2, 2019 18:12

myii changed the title ~~Replace serverspec with inspec tests via. new CI matrix~~ Replace serverspec with inspec tests via. new CI matrix (inc. semantic-release) May 2, 2019

myii mentioned this pull request May 2, 2019

Establish template-formula as the go-to resource for all formula development saltstack-formulas/template-formula#21

Open

71 tasks

myii force-pushed the chore/add-new-ci-matrix branch from 4cee3ca to 668f070 Compare May 3, 2019 08:46

javierbertoli mentioned this pull request May 4, 2019

travis + opensuse repos are not playing well together netmanagers/salt-image-builder#4

Closed

myii force-pushed the chore/add-new-ci-matrix branch 5 times, most recently from 23ae027 to d0c340a Compare May 6, 2019 03:10

myii added 5 commits May 6, 2019 04:19

ci(kitchen+travis): use newly available pre-salted images

7b7aadc

docs(readme): update Testing section for inspec

4cfde8d

test(inspec): enable use_upstream_repo for debian & centos-6

49fdd33

* Add new `centos-6` image * Install `9.6` from upstream repo

test(inspec): add tests for multiple ports and postgres versions

bf6a653

* Fix `opensuse-leap-15` (enable `suse` and workaround `service` bug) * Use port `5433` for `debian` and `opensuse-leap-15` * Use upstream repo version `10` for `debian`

feat: implement semantic-release

7d3aa19

myii force-pushed the chore/add-new-ci-matrix branch from d0c340a to 7d3aa19 Compare May 6, 2019 03:19

daks merged commit 2a54616 into saltstack-formulas:master May 6, 2019

saltstack-formulas-travis added the released label May 6, 2019

myii deleted the chore/add-new-ci-matrix branch May 6, 2019 08:36

myii mentioned this pull request May 14, 2019

test(locale): improve test using locale en_US.UTF-8 #269

Merged

Replace serverspec with inspec tests via. new CI matrix (inc. semantic-release) #262

Replace serverspec with inspec tests via. new CI matrix (inc. semantic-release) #262

Uh oh!

Conversation

myii commented Apr 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daks left a comment

Choose a reason for hiding this comment

Uh oh!

myii commented Apr 25, 2019

Uh oh!

vutny left a comment

Choose a reason for hiding this comment

Uh oh!

daks commented Apr 26, 2019

Uh oh!

myii commented Apr 30, 2019

Uh oh!

aboe76 left a comment

Choose a reason for hiding this comment

Uh oh!

myii commented May 2, 2019

Uh oh!

daks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

myii commented May 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

myii commented May 3, 2019

Uh oh!

myii commented May 6, 2019

Uh oh!

saltstack-formulas-travis commented May 6, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Replace `serverspec` with `inspec` tests via. new CI matrix (inc. `semantic-release`) #262

Replace `serverspec` with `inspec` tests via. new CI matrix (inc. `semantic-release`) #262

myii commented Apr 25, 2019 •

edited

Loading

myii commented May 2, 2019 •

edited

Loading