Skip to content

Isolate tests by cluster and make run interface simpler#4593

Merged
kleimkuhler merged 44 commits intomainfrom
kevleim/test-cluster-configs
Jun 24, 2020
Merged

Isolate tests by cluster and make run interface simpler#4593
kleimkuhler merged 44 commits intomainfrom
kevleim/test-cluster-configs

Conversation

@kleimkuhler
Copy link
Contributor

@kleimkuhler kleimkuhler commented Jun 12, 2020

Summary

Change the default behavior of integration tests to be isolated by cluster.
Additionally, make running one or all tests easier than the current process.

These changes are explained more in the Testing
RFC

Changes

This is a script used only by Linkerd developers, but there is a lot of useful
usage examples and explanations in bin/tests --help output:

Run Linkerd integration tests.

Optionally specify one of the following tests: [upgrade helm helm-upgrade uninstall deep external-issuer]

Usage:
    tests [--images] [--images-host ssh://linkerd-docker] [--name test-name] [--skip-kind-create] /path/to/linkerd

Examples:
    # Run all tests in isolated clusters
    tests /path/to/linkerd

    # Run single test in isolated clusters
    tests --name test-name /path/to/linkerd

    # Skip KinD cluster creation and run all tests in default cluster context
    tests --skip-kind-create /path/to/linkerd

    # Load images from tar files located under the 'image-archives' directory
    # Note: This is primarly for CI
    tests --images /path/to/linkerd

    # Retrieve images from a remote docker instance and then load them into KinD
    # Note: This is primarly for CI
    tests --images --images-host ssh://linkerd-docker /path/to/linkerd

Available Commands:
    --name: the argument to this option is the specific test to run
    --skip-kind-create: skip KinD cluster creation step and run tests in an existing cluster.
    --images: (Primarily for CI) use 'kind load image-archive' to load the images from local .tar files in the current directory.
    --images-host: (Primarily for CI) the argument to this option is used as the remote docker instance from which images are first retrieved (using 'docker save') to be then loaded into KinD. This command requires --images.

Run all tests

Old:

bin/test-run $PWD/bin/linkerd

New:

bin/tests $PWD/bin/linkerd

Run single test (upgrade for example):

Current:

. bin/_test-run.sh
init_test_run $PWD/bin/linkerd
upgrade_integration_tests

New:

bin/tests --name upgrade $PWD/bin/linkerd

Run tests in isolated KinD clusters

Current: Not possible without running single tests in newly created clusters
manually

New:

bin/tests $PWD/bin/linkerd

Run tests in isolated namespaces on an existing cluster

Old:

bin/test-run $PWD/bin/linkerd

New:

bin/tests --skip-kind-create $PWD/bin/linkerd

CI

kind_integration has been updated so that it does not create a KinD cluster as
part of its test setup.

cloud_integration passes the --skip-kind-create flag so that the tests are
run serially in a non-KinD cluster.

Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
@kleimkuhler kleimkuhler requested a review from alpeb as a code owner June 12, 2020 20:08
Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
@joakimr-axis
Copy link
Contributor

@joakimr-axis Let me know if you have any feedback on this! It touches a lot of the testing scripts that you've had an eye on.

Sure thing! I have added a few suggestions now!

Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
Copy link
Member

@alpeb alpeb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kleimkuhler, all this looks pretty good to me 👍

One thing I noticed is the calls to helm delete were removed. This might cause issues when running all the tests locally, as both the helm install and helm upgrade tests use the same helm release name. And also the helm upgrade test should remove its release as to not conflict with a further test run (and also to leave the user's helm config as it was before).

Probably you're already aware of it, but this still needs to add the edge upgrade test that got just merged into master, and to propagate the changes from kind_integration.yml and cloud_integration.yml into release.yml.

@alpeb
Copy link
Member

alpeb commented Jun 16, 2020

Also I've just noticed the helm upgrade test is currently failing silently in master because the latest_stable function is no longer defined. Depending on how long will the current PR going to get merged, it might be worth fixing that in a separate PR to have it run for this week's edge and next week's stable.

@kleimkuhler
Copy link
Contributor Author

@alpeb Good catch! I opened #4622

@kleimkuhler kleimkuhler requested a review from a team as a code owner June 17, 2020 21:46
Copy link
Member

@adleong adleong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks awesome!


The integration test suite operates on your currently configured Kubernetes
cluster. Prior to running the test suite, verify that:
The integration tests will configure their own KinD clusters by default. There
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't kind a prerequisite? What about resource requirements? Is kind running on docker-for-desktop on a 4 year old macbook going to cut it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't kind a prerequisite? What about resource requirements?

The script uses bind/kind which downloads kind if it is not present. You don't need kind in your PATH to use the script.

Is kind running on docker-for-desktop on a 4 year old macbook going to cut it?

I haven't had any issues with running kind locally on my 2015 macbook. If anything, Docker For Mac with Kubernetes turned on seems to use a lot more resources than kind ever does; I rarely use Docker For Mac these days because of that.

Also keep in mind we already use kind on the GitHub Action runners which I think are 2-core machines with 7 GB of ram. Knowing we don't have issues on those, I think we're good to assume running this locally will be fine as well.

Signed-off-by: Kevin Leimkuhler <[email protected]>
Signed-off-by: Kevin Leimkuhler <[email protected]>
@olix0r olix0r changed the base branch from master to main June 24, 2020 18:37
@kleimkuhler kleimkuhler merged commit 4372ed5 into main Jun 24, 2020
@kleimkuhler kleimkuhler deleted the kevleim/test-cluster-configs branch June 24, 2020 21:06
kleimkuhler added a commit that referenced this pull request Jun 25, 2020
## Description

As discussed [here](#4653 (comment)), the `kind_integration` job of the release workflow was not kept in sync with the changes made in #4593.

Until GitHub actions can reuse yaml for separate workflows, these sections are supposed to be kept in sync.

This would be an issue if we had tried doing a release since #4593 merged, but that has not happened yet.

## Changes

This updates the release workflow `kind_integration` job to use the use new test interface, mainly removing cluster creation and image loading as necessary prerequisites.

Signed-off-by: Kevin Leimkuhler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants