Skip to content
This repository was archived by the owner on Jun 24, 2020. It is now read-only.

Conversation

@houshengbo
Copy link

@houshengbo houshengbo commented Feb 5, 2020

Fixes #287

Proposed Changes

  • This PR adds the tests to verify the correct number and names of knative serving
    deployments.

  • The test tag postupgrade is added, marking the tests to run after upgrade to the
    latest HEAD of operator, with the latest generated manifest of knative serving.

Release Note


@googlebot googlebot added the cla: yes Author(s) signed a CLA. label Feb 5, 2020
Copy link
Contributor

@knative-prow-robot knative-prow-robot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@houshengbo: 0 warnings.

Details

In response to this:

This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.

Fixes #

Proposed Changes

Release Note


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: houshengbo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

1 similar comment
@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo houshengbo force-pushed the upgrade-tests branch 2 times, most recently from 08046ce to dcd0b59 Compare February 5, 2020 20:17
@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo houshengbo force-pushed the upgrade-tests branch 3 times, most recently from fb6ba5c to 6156800 Compare February 5, 2020 21:02
@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

@houshengbo
Copy link
Author

/test pull-knative-serving-operator-integration-tests-on-latest-serving

import (
"fmt"
"errors"
"k8s.io/apimachinery/pkg/util/wait"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we group the imports by

  • go libs
  • knative libs
  • everything
    ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will align them in the correct order.

@@ -0,0 +1,230 @@
/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

common.go is overly generic.
can we break it out to
verify.go - to cover all the assert/verify helpers
setup.go - to cover all the setup/creation helpers?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will create a file called setup.go, and put it under the folder client.
The file verify.go can be put under the folder resources, since it verifies everything about the knative resources.



// KnativeServingVerify verifies if the KnativeServing can reach the READY status.
func KnativeServingVerify(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im conflicted about the naming
i understand the CRD is called KnativeServing, but a method that says KnativeServingVerify is confusing for me.
maybe append CR at the end, as we are verifying the CR status.
thoughts?

// Get the existing KS without any spec
ks, err := clients.KnativeServing().Get(names.KnativeServing, metav1.GetOptions{})
// Add config to its spec
ks.Spec = v1alpha1.KnativeServingSpec{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pass in the configmap to this helper.
that way it can be reused for other changes

}

// KnativeServingConfigure verifies that KnativeServing config is set properly
func KnativeServingConfigure(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debating if we need to prefix everything with KnativeServing

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use KS for short.

}

// DeploymentRecreation verify whether all the deployments for knative serving are able to recreate, when they are deleted.
func DeploymentRecreation(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would break this into separate methods
deleteDeployments
verifyDeployments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can keep this method, but rename it DeleteAndVerifyDeployments, to make it clear all that the method is doing.

}

// KnativeServingDelete deletes tha KnativeServing to see if all resources will be deleted
func KnativeServingDelete(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.
split this into smaller pieces.

readonly LATEST_SERVING_RELEASE_VERSION=$(git describe --match "v[0-9]*" --abbrev=0)
# Istio version we test with
readonly ISTIO_VERSION=1.1.3
readonly ISTIO_VERSION="1.4.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to determine this dynamically? the version should be in a file in third_party/?

Copy link
Author

@houshengbo houshengbo Feb 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.4-latest is taken as default, if this param is not set.
HOWEVER, the directory is not available under the path, https://raw.githubusercontent.com/knative/serving, but it is available under the source code folder.
It only happens to this 1.X-latest directory, not to others like 1.4.2.
I have to set a specific version here to access via https://raw.githubusercontent.com/knative/serving/\<version>

@@ -0,0 +1,230 @@
/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would prefer not to use the "common" folder name, as typically ends up the dumping grounds for everything.

t.Run("verify resources", func(t *testing.T) {
common.KnativeServingVerify(t, clients, names)
// TODO: We only verify the deployment, but we need to add other resources as well, like ServiceAccount, ClusterRoleBinding, etc.
ExpectedDeployments := []string{"networking-istio", "webhook", "controller", "activator", "autoscaler-hpa",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this capitalized

Vincent Hou added 2 commits February 10, 2020 10:35
This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.
@houshengbo houshengbo force-pushed the upgrade-tests branch 2 times, most recently from 23b427b to 841a22f Compare February 10, 2020 17:12
local base_url="https://raw.githubusercontent.com/knative/serving/v${LATEST_SERVING_RELEASE_VERSION}"
local base_url="https://raw.githubusercontent.com/knative/serving/${LATEST_SERVING_RELEASE_VERSION}"
# Decide the Istio configuration to install.
if [[ -z "$ISTIO_VERSION" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this ever be false? isn't it always set and can't be overwritten?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove it.

function knative_setup() {
function install_latest_operator_release() {
header "Installing Knative Serving operator latest public release"
local url="https://github.com/knative/serving-operator/releases/download/${LATEST_SERVING_RELEASE_VERSION}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, just legacy.

local url="https://github.com/knative/serving-operator/releases/download/${LATEST_SERVING_RELEASE_VERSION}"
local yaml="serving-operator.yaml"

local RELEASE_YAML="$(mktemp)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should the var be lower case?
im assuming all local var using lower case for consistency

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

)

// KSOperatorCRVerifyStatus verifies if the KnativeServing can reach the READY status.
func KSOperatorCRVerifyStatus(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the method is fatal, consider renaming it to assert
or have it return an error.

)

// KSOperatorCRVerifyStatus verifies if the KnativeServing can reach the READY status.
func KSOperatorCRVerifyStatus(t *testing.T, clients *test.Clients, names test.ResourceNames) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please include Ready in the method name.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}

// CreateKnativeServing creates a KnativeServing with the name names.KnativeServing under the namespace names.Namespace.
func CreateKnativeServing(clients servingv1alpha1.KnativeServingInterface, names test.ResourceNames) (*v1alpha1.KnativeServing, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rename this to EnsureKnativeServingExists?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -1,7 +1,7 @@
// +build e2e

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this deployment test different than the post upgrade test?

Copy link
Author

@houshengbo houshengbo Feb 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commonality is that at the beginning we need to create and make sure the operator CR is ready, and in the end remove the operator CR.

The difference is that integration tests tagged with e2e will run the verification of the knative serving deployments including configuration and recreation. The tests tagged with postupgrade will verify what should be ready after upgrading the operator from the previous release, like whether we have exactly the same number and names of deployments as expected(like in eventing operator, we have got a new deployment and removed an old one. The post upgrade tests there verify if we have the correct list of deployment). Later, I will move all the existing postupgrade tests available in knative serving repo into this post upgrade test in operator, so that we make operator the default and optimal tool to upgrade.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next step, I will open PRs to port the post upgrade tests in serving here to operator.

@k4leung4
Copy link
Contributor

thanks for the change

/lgtm

@knative-prow-robot knative-prow-robot merged commit 7a1422c into knative:master Feb 11, 2020
houshengbo pushed a commit to houshengbo/serving-operator that referenced this pull request Feb 17, 2020
* Add upgrade tests into the e2e-tests-latest-serving

This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.

* Install the latest release of the operator and then upgrade

* Split the long functions and rename the common package

* Refactor the PR based on the comments
houshengbo pushed a commit to houshengbo/serving-operator that referenced this pull request Feb 17, 2020
* Add upgrade tests into the e2e-tests-latest-serving

This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.

* Install the latest release of the operator and then upgrade

* Split the long functions and rename the common package

* Refactor the PR based on the comments
houshengbo pushed a commit to houshengbo/serving-operator that referenced this pull request Feb 17, 2020
* Add upgrade tests into the e2e-tests-latest-serving

This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.

* Install the latest release of the operator and then upgrade

* Split the long functions and rename the common package

* Refactor the PR based on the comments
knative-prow-robot pushed a commit that referenced this pull request Feb 18, 2020
* Support reading KubeConfig from KUBECONFIG env var (#293)

* Add upgrade tests into the e2e-tests-latest-serving (#286)

* Add upgrade tests into the e2e-tests-latest-serving

This PR adds the tests to verify the correct number and names of knative serving
deployments.

The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.

* Install the latest release of the operator and then upgrade

* Split the long functions and rename the common package

* Refactor the PR based on the comments

* Update the library from jcrossley3/manifestival to manifestival

* Update pkg/reconciler/knativeserving/common/gateway.go

please fix

Co-Authored-By: Matt Moore <[email protected]>

Co-authored-by: Ali Ok <[email protected]>
Co-authored-by: Kenny Leung <[email protected]>
Co-authored-by: Matt Moore <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade tests to be added into this repo to test the upgrade of operator with knative serving

5 participants