-
Notifications
You must be signed in to change notification settings - Fork 44
Add upgrade tests into the e2e-tests-latest-serving #286
Conversation
knative-prow-robot
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@houshengbo: 0 warnings.
Details
In response to this:
This PR adds the tests to verify the correct number and names of knative serving
deployments.The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: houshengbo The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
9880fa8 to
e02ae0d
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
1 similar comment
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
0b14b09 to
0c21de8
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
0c21de8 to
a9f68ed
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
a9f68ed to
215ff57
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
08046ce to
dcd0b59
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
dcd0b59 to
36cd083
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
fb6ba5c to
6156800
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
6156800 to
76f83c1
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
76f83c1 to
f43163a
Compare
cc561ac to
43c6a5c
Compare
|
/test pull-knative-serving-operator-integration-tests-on-latest-serving |
43c6a5c to
c86cc25
Compare
test/common/common.go
Outdated
| import ( | ||
| "fmt" | ||
| "errors" | ||
| "k8s.io/apimachinery/pkg/util/wait" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we group the imports by
- go libs
- knative libs
- everything
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will align them in the correct order.
| @@ -0,0 +1,230 @@ | |||
| /* | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
common.go is overly generic.
can we break it out to
verify.go - to cover all the assert/verify helpers
setup.go - to cover all the setup/creation helpers?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will create a file called setup.go, and put it under the folder client.
The file verify.go can be put under the folder resources, since it verifies everything about the knative resources.
test/common/common.go
Outdated
|
|
||
|
|
||
| // KnativeServingVerify verifies if the KnativeServing can reach the READY status. | ||
| func KnativeServingVerify(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
im conflicted about the naming
i understand the CRD is called KnativeServing, but a method that says KnativeServingVerify is confusing for me.
maybe append CR at the end, as we are verifying the CR status.
thoughts?
test/common/common.go
Outdated
| // Get the existing KS without any spec | ||
| ks, err := clients.KnativeServing().Get(names.KnativeServing, metav1.GetOptions{}) | ||
| // Add config to its spec | ||
| ks.Spec = v1alpha1.KnativeServingSpec{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we pass in the configmap to this helper.
that way it can be reused for other changes
test/common/common.go
Outdated
| } | ||
|
|
||
| // KnativeServingConfigure verifies that KnativeServing config is set properly | ||
| func KnativeServingConfigure(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
debating if we need to prefix everything with KnativeServing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can use KS for short.
test/common/common.go
Outdated
| } | ||
|
|
||
| // DeploymentRecreation verify whether all the deployments for knative serving are able to recreate, when they are deleted. | ||
| func DeploymentRecreation(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would break this into separate methods
deleteDeployments
verifyDeployments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you can keep this method, but rename it DeleteAndVerifyDeployments, to make it clear all that the method is doing.
test/common/common.go
Outdated
| } | ||
|
|
||
| // KnativeServingDelete deletes tha KnativeServing to see if all resources will be deleted | ||
| func KnativeServingDelete(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above.
split this into smaller pieces.
| readonly LATEST_SERVING_RELEASE_VERSION=$(git describe --match "v[0-9]*" --abbrev=0) | ||
| # Istio version we test with | ||
| readonly ISTIO_VERSION=1.1.3 | ||
| readonly ISTIO_VERSION="1.4.2" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a way to determine this dynamically? the version should be in a file in third_party/?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1.4-latest is taken as default, if this param is not set.
HOWEVER, the directory is not available under the path, https://raw.githubusercontent.com/knative/serving, but it is available under the source code folder.
It only happens to this 1.X-latest directory, not to others like 1.4.2.
I have to set a specific version here to access via https://raw.githubusercontent.com/knative/serving/\<version>
| @@ -0,0 +1,230 @@ | |||
| /* | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would prefer not to use the "common" folder name, as typically ends up the dumping grounds for everything.
| t.Run("verify resources", func(t *testing.T) { | ||
| common.KnativeServingVerify(t, clients, names) | ||
| // TODO: We only verify the deployment, but we need to add other resources as well, like ServiceAccount, ClusterRoleBinding, etc. | ||
| ExpectedDeployments := []string{"networking-istio", "webhook", "controller", "activator", "autoscaler-hpa", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is this capitalized
This PR adds the tests to verify the correct number and names of knative serving deployments. The test tag postupgrade is added, marking the tests to run after upgrade to the latest HEAD of operator, with the latest generated manifest of knative serving.
c86cc25 to
84c835f
Compare
84c835f to
fde76b9
Compare
23b427b to
841a22f
Compare
841a22f to
5b38718
Compare
test/e2e-common.sh
Outdated
| local base_url="https://raw.githubusercontent.com/knative/serving/v${LATEST_SERVING_RELEASE_VERSION}" | ||
| local base_url="https://raw.githubusercontent.com/knative/serving/${LATEST_SERVING_RELEASE_VERSION}" | ||
| # Decide the Istio configuration to install. | ||
| if [[ -z "$ISTIO_VERSION" ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can this ever be false? isn't it always set and can't be overwritten?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will remove it.
test/e2e-upgrade-tests.sh
Outdated
| function knative_setup() { | ||
| function install_latest_operator_release() { | ||
| header "Installing Knative Serving operator latest public release" | ||
| local url="https://github.com/knative/serving-operator/releases/download/${LATEST_SERVING_RELEASE_VERSION}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any reason not to combine line 41 and 42 together?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, just legacy.
test/e2e-upgrade-tests.sh
Outdated
| local url="https://github.com/knative/serving-operator/releases/download/${LATEST_SERVING_RELEASE_VERSION}" | ||
| local yaml="serving-operator.yaml" | ||
|
|
||
| local RELEASE_YAML="$(mktemp)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should the var be lower case?
im assuming all local var using lower case for consistency
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
test/resources/verify.go
Outdated
| ) | ||
|
|
||
| // KSOperatorCRVerifyStatus verifies if the KnativeServing can reach the READY status. | ||
| func KSOperatorCRVerifyStatus(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the method is fatal, consider renaming it to assert
or have it return an error.
test/resources/verify.go
Outdated
| ) | ||
|
|
||
| // KSOperatorCRVerifyStatus verifies if the KnativeServing can reach the READY status. | ||
| func KSOperatorCRVerifyStatus(t *testing.T, clients *test.Clients, names test.ResourceNames) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please include Ready in the method name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
test/resources/knativeserving.go
Outdated
| } | ||
|
|
||
| // CreateKnativeServing creates a KnativeServing with the name names.KnativeServing under the namespace names.Namespace. | ||
| func CreateKnativeServing(clients servingv1alpha1.KnativeServingInterface, names test.ResourceNames) (*v1alpha1.KnativeServing, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe rename this to EnsureKnativeServingExists?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
| @@ -1,7 +1,7 @@ | |||
| // +build e2e | |||
|
|
|||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how is this deployment test different than the post upgrade test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The commonality is that at the beginning we need to create and make sure the operator CR is ready, and in the end remove the operator CR.
The difference is that integration tests tagged with e2e will run the verification of the knative serving deployments including configuration and recreation. The tests tagged with postupgrade will verify what should be ready after upgrading the operator from the previous release, like whether we have exactly the same number and names of deployments as expected(like in eventing operator, we have got a new deployment and removed an old one. The post upgrade tests there verify if we have the correct list of deployment). Later, I will move all the existing postupgrade tests available in knative serving repo into this post upgrade test in operator, so that we make operator the default and optimal tool to upgrade.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Next step, I will open PRs to port the post upgrade tests in serving here to operator.
|
thanks for the change /lgtm |
* Add upgrade tests into the e2e-tests-latest-serving This PR adds the tests to verify the correct number and names of knative serving deployments. The test tag postupgrade is added, marking the tests to run after upgrade to the latest HEAD of operator, with the latest generated manifest of knative serving. * Install the latest release of the operator and then upgrade * Split the long functions and rename the common package * Refactor the PR based on the comments
* Add upgrade tests into the e2e-tests-latest-serving This PR adds the tests to verify the correct number and names of knative serving deployments. The test tag postupgrade is added, marking the tests to run after upgrade to the latest HEAD of operator, with the latest generated manifest of knative serving. * Install the latest release of the operator and then upgrade * Split the long functions and rename the common package * Refactor the PR based on the comments
* Add upgrade tests into the e2e-tests-latest-serving This PR adds the tests to verify the correct number and names of knative serving deployments. The test tag postupgrade is added, marking the tests to run after upgrade to the latest HEAD of operator, with the latest generated manifest of knative serving. * Install the latest release of the operator and then upgrade * Split the long functions and rename the common package * Refactor the PR based on the comments
* Support reading KubeConfig from KUBECONFIG env var (#293) * Add upgrade tests into the e2e-tests-latest-serving (#286) * Add upgrade tests into the e2e-tests-latest-serving This PR adds the tests to verify the correct number and names of knative serving deployments. The test tag postupgrade is added, marking the tests to run after upgrade to the latest HEAD of operator, with the latest generated manifest of knative serving. * Install the latest release of the operator and then upgrade * Split the long functions and rename the common package * Refactor the PR based on the comments * Update the library from jcrossley3/manifestival to manifestival * Update pkg/reconciler/knativeserving/common/gateway.go please fix Co-Authored-By: Matt Moore <[email protected]> Co-authored-by: Ali Ok <[email protected]> Co-authored-by: Kenny Leung <[email protected]> Co-authored-by: Matt Moore <[email protected]>
Fixes #287
Proposed Changes
This PR adds the tests to verify the correct number and names of knative serving
deployments.
The test tag postupgrade is added, marking the tests to run after upgrade to the
latest HEAD of operator, with the latest generated manifest of knative serving.
Release Note