Introduce test-scale script by siggy · Pull Request #2578 · linkerd/linkerd2

siggy · 2019-03-28T01:25:57Z

Introduce a bin/test-scale script to deploy Linkerd alongside sample
apps at scale.

This script deploys the following:

Linkerd control-plane, with service profiles
5 namespaces x 5 replicas of each:
- Emojivoto demo app
- Books demo app, with service profiles
- Lifecycle / bb test environment

Signed-off-by: Andrew Seigner [email protected]

l5d-bot · 2019-03-28T01:37:11Z

Integration test results for 3d3851e: success 🎉
Log output: https://gist.github.com/26616d46fb431d0b4481445414dcfd13

l5d-bot · 2019-03-28T07:02:55Z

Integration test results for fb8a7eb: success 🎉
Log output: https://gist.github.com/23f41ac2b744fb75c47f300305a2806c

l5d-bot · 2019-03-28T19:53:21Z

Integration test results for d071c8f: success 🎉
Log output: https://gist.github.com/f2cdc6099c9c014151485ac3ea779fc5

harsh-98 · 2019-03-29T07:02:30Z

bin/test-scale

+BOOKS_APP=$(curl -s https://raw.githubusercontent.com/BuoyantIO/booksapp/master/k8s/mysql-app.yml)
+
+# add "-sleep=10ms" param to the traffic app (~100rps)
+traffic_param="        - \"-webapp:7000\""


In https://raw.githubusercontent.com/BuoyantIO/booksapp/master/k8s/mysql-app.yml, "webapp:7000" is given instead of "-webapp:7000". Therefore, the value $traffic_param is not substituted by $sleep_param. Also in sleep parameter extra - should be removed.

@harsh-98 Nice catch, thanks!

l5d-bot · 2019-03-29T07:46:33Z

Integration test results for 1cfcc39: success 🎉
Log output: https://gist.github.com/218a4a3f890a9fa5156c9b6c220c75e6

Introduce a `bin/test-scale` script to deploy Linkerd alongside sample apps at scale. This script deploys the following: - Linkerd control-plane, with service profiles - 5 namespaces x 5 replicas of each: - Emojivoto demo app - Books demo app, with service profiles - Lifecycle / bb test environment Fixes #2517 Signed-off-by: Andrew Seigner <[email protected]>

Signed-off-by: Andrew Seigner <[email protected]>

l5d-bot · 2019-03-30T00:11:30Z

Integration test results for 0e6b00c: success 🎉
Log output: https://gist.github.com/5318eafb9afd1a8709b056ed8b7df177

alpeb

Code looks good 👍
My Minikube is choking on it, but I'll let you know if I can have it work.
Any chance of adding these namespaces into bin/test-cleanup?

alpeb · 2019-04-01T14:32:51Z

bin/test-scale

+if [ -z "$linkerd_path" ]; then
+    echo "usage: $(basename "$0") /path/to/linkerd [namespace]" >&2
+    exit 64
+fi


Because of set -eu, if no args args are provided to the script it fails at line 50 without a chance to output the usage message.

alpeb · 2019-04-01T16:45:49Z

I could make it work for 3 namespaces and 3 replicas

But the lifecycle namespaces stop showing stats after the first redeployment. This is the log from the redeployer pod:

redeploying bb-terminus deployment
error: SchemaError(io.k8s.api.apps.v1beta2.DaemonSetUpdateStrategy): invalid object doesn't have additional properties
sleeping for 60 seconds...

Signed-off-by: Andrew Seigner <[email protected]>

siggy · 2019-04-01T17:24:33Z

@alpeb Re: the cleanup script, I opted to document a bin/test-cleanup l5d-scale command:
https://github.com/linkerd/linkerd2/pull/2578/files#diff-6bbf111168957115fe11f1cc2a6d0277R347

That script already defaults to l5d-integration to accomodate cleaning up the integration tests, and I didn't want to use the same default namespaces for the scale tests. Another option would be to create a bin/test-scale-cleanup script that wraps bin/test-cleanup with a new default, but I was trying to avoid adding yet another script into bin. :)

Re: that redeployer issue, I'm not really sure what's going on. I'm surprised to see an error related to DaemonSetUpdateStrategy, as everything in lifecycle is a deployment. What kind of k8s cluster is this? I've been testing with AKS and Docker For Mac.

For reference, here's the relevant part of that script:
https://github.com/linkerd/linkerd-examples/blob/master/lifecycle/lifecycle.yml#L383-L390
...and the expected output from redeployer:

$ kubectl -n l5d-scale-lifecycle-1 logs -f deploy/redeployer redeployer
found 2 running pods
pod "bb-terminus-78669d6ccd-2sj9p" deleted
sleeping for 30 seconds...
pod "bb-terminus-78669d6ccd-xkt4x" deleted
bouncing services...
service "bb-terminus1" deleted
service "bb-terminus1" created
service "bb-terminus2" deleted
service "bb-terminus2" created
service "bb-terminus3" deleted
service "bb-terminus3" created
service "bb-terminus4" deleted
service "bb-terminus4" created
service "bb-terminus5" deleted
service "bb-terminus5" created
service "bb-terminus6" deleted
service "bb-terminus6" created
service "bb-terminus7" deleted
service "bb-terminus7" created
service "bb-terminus8" deleted
service "bb-terminus8" created
service "bb-terminus9" deleted
service "bb-terminus9" created
service "bb-terminus10" deleted
service "bb-terminus10" created
sleeping for 30 seconds...
redeploying bb-terminus deployment
deployment.extensions "bb-terminus" configured
sleeping for 60 seconds...
found 2 running pods

l5d-bot · 2019-04-01T17:30:11Z

Integration test results for f789164: success 🎉
Log output: https://gist.github.com/1c0663b9ec95cb38f248843bb4b83c91

alpeb · 2019-04-01T17:31:24Z

Ok I think bin/test-cleanup l5d-scale is enough 👍

Regarding that daemonset error it appears it's some sort of RBAC error despite the error message. I'm testing with Minikube on Linux. Still digging...

alpeb

It all worked fine after upgrading the lachlanevenson/k8s-kubectl image version tag 👍

Introduce a `bin/test-scale` script to deploy Linkerd alongside sample apps at scale. This script deploys the following: - Linkerd control-plane, with service profiles - 5 namespaces x 5 replicas of each: - Emojivoto demo app - Books demo app, with service profiles - Lifecycle / bb test environment Fixes linkerd#2517 Signed-off-by: Andrew Seigner <[email protected]> Signed-off-by: [email protected] <[email protected]>

siggy added the area/test label Mar 28, 2019

siggy self-assigned this Mar 28, 2019

siggy force-pushed the siggy/scale-test branch from 3d3851e to fb8a7eb Compare March 28, 2019 06:51

admc mentioned this pull request Mar 28, 2019

Linkerd 2.3 release test plan #2459

Closed

25 tasks

siggy added the priority/P0 Release Blocker label Mar 28, 2019

siggy mentioned this pull request Mar 28, 2019

Proxy restarted due to readiness/liveness probe failure #2585

Closed

siggy force-pushed the siggy/scale-test branch from fb8a7eb to d071c8f Compare March 28, 2019 19:46

harsh-98 reviewed Mar 29, 2019

View reviewed changes

siggy added 2 commits March 29, 2019 16:48

fix sleep param for books traffic

0e6b00c

Signed-off-by: Andrew Seigner <[email protected]>

siggy force-pushed the siggy/scale-test branch from 1cfcc39 to 0e6b00c Compare March 29, 2019 23:48

alpeb reviewed Apr 1, 2019

View reviewed changes

fix unbound variable error

f789164

Signed-off-by: Andrew Seigner <[email protected]>

alpeb approved these changes Apr 1, 2019

View reviewed changes

siggy merged commit 38f504b into master Apr 1, 2019

siggy deleted the siggy/scale-test branch April 1, 2019 19:52

Conversation

siggy commented Mar 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l5d-bot commented Mar 28, 2019

Uh oh!

l5d-bot commented Mar 28, 2019

Uh oh!

l5d-bot commented Mar 28, 2019

Uh oh!

harsh-98 Mar 29, 2019

Choose a reason for hiding this comment

Uh oh!

siggy Mar 29, 2019

Choose a reason for hiding this comment

Uh oh!

l5d-bot commented Mar 29, 2019

Uh oh!

l5d-bot commented Mar 30, 2019

Uh oh!

alpeb left a comment

Choose a reason for hiding this comment

Uh oh!

alpeb Apr 1, 2019

Choose a reason for hiding this comment

Uh oh!

alpeb commented Apr 1, 2019

Uh oh!

siggy commented Apr 1, 2019

Uh oh!

l5d-bot commented Apr 1, 2019

Uh oh!

alpeb commented Apr 1, 2019

Uh oh!

alpeb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

siggy commented Mar 28, 2019 •

edited

Loading