Skip to content

CI: Introduce flaky test finder #38523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thaJeztah merged 1 commit into from
Jan 12, 2019
+30 −0
Merged

CI: Introduce flaky test finder #38523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,9 @@ test-integration-cli: test-integration ## (DEPRECATED) use test-integration
test-integration: build ## run the integration tests
$(DOCKER_RUN_DOCKER) hack/make.sh dynbinary test-integration

test-integration-flaky: build ## run the stress test for all new integration tests
$(DOCKER_RUN_DOCKER) hack/make.sh dynbinary test-integration-flaky

test-unit: build ## run the unit tests
$(DOCKER_RUN_DOCKER) hack/test/unit

Expand Down
1 change: 1 addition & 0 deletions hack/ci/janky
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,6 @@ hack/make.sh \
binary-daemon \
dynbinary \
test-docker-py \
test-integration-flaky \
test-integration \
cross
26 changes: 26 additions & 0 deletions hack/make/test-integration-flaky
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
set -e -o pipefail

source hack/validate/.validate
new_tests=$(
validate_diff --diff-filter=ACMR --unified=0 -- 'integration/*_test.go' |
grep -E '^(\+func )(.*)(\*testing)' || true
)

if [ -z "$new_tests" ]; then
echo 'No new tests added to integration.'
return
fi

echo
echo "Found new integrations tests:"
echo "$new_tests"
echo "Running stress test for them."

(
TESTARRAY=$(echo "$new_tests" | sed 's/+func //' | awk -F'\\(' '{print $1}' | tr '\n' '|')
export TESTFLAGS="-test.count 5 -test.run ${TESTARRAY%?}"
export TEST_REPEAT=5
Copy link
Contributor

@tiborvass tiborvass Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@olljanat @thaJeztah why set both -test.count 5 and TEST_REPEAT 5 ? That will run 25 times. Also, if you do set -test.count you need to export TIMEOUT=... (which will set -test.timeout) to 5 times the existing TIMEOUT.

Copy link
Contributor Author

@olljanat olljanat Feb 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 runs is not enough to find example known flaky test TestCreateServiceSecretFileMode. Look comments above when I tested to run this to existing tests.

Difference between this and just run 25 times is this will restart daemon after every 5 test runs.

Increasing timeout of course can be done if needed.

Btw. Did you saw this timeouting on some PR now?

Copy link
Contributor

@tiborvass tiborvass Feb 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@olljanat yes on our internal EE fork. I'm fixing the bug in #38693

echo "Using test flags: $TESTFLAGS"
source hack/make/test-integration
)