Gitlab + Flutter CI: Testing very slow #100751
Description
I currently have a Flutter project with over 700 tests which has turned my CI from 15min to almost an hour in the last couple months of development. I've been trying to find ways to reduce the time, but comparing the test times between CI and local, I think there might be an issue with either Gitlab or Flutter (so raising issues with both).
LOCAL
PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test --coverage
04:04 +707: All tests passed!
GITLAB CI
$ flutter test --coverage
45:30 +707: All tests passed!
It is definitely the test phase of the pipeline that is causing issues as when I look at each jobs time there is a noticeable difference:
code_quality - 00:05:24
test - 00:48:47
coverage - 00:02:06 (82.2%)
semantic-version - 00:01:20
additionally, I've tried removing the coverage for certain pipelines to reduce the time which has helped a little but there is still a noticeable difference between local and CI:
LOCAL
PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test
01:34 +707: All tests passed!
GITLAB CI
$ flutter test
08:31 +707: All tests passed!
I've also run the tests in verbose which show nothing out of the ordinary apart from very slow times.
raw.txt
and finally, I've looked at the report.xml in Gitlab and can see that while each test takes ~2sec, the total time is only around 2min despite the CI taking well over 40min.
below is the .gitlab-ci.yml
stages: # List of stages for jobs, and their order of execution
- analyze
- test
- coverage
- semantic-version
default:
image: cirrusci/flutter:latest
cache:
paths:
- /flutter/bin/cache/dart-sdk
code_quality:
stage: analyze
before_script:
- pub global activate dart_code_metrics
- export PATH="$PATH":"$HOME/.pub-cache/bin"
script:
- flutter --version
- flutter analyze
- metrics lib -r codeclimate > gl-code-quality-report.json
rules:
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
allow_failure: true
artifacts:
reports:
codequality: gl-code-quality-report.json
test-all:
stage: test
script:
- export PATH="$PATH":"$HOME/.pub-cache/bin"
- pub global activate junitreport
- flutter test --machine --coverage | tojunit -o report.xml
only:
refs:
- main
- develop
artifacts:
paths:
- coverage
reports:
junit: report.xml
test-dev:
stage: test
script:
- export PATH="$PATH":"$HOME/.pub-cache/bin"
- pub global activate junitreport
- flutter test --machine | tojunit -o report.xml
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
artifacts:
reports:
junit: report.xml
coverage: # This job runs in the test stage.
stage: coverage # It only starts when the job in the build stage completes successfully.
script:
- lcov --summary coverage/lcov.info
- lcov --remove coverage/lcov.info
"lib/config/*"
"lib/application/l10n/l10n.dart"
"lib/application/l10n/**/*"
"lib/domain/repositories/*"
"lib/injection.config.dart"
"lib/presentation/routes/*"
"lib/infrastructure/repositories/firebase_injectable_module.dart"
"**/mock_*.dart"
"**/*.g.dart"
"**/*.gr.dart"
"**/*.freezed.dart"
"**/*.mocks.dart"
"**/*.config.dart"
-o coverage/clean_lcov.info
- genhtml coverage/clean_lcov.info --output=coverage
- curl -Os https://uploader.codecov.io/latest/linux/codecov
- chmod +x codecov
- ./codecov -t $CODECOV_TOKEN
- mv coverage/ public/
only:
refs:
- main
- develop
coverage: '/lines\.*: \d+\.\d+\%/'
artifacts:
paths:
- public
semantic-version:
image: node:16
stage: semantic-version
only:
refs:
- main
- develop
script:
- touch CHANGELOG.md
- npm install @semantic-release/gitlab @semantic-release/changelog
- npx semantic-release
artifacts:
paths:
- CHANGELOG.md