-
Notifications
You must be signed in to change notification settings - Fork 26.3k
Closed
Labels
module: ciRelated to continuous integrationRelated to continuous integrationtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate moduleThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
Description
We have the concept of sharding of tests for specific build configurations within CircleCI and we'd like to have the same concept within Github Actions.
For sharding I can envision us enabling sharding with Github Actions by utilizing dynamic matrix values to create multiple test jobs that will act as shards. An example of dynamic matrix values can be found here:
pytorch/.github/workflows/build_linux_conda.yml
Lines 10 to 33 in e66015d
| generate-build-matrix: | |
| if: ${{ github.repository_owner == 'pytorch' }} | |
| runs-on: ubuntu-18.04 | |
| outputs: | |
| matrix: ${{ steps.set-matrix.outputs.matrix }} | |
| container: | |
| image: python:3.9 | |
| steps: | |
| - name: Clone pytorch/pytorch | |
| uses: actions/checkout@v2 | |
| - name: Generating build matrix | |
| id: set-matrix | |
| run: | | |
| # outputting for debugging purposes | |
| MATRIX=$(python .github/scripts/generate_binary_build_matrix.py conda) | |
| echo "${MATRIX}" | |
| echo "::set-output name=matrix::${MATRIX}" | |
| build-conda: | |
| if: ${{ github.repository_owner == 'pytorch' }} | |
| needs: generate-build-matrix | |
| runs-on: linux.2xlarge | |
| strategy: | |
| matrix: ${{ fromJson(needs.generate-build-matrix.outputs.matrix) }} | |
| fail-fast: false |
How would this work?
- Create a job as depenendcy to the
testjob that generates a json blob (probably through a python script) - Use that json blob as an input into the
testjob as a matrix value
cc @ezyang @seemethere @malfet @walterddr @pytorch/pytorch-dev-infra
janeyx99 and samestep
Metadata
Metadata
Assignees
Labels
module: ciRelated to continuous integrationRelated to continuous integrationtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate moduleThis issue has been looked at a team member, and triaged and prioritized into an appropriate module