Skip to content

Add dynamic sharding for Github Actions workflows #59401

@seemethere

Description

@seemethere

We have the concept of sharding of tests for specific build configurations within CircleCI and we'd like to have the same concept within Github Actions.

For sharding I can envision us enabling sharding with Github Actions by utilizing dynamic matrix values to create multiple test jobs that will act as shards. An example of dynamic matrix values can be found here:

generate-build-matrix:
if: ${{ github.repository_owner == 'pytorch' }}
runs-on: ubuntu-18.04
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
container:
image: python:3.9
steps:
- name: Clone pytorch/pytorch
uses: actions/checkout@v2
- name: Generating build matrix
id: set-matrix
run: |
# outputting for debugging purposes
MATRIX=$(python .github/scripts/generate_binary_build_matrix.py conda)
echo "${MATRIX}"
echo "::set-output name=matrix::${MATRIX}"
build-conda:
if: ${{ github.repository_owner == 'pytorch' }}
needs: generate-build-matrix
runs-on: linux.2xlarge
strategy:
matrix: ${{ fromJson(needs.generate-build-matrix.outputs.matrix) }}
fail-fast: false

How would this work?

  1. Create a job as depenendcy to the test job that generates a json blob (probably through a python script)
  2. Use that json blob as an input into the test job as a matrix value

cc @ezyang @seemethere @malfet @walterddr @pytorch/pytorch-dev-infra

Metadata

Metadata

Assignees

Labels

module: ciRelated to continuous integrationtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions