Current Status
11:00 PST - Detected
11:20 PST - New CI traffic moved to LF: pytorch/test-infra#5132
11:35 PST - Removed merge-blocking
12:29 PST - Errors recovered, waiting to see if scale-up-chron will recover the queued runners
14:30ish PST - Queue time started to recover, now waiting for the scale to pick up queued jobs
16:11 PST - closing as queue time recovered
Error looks like
Queued runners:
Evaluating manywheel-py3_11-cuda12_6-test.if
Evaluating: (success() && ((github.repository_owner == 'pytorch')))
Expanded: (true && ('pytorch' == 'pytorch'))
Result: true
Evaluating manywheel-py3_11-cuda12_6-test.test.if
Evaluating: success()
Result: true
Requested labels: linux.4xlarge.nvidia.gpu
Job defined at: pytorch/pytorch/.github/workflows/_binary-test-linux.yml@refs/tags/ciflow/binaries_libtorch/175894
Reusable workflow chain:
pytorch/pytorch/.github/workflows/generated-linux-binary-manywheel-nightly.yml@refs/tags/ciflow/binaries_libtorch/175894 (1288e1b2f16d918eeb47bd386bfeaf09b15f84ad)
-> pytorch/pytorch/.github/workflows/_binary-test-linux.yml@refs/tags/ciflow/binaries_libtorch/175894 (1288e1b2f16d918eeb47bd386bfeaf09b15f84ad)
Waiting for a runner to pick up this job...
User impact
How does this affect users of PyTorch CI?
Root cause
Meta's internal
Mitigation
- Moved traffic to LF;
- You should be able to run your CI by canceling and re-trying the WORKFLOW (not only jobs);
Prevention/followups
How do we prevent issues like this in the future?
cc @seemethere @malfet @pytorch/pytorch-dev-infra
Current Status
11:00 PST - Detected
11:20 PST - New CI traffic moved to LF: pytorch/test-infra#5132
11:35 PST - Removed merge-blocking
12:29 PST - Errors recovered, waiting to see if scale-up-chron will recover the queued runners
14:30ish PST - Queue time started to recover, now waiting for the scale to pick up queued jobs
16:11 PST - closing as queue time recovered
Error looks like
Queued runners:
User impact
How does this affect users of PyTorch CI?
Root cause
Meta's internal
Mitigation
Prevention/followups
How do we prevent issues like this in the future?
cc @seemethere @malfet @pytorch/pytorch-dev-infra