Clear speculation log on tensorify restart #140907

bobrenjc93 · 2024-11-17T17:58:48Z

Stack from ghstack (oldest at bottom):

Extracted out from other PR since it seems to break tlp python test/inductor/test_compiled_optimizers.py CompiledOptimizerParityTestsCUDA.test_correctness_Adafactor_use_closure_False_cuda_float32 for some reason...

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames

This reverts commit 08f8d59274015aff4513f4b8f946bc287b4cc70f. [ghstack-poisoned]

pytorch-bot · 2024-11-17T17:58:52Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140907

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

❌ 27 New Failures, 7 Unrelated Failures

As of commit 66e63de with merge base c3fbec7 ():

NEW FAILURES - The following jobs have failed:

inductor / unit-test / cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True
inductor / unit-test / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
'Test'
inductor / unit-test / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
'Test'
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_utils.py::TestDynamoTimed::test_dynamo_timed
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_cpu
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_Adafactor_use_closure_False_cuda_float32
pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge) (gh)
test_resnet18
pull / linux-focal-py3.11-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-py3.11-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-py3.12-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, lf.linux.2xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-focal-py3.11-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-focal-py3.12-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-focal-py3.9-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, lf.linux.2xlarge) (gh) (disabled by #138584)
export/test_retraceability.py::RetraceExportTestExport::test_unbacked_bindings_for_divisible_u_symint_retraceability

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This reverts commit 08f8d59274015aff4513f4b8f946bc287b4cc70f. ghstack-source-id: b6aaedb Pull Request resolved: #140907

Extracted out from other PR since it seems to break `tlp python test/inductor/test_compiled_optimizers.py CompiledOptimizerParityTestsCUDA.test_correctness_Adafactor_use_closure_False_cuda_float32` for some reason... cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Clear speculation log on tensorify restart

ec3d395

This reverts commit 08f8d59274015aff4513f4b8f946bc287b4cc70f. [ghstack-poisoned]

This was referenced Nov 17, 2024

Specialize symfloats when getting fake value involves complex args #140832

Closed

don't specialize when grad tracking tensors are activated #140828

Closed

bobrenjc93 mentioned this pull request Nov 17, 2024

Add trunc to z3 validator #140886

Closed

pytorch-bot bot added ciflow/inductor module: dynamo labels Nov 17, 2024

bobrenjc93 added a commit that referenced this pull request Nov 17, 2024

Clear speculation log on tensorify restart

8eecf38

This reverts commit 08f8d59274015aff4513f4b8f946bc287b4cc70f. ghstack-source-id: b6aaedb Pull Request resolved: #140907

bobrenjc93 added topic: not user facing topic category keep-going Don't stop on first failure, keep running tests until the end labels Nov 17, 2024

bobrenjc93 mentioned this pull request Nov 17, 2024

disable wrapping symfloats on xla #140908

Closed

bobrenjc93 added 2 commits November 17, 2024 17:31

bobrenjc93 mentioned this pull request Nov 18, 2024

disable tensorify floats when cuda graphs is on #140983

Closed

bobrenjc93 mentioned this pull request Nov 19, 2024

Also check for attention mask shape for _sfdp_params_check #141003

Closed

bobrenjc93 added 7 commits November 19, 2024 10:45

bobrenjc93 closed this Nov 25, 2024

github-actions bot deleted the gh/bobrenjc93/130/head branch December 26, 2024 02:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Clear speculation log on tensorify restart #140907

Clear speculation log on tensorify restart #140907

Uh oh!

bobrenjc93 commented Nov 17, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 17, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Clear speculation log on tensorify restart #140907

Clear speculation log on tensorify restart #140907

Uh oh!

Conversation

bobrenjc93 commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140907

❗ 1 Active SEVs

❌ 27 New Failures, 7 Unrelated Failures

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bobrenjc93 commented Nov 17, 2024 •

edited

Loading

pytorch-bot bot commented Nov 17, 2024 •

edited

Loading