refactor tensorify restart logic to use sources #140899

bobrenjc93 · 2024-11-17T05:18:00Z

Stack from ghstack (oldest at bottom):

cc @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv @voznesenskym @penguinwu @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @jiayisunx @chenyang78 @kadeng @chauhang @amjames

[ghstack-poisoned]

pytorch-bot · 2024-11-17T05:18:03Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140899

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

❌ 27 New Failures, 6 Unrelated Failures

As of commit ec65f60 with merge base c3fbec7 ():

NEW FAILURES - The following jobs have failed:

inductor / unit-test / cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_logging.py::LoggingTests::test_guards_recompiles
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_cpu
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_cpu
pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge) (gh)
test_dynamic_decoder
pull / linux-focal-py3.11-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-focal-py3.11-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
dynamo/test_structured_trace.py::StructuredTraceTest::test_guards_recompiles
pull / linux-focal-py3.11-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-focal-py3.12-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_decoupled_weight_decay_cpu
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_False_epilogue_leaky_relu_cpu_bfloat16_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, lf.linux.2xlarge) (gh)
inductor/test_cpu_cpp_wrapper.py::DynamicShapesCppWrapperCpuTests::test_linear_with_pointwise_batch_size_384_in_features_196_out_features_385_bias_True_epilogue_leaky_relu_cpu_bfloat16_dynamic_shapes_cpp_wrapper
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, lf.linux.2xlarge) (gh)
dynamo/test_structured_trace.py::StructuredTraceTest::test_guards_recompiles
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 3, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_weight_decay_cpu
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 4, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerTests::test_radam_cpu

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh) (disabled by #138585)
export/test_export_nonstrict.py::NonStrictExportTestExport::test_unbacked_bindings_for_divisible_u_symint_non_strict
pull / linux-focal-py3.11-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh) (disabled by #138582)
export/test_export_training_ir_to_run_decomp.py::TrainingIRToRunDecompExportNonStrictTestExport::test_unbacked_bindings_for_divisible_u_symint_training_ir_to_decomp_non_strict
pull / linux-focal-py3.12-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh) (disabled by #138582)
export/test_export_training_ir_to_run_decomp.py::TrainingIRToRunDecompExportNonStrictTestExport::test_unbacked_bindings_for_divisible_u_symint_training_ir_to_decomp_non_strict
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh) (disabled by #138582)
export/test_export_training_ir_to_run_decomp.py::TrainingIRToRunDecompExportNonStrictTestExport::test_unbacked_bindings_for_divisible_u_symint_training_ir_to_decomp_non_strict
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh) (disabled by #138585)
export/test_export_nonstrict.py::NonStrictExportTestExport::test_unbacked_bindings_for_divisible_u_symint_non_strict
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 5, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh) (disabled by #138585)
export/test_export_nonstrict.py::NonStrictExportTestExport::test_unbacked_bindings_for_divisible_u_symint_non_strict

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

refactor tensorify restart logic to use sources

96e2554

[ghstack-poisoned]

This was referenced Nov 17, 2024

Specialize symfloats during equality checks #140830

Closed

don't specialize when grad tracking tensors are activated #140828

Closed

bobrenjc93 mentioned this pull request Nov 17, 2024

Specialize symfloats when getting fake value involves complex args #140832

Closed

pytorch-bot bot added ciflow/inductor module: dynamo release notes: fx release notes category labels Nov 17, 2024

facebook-github-bot added the fx label Nov 17, 2024

Update on "refactor tensorify restart logic to use sources"

1b0390f

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

f3b9567

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 added ciflow/trunk Trigger trunk jobs on your pull request keep-going Don't stop on first failure, keep running tests until the end topic: not user facing topic category labels Nov 17, 2024

Update on "refactor tensorify restart logic to use sources"

4bc5870

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 added 3 commits November 16, 2024 21:44

Update on "refactor tensorify restart logic to use sources"

fecb46b

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

972ca2f

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

77b014a

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 mentioned this pull request Nov 17, 2024

Address PR comments (canary) #140904

Closed

Update on "refactor tensorify restart logic to use sources"

e3b51bb

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

This was referenced Nov 17, 2024

Clear speculation log on tensorify restart #140907

Closed

disable wrapping symfloats on xla #140908

Closed

bobrenjc93 added 2 commits November 17, 2024 17:31

Update on "refactor tensorify restart logic to use sources"

f531297

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

c01c0bd

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 mentioned this pull request Nov 18, 2024

disable tensorify floats when cuda graphs is on #140983

Closed

Update on "refactor tensorify restart logic to use sources"

926da94

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 mentioned this pull request Nov 19, 2024

Also check for attention mask shape for _sfdp_params_check #141003

Closed

bobrenjc93 added 7 commits November 19, 2024 10:45

Update on "refactor tensorify restart logic to use sources"

33524f3

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

a0b3d22

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

7de420a

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

79186cd

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

cc0b2f3

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

cba7c90

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Update on "refactor tensorify restart logic to use sources"

ec65f60

cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

bobrenjc93 closed this Nov 25, 2024

github-actions bot deleted the gh/bobrenjc93/128/head branch December 26, 2024 02:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor tensorify restart logic to use sources #140899

refactor tensorify restart logic to use sources #140899

Uh oh!

bobrenjc93 commented Nov 17, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 17, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

refactor tensorify restart logic to use sources #140899

refactor tensorify restart logic to use sources #140899

Uh oh!

Conversation

bobrenjc93 commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140899

❗ 1 Active SEVs

❌ 27 New Failures, 6 Unrelated Failures

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bobrenjc93 commented Nov 17, 2024 •

edited

Loading

pytorch-bot bot commented Nov 17, 2024 •

edited

Loading