Fix dependency on filesystem on Linux #137209

atalman · 2024-10-02T19:36:20Z

Similar to: #134494
We are seeing come back of #133437 due to use of filesystem on Linux

pytorch-bot · 2024-10-02T19:36:24Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137209

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 Cancelled Jobs, 1 Unrelated Failure

As of commit 01d8bea with merge base 8962610 ():

CANCELLED JOBS - The following jobs were cancelled. Please retry:

linux-s390x-binary-manywheel / manywheel-py3_10-cpu-s390x-build / build (gh)
##[error]The operation was canceled.
linux-s390x-binary-manywheel / manywheel-py3_13-cpu-s390x-build / build (gh)
##[error]The operation was canceled.
linux-s390x-binary-manywheel / manywheel-py3_9-cpu-s390x-build / build (gh)
##[error]The operation was canceled.
windows-binary-wheel / wheel-py3_10-cuda12_1-build (gh)
windows-binary-wheel / wheel-py3_12-cuda12_4-build (gh)

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh) (disabled by #134600, #132861, #132862)
test_transformers.py::TestSDPAPrivateUse1Only::test_fused_sdp_choice_privateuseone

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kit1980 · 2024-10-02T19:38:14Z

torch/csrc/inductor/aoti_torch/shim_common.cpp

+  if (getcwd(currentPath, sizeof(currentPath)) != nullptr) {
+    return std::string(currentPath);
+  } else {
+    throw std::runtime_error("Failed to get current path");


When this is possible?

When your current working directory is deleted for example

See https://godbolt.org/z/4on5Y6nsG

malfet · 2024-10-02T20:09:05Z

torch/csrc/inductor/aoti_torch/shim_common.cpp

-#if __has_include("filesystem")
-  return fs::current_path();
+std::string get_current_path() {
+#ifdef _WIN32


I would change it to #if __has_include("filesystem") && !defined(__LINUX__) as we want to use those on both Windows and Mac, where we don't have this problem

malfet · 2024-10-02T20:09:24Z

torch/csrc/inductor/aoti_torch/shim_common.cpp


 bool create_directories(const std::string& path) {
-#if __has_include("filesystem")
+#ifdef _WIN32


Same as above, don't restrict it to windows and check that CXX11 ABI is present

atalman · 2024-10-02T22:28:56Z

@pytorchmergebot merge

pytorchmergebot · 2024-10-02T22:30:40Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-02T22:35:57Z

Merge failed

Reason: 2 jobs have failed, first few of them are: windows-binary-wheel / wheel-py3_10-cuda12_1-build, windows-binary-wheel / wheel-py3_12-cuda12_4-build

Details for Dev Infra team

Raised by workflow job

atalman · 2024-10-03T00:16:46Z

@pytorchmergebot merge -f "all required jobs are passing"

pytorchmergebot · 2024-10-03T00:18:17Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

kit1980 · 2024-10-03T00:21:39Z

@pytorchbot cherry-pick -c critical --onto release/2.5

pytorchbot · 2024-10-03T00:25:20Z

Cherry picking #137209

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 6241006c28ddc962e11dfd8e7185f4a02ad651d0 returned non-zero exit code 1

Auto-merging torch/csrc/inductor/aoti_torch/shim_common.cpp
CONFLICT (content): Merge conflict in torch/csrc/inductor/aoti_torch/shim_common.cpp
error: could not apply 6241006c28... Fix dependency on filesystem on Linux (#137209)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

kit1980 · 2024-10-03T00:46:30Z

Manually cherry-picking into 2.5 - #137242

* Apply changes from #135374 * Fix dependency on filesystem on Linux (#137209) Similar to: #134494 We are seeing come back of #133437 due to use of filesystem on Linux Pull Request resolved: #137209 Approved by: https://github.com/kit1980, https://github.com/malfet --------- Co-authored-by: atalman <[email protected]>

Summary: Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally Note: Seems like there'll be issues with using`filesystem` header in linux: pytorch#137209 So here I use string manipulation instead of `filesystem::path` Test Plan: ``` buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r torchbind_aoti buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r aot_compile buck2 build --flagfile fbsource//arvr/mode/win/dev fbsource//arvr/libraries/audio/AudioSDK/Research/Source/BSI/UnitTests:meridian_bsi_tests buck2 build --flagfile fbsource//arvr/mode/embedded/linux/clang-aarch64-release fbsource//arvr/third-party/signalarity/latest-release/src/shaker/ha:signalarity_realtime_libtorch_pipeline_ ``` Differential Revision: D69500038

Summary: Pull Request resolved: pytorch#149747 Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally Note: Seems like there'll be issues with using`filesystem` header in linux: pytorch#137209 So here I use string manipulation instead of `filesystem::path` Test Plan: ``` buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r torchbind_aoti buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r aot_compile buck2 build --flagfile fbsource//arvr/mode/win/dev fbsource//arvr/libraries/audio/AudioSDK/Research/Source/BSI/UnitTests:meridian_bsi_tests buck2 build --flagfile fbsource//arvr/mode/embedded/linux/clang-aarch64-release fbsource//arvr/third-party/signalarity/latest-release/src/shaker/ha:signalarity_realtime_libtorch_pipeline_ ``` Differential Revision: D69500038

Summary: Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method D70746626 - Support None output type Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally (more details in internal Diff summary). Note on using `filesystem`: Seems like there'll be [issues](pytorch#137209) with using`filesystem` header in linux, so here I use string manipulation instead of `filesystem::path`. Test Plan: ``` buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r torchbind_aoti buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r aot_compile buck2 build --flagfile fbsource//arvr/mode/win/dev fbsource//arvr/libraries/audio/AudioSDK/Research/Source/BSI/UnitTests:meridian_bsi_tests buck2 build --flagfile fbsource//arvr/mode/embedded/linux/clang-aarch64-release fbsource//arvr/third-party/signalarity/latest-release/src/shaker/ha:signalarity_realtime_libtorch_pipeline_ ``` Reviewed By: desertfire Differential Revision: D69500038

Summary: The original Diff D69500038 is reverted due to a false alarm on trunk health. Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method D70746626 - Support None output type Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally (more details in internal Diff summary). Note on using `filesystem`: Seems like there'll be [issues](pytorch#137209) with using`filesystem` header in linux, so here I use string manipulation instead of `filesystem::path`. Test Plan: ``` buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r torchbind_aoti buck run fbcode//mode/dev-nosan //caffe2/test/inductor:torchbind -- -r aot_compile buck2 build --flagfile fbsource//arvr/mode/win/dev fbsource//arvr/libraries/audio/AudioSDK/Research/Source/BSI/UnitTests:meridian_bsi_tests buck2 build --flagfile fbsource//arvr/mode/embedded/linux/clang-aarch64-release fbsource//arvr/third-party/signalarity/latest-release/src/shaker/ha:signalarity_realtime_libtorch_pipeline_ ``` Differential Revision: D72063691

Summary: The original Diff D69500038 is reverted due to a false alarm on trunk health. Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method D70746626 - Support None output type Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally (more details in internal Diff summary). Note on using `filesystem`: Seems like there'll be [issues](#137209) with using`filesystem` header in linux, so here I use string manipulation instead of `filesystem::path`. Test Plan: ``` test/inductor:torchbind -- -r torchbind_aoti test/inductor:torchbind -- -r aot_compile ``` Differential Revision: D72063691 Pull Request resolved: #150196 Approved by: https://github.com/hl475, https://github.com/desertfire

Summary: The original Diff D69500038 is reverted due to a false alarm on trunk health. Implement torchbind support in OSSProxyExecutor. Exactly the same as the implementation in FbProxyExecutor. D69693697 - fbProxyExecutor D69887230 - fbProxyExecutor but for torchbind method D70746626 - Support None output type Other changes: - When generating the schema of the CallTrochBind HOP, the arg name of the torchbind object arg should be the same as the torchbind method's torchbind object arg (instead of `obj`). - In `AOTIModelPackageLoader`, we extract everything in `data/constants` to `tmp_dir/data/aot_inductor/<model>/` folder, so the torchbind objs exist in the same folder as the rest of the files (e.g. cpp, so). This is to be consistent of how files are packaged internally (more details in internal Diff summary). Note on using `filesystem`: Seems like there'll be [issues](pytorch#137209) with using`filesystem` header in linux, so here I use string manipulation instead of `filesystem::path`. Test Plan: ``` test/inductor:torchbind -- -r torchbind_aoti test/inductor:torchbind -- -r aot_compile ``` Differential Revision: D72063691 Pull Request resolved: pytorch#150196 Approved by: https://github.com/hl475, https://github.com/desertfire

Fix dependency on filesystem on Linux

6656d6a

pytorch-bot bot added the ciflow/inductor label Oct 2, 2024

atalman added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 2, 2024

kit1980 approved these changes Oct 2, 2024

View reviewed changes

fix_return_value

4c94bc7

atalman added topic: not user facing topic category ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR labels Oct 2, 2024

malfet approved these changes Oct 2, 2024

View reviewed changes

atalman added 5 commits October 2, 2024 13:20

apply suggestion

57b76c6

fix_macro

4219fd7

test

c1c4133

lint

bdc01d3

fixlint

01d8bea

pytorchmergebot added the merging label Oct 2, 2024

pytorchmergebot removed the merging label Oct 2, 2024

kit1980 self-assigned this Oct 2, 2024

pytorchmergebot added the merging label Oct 3, 2024

pytorchmergebot added the Merged label Oct 3, 2024

pytorchmergebot closed this in 6241006 Oct 3, 2024

pytorchmergebot removed the merging label Oct 3, 2024

kit1980 mentioned this pull request Oct 3, 2024

[RELEASE-ONLY CHANGES] Fix dependency on filesystem on Linux #137241

Closed

kit1980 mentioned this pull request Oct 3, 2024

[RELEASE-ONLY CHANGES] Fix dependency on filesystem on Linux #137242

Merged

yushangdi mentioned this pull request Mar 28, 2025

[reland] Support torchbind in OSS proxy executor #150196

Closed

Fix dependency on filesystem on Linux #137209

Fix dependency on filesystem on Linux #137209

Uh oh!

Conversation

atalman commented Oct 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137209

❌ 5 Cancelled Jobs, 1 Unrelated Failure

Uh oh!

kit1980 Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

malfet Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

malfet Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

malfet Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

malfet Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

atalman commented Oct 2, 2024

Uh oh!

pytorchmergebot commented Oct 2, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 2, 2024

Merge failed

Uh oh!

atalman commented Oct 3, 2024

Uh oh!

pytorchmergebot commented Oct 3, 2024

Merge started

Uh oh!

kit1980 commented Oct 3, 2024

Uh oh!

pytorchbot commented Oct 3, 2024

Cherry picking #137209

Uh oh!

kit1980 commented Oct 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

atalman commented Oct 2, 2024 •

edited

Loading

pytorch-bot bot commented Oct 2, 2024 •

edited

Loading