Skip to content

[DevX] Add tool and doc on partial debug builds#116521

Closed
malfet wants to merge 4 commits intomainfrom
malfet/add-incremental-build-tool
Closed

[DevX] Add tool and doc on partial debug builds#116521
malfet wants to merge 4 commits intomainfrom
malfet/add-incremental-build-tool

Conversation

@malfet
Copy link
Contributor

@malfet malfet commented Dec 28, 2023 •
edited
Loading

Turned command sequence mentioned in https://dev-discuss.pytorch.org/t/how-to-get-a-fast-debug-build/1597 and in various discussions into a tool that I use almost daily to debug crashes or correctness issues in the codebase

Essentially it allows one to turn this:

Process 87729 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00000001023d55a8 libtorch_python.dylib`at::indexing::impl::applySelect(at::Tensor const&, long long, c10::SymInt, long long, c10::Device const&, std::__1::optional<c10::ArrayRef<c10::SymInt>> const&)
libtorch_python.dylib`at::indexing::impl::applySelect:
->  0x1023d55a8 <+0>:  sub    sp, sp, #0xd0
    0x1023d55ac <+4>:  stp    x24, x23, [sp, #0x90]
    0x1023d55b0 <+8>:  stp    x22, x21, [sp, #0xa0]
    0x1023d55b4 <+12>: stp    x20, x19, [sp, #0xb0]

into this

Process 87741 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00000001024e2628 libtorch_python.dylib`at::indexing::impl::applySelect(self=0x00000001004ee8a8, dim=0, index=(data_ = 3), real_dim=0, (null)=0x000000016fdfe535, self_sizes= Has Value=true ) at TensorIndexing.h:239:7
   236 	    const at::Device& /*self_device*/,
   237 	    const c10::optional<SymIntArrayRef>& self_sizes) {
   238 	  // See NOTE [nested tensor size for indexing]
-> 239 	  if (self_sizes.has_value()) {
   240 	    auto maybe_index = index.maybe_as_int();
   241 	    if (maybe_index.has_value()) {
   242 	      TORCH_CHECK_INDEX(

while retaining good performance for the rest of the codebase

@malfet
[DevX] Add tool and doc on partial debug builds
90ea5b3 
Turned command sequence mentioned in https://dev-discuss.pytorch.org/t/how-to-get-a-fast-debug-build/1597 and in varios discussions into a tool that I use almost daily to debug crashes or correctness issues in the codebase
@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Dec 28, 2023
@pytorch-bot
Copy link

pytorch-bot bot commented Dec 28, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/116521

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit bfe0163 with merge base 87da0e1 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@malfet malfet requested review from a team and drisspg December 28, 2023 22:26
atalman
atalman approved these changes Dec 28, 2023
View reviewed changes
Copy link
Contributor

@atalman atalman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but lint needs to be fixed

malfet and others added 3 commits December 29, 2023 03:42
@malfet
Fix lint
ea4ad75 
Also, make it more portable
@malfet
TOC
765728d
@malfet
TOC1
bfe0163
@malfet
Copy link
Contributor Author

malfet commented Dec 29, 2023

@pytorchbot merge -f "lint is green"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Dec 29, 2023

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@albanD
Copy link
Collaborator

albanD commented Dec 29, 2023

I'm curious what you think is the UX tradeoff between a separate script vs using an env variable + the usual setup.py develop ?

@malfet malfet deleted the malfet/add-incremental-build-tool branch January 20, 2024 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@atalman atalman atalman approved these changes

@drisspg drisspg Awaiting requested review from drisspg

Assignees

No one assigned

Labels

Merged release notes: devx topic: improvements topic category topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@malfet @pytorchmergebot @albanD @atalman