Skip to content

fix memory profile for partial graph run#11911

Merged
pengwa merged 9 commits intomasterfrom
pengwa/memory_profile_fix
Jun 24, 2022
Merged

fix memory profile for partial graph run#11911
pengwa merged 9 commits intomasterfrom
pengwa/memory_profile_fix

Conversation

@pengwa
Copy link
Contributor

@pengwa pengwa commented Jun 20, 2022

Description: fix memory profile for partial graph run

Memory Profile is helpful for analyze memory peak used in static ORT planning.

This PR targets to fix the issues when enable memory profile in ORTModule partial graph run. Two issues:

  1. Loose the failure if some tensors are missing in track.
  2. Only summarized after backward partial graph run.

Validated with
./build.sh --config $flavor --use_cuda --enable_training --build_wheel --skip_tests --cuda_version=11.3 --parallel 8 --enable_training_torch_interop --cmake_extra_defines onnxruntime_ENABLE_MEMORY_PROFILE=ON

and

pytest -k test_forward_call_single_positional_argument orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py

Motivation and Context

  • Why is this change required? What problem does it solve?
  • If it fixes an open issue, please link to the issue here.

@pengwa pengwa added the training issues related to ONNX Runtime training; typically submitted using template label Jun 20, 2022
@pengwa pengwa requested review from Lafi7e, askhade, tlh20 and wezuo June 20, 2022 08:10
wezuo
wezuo previously approved these changes Jun 23, 2022
@pengwa pengwa merged commit 0d6cbc6 into master Jun 24, 2022
@pengwa pengwa deleted the pengwa/memory_profile_fix branch June 24, 2022 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

training issues related to ONNX Runtime training; typically submitted using template

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants