Skip to content

Conversation

@yuslepukhin
Copy link
Member

Description

Move moving weights to memory to the end of Graph::Resolve().
Modify Inject so it copies data into TensorProto according to the C API docs.

Motivation and Context

TypeAndShape inference runs as a part of Resolve() and it unable to inspect and load the initializers that point to OrtValues at that time. We choose to move TensorProto to OrtValue conversion at the end of Resolve().

References: #25579

@yuslepukhin yuslepukhin requested a review from skottmckay August 1, 2025 03:02
@yuslepukhin yuslepukhin marked this pull request as ready for review August 1, 2025 17:17
@jywu-msft
Copy link
Member

is the test failure real or a flakey test?

[----------] 3 tests from WriteScores
1: [ RUN ] WriteScores.multiple_scores_transform_none
1: unknown file: error: SEH exception with code 0xc0000006 thrown in the test fixture's constructor.
1: Stack trace:
1: 00007FFCEAF3E5CF: _C_specific_handler
1: 00007FFCF9F0623F: _chkstk
1: 00007FFCF9E145D7: RtlWow64GetCurrentCpuArea
1: 00007FFCF9F05B7E: KiUserExceptionDispatcher
1: 00007FFCF9E170C3: RtlVirtualUnwind2
1: 00007FFCF9E14548: RtlWow64GetCurrentCpuArea
1: 00007FFCF9F05B7E: KiUserExceptionDispatcher
1: 00007FFCEAA478B0: std::_Random_device
1: 00007FF777BA3A12: WriteScores::WriteScores
1: 00007FF777BA1F15: testing::internal::TestFactoryImpl<WriteScores_multiple_scores_transform_none_Test>::CreateTest
1: 00007FF778C80192: testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase,testing::Test *>
1: 00007FF778C800A6: testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase,testing::Test *>
1: 00007FF778CA617B: testing::TestInfo::Run
1: 00007FF778CA6341: testing::TestSuite::Run
1: 00007FF778CA6E25: testing::internal::UnitTestImpl::RunAllTests
1: ... Google Test internal frames ...

@yuslepukhin
Copy link
Member Author

is the test failure real or a flakey test?

[----------] 3 tests from WriteScores 1: [ RUN ] WriteScores.multiple_scores_transform_none 1: unknown file: error: SEH exception with code 0xc0000006 thrown in the test fixture's constructor. 1: Stack trace: 1: 00007FFCEAF3E5CF: _C_specific_handler 1: 00007FFCF9F0623F: _chkstk 1: 00007FFCF9E145D7: RtlWow64GetCurrentCpuArea 1: 00007FFCF9F05B7E: KiUserExceptionDispatcher 1: 00007FFCF9E170C3: RtlVirtualUnwind2 1: 00007FFCF9E14548: RtlWow64GetCurrentCpuArea 1: 00007FFCF9F05B7E: KiUserExceptionDispatcher 1: 00007FFCEAA478B0: std::_Random_device 1: 00007FF777BA3A12: WriteScores::WriteScores 1: 00007FF777BA1F15: testing::internal::TestFactoryImpl<WriteScores_multiple_scores_transform_none_Test>::CreateTest 1: 00007FF778C80192: testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase,testing::Test *> 1: 00007FF778C800A6: testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase,testing::Test *> 1: 00007FF778CA617B: testing::TestInfo::Run 1: 00007FF778CA6341: testing::TestSuite::Run 1: 00007FF778CA6E25: testing::internal::UnitTestImpl::RunAllTests 1: ... Google Test internal frames ...

This appears to be real. I am getting help with NNAPI.
I am expecting to get QC machine this afternoon.

@yuslepukhin yuslepukhin merged commit ca8ee36 into main Aug 2, 2025
92 checks passed
@yuslepukhin yuslepukhin deleted the yuslepukhin/move_ortinitializers_resolve branch August 2, 2025 01:38
snnn pushed a commit that referenced this pull request Aug 3, 2025
### Description
<!-- Describe your changes. -->
Move moving weights to memory to the end of Graph::Resolve().
Modify Inject so it copies data into TensorProto according to the C API
docs.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
TypeAndShape inference runs as a part of `Resolve()` and it unable to
inspect and load the initializers that point to OrtValues at that time.
We choose to move TensorProto to OrtValue conversion at the end of
`Resolve()`.

References: #25579
adrianlizarraga pushed a commit that referenced this pull request Aug 4, 2025
### Description
<!-- Describe your changes. -->
Move moving weights to memory to the end of Graph::Resolve().
Modify Inject so it copies data into TensorProto according to the C API
docs.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
TypeAndShape inference runs as a part of `Resolve()` and it unable to
inspect and load the initializers that point to OrtValues at that time.
We choose to move TensorProto to OrtValue conversion at the end of
`Resolve()`.

References: #25579

Co-authored-by: Dmitri Smirnov <[email protected]>
sanketkaleoss pushed a commit to sanketkaleoss/onnxruntime that referenced this pull request Aug 11, 2025
…ft#25626)

### Description
<!-- Describe your changes. -->
Move moving weights to memory to the end of Graph::Resolve().
Modify Inject so it copies data into TensorProto according to the C API
docs.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
TypeAndShape inference runs as a part of `Resolve()` and it unable to
inspect and load the initializers that point to OrtValues at that time.
We choose to move TensorProto to OrtValue conversion at the end of
`Resolve()`.

References: microsoft#25579
gedoensmax pushed a commit to gedoensmax/onnxruntime that referenced this pull request Sep 2, 2025
…ft#25626)

### Description
<!-- Describe your changes. -->
Move moving weights to memory to the end of Graph::Resolve().
Modify Inject so it copies data into TensorProto according to the C API
docs.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
TypeAndShape inference runs as a part of `Resolve()` and it unable to
inspect and load the initializers that point to OrtValues at that time.
We choose to move TensorProto to OrtValue conversion at the end of
`Resolve()`.

References: microsoft#25579
@snnn snnn mentioned this pull request Sep 16, 2025
yuslepukhin added a commit that referenced this pull request Oct 30, 2025
…lues early (#26345)

### Description
Converts weights early and revert "Properly remove in-memory references
(#25652)"
This reverts commit 3ca49d8 and makes
appropriate adjustments for the current state of the code.

This PR is made possible and on the heels of:
#26263
#25833.

Previous history:
#23979
#25320
#25626
#25652

The first change (#26263)
allows us to convert initializers to OrtValues early and save lots of
memory at model loading time.

Specifically, for Phi-4-mini-instruct-INT4 model before and after looks
like this:

**Before**
<img width="1204" height="124" alt="Before change DEBUG 2025-10-16
144819"
src="https://github.com/user-attachments/assets/674ff75b-057f-498a-a906-0140d59d46e6"
/>

**After**

<img width="997" height="114" alt="After change DEBUG 2025-10-16 144819"
src="https://github.com/user-attachments/assets/df1783af-7f50-4cd2-b3ad-6868f23be53f"
/>

The two peaks represent memory usage at optimization time (8.1Gb before)
and after weights memory mapping (6.5Gb)
After this change corresponding numbers look 3.5Gb and 4.7Gb
respectively.
Most of the savings during optimization phase come from
`ConstantFolding` where we are able to reuse the resulting OrtValues
directly for the new initializers.

This PR concludes a series of PRs converting initializers to OrtValues.

Memory consumption before the conversion began was 9.3Gb and 6.7Gb
respectively. We are saving almost 6Gb during optimization and 2Gb for
the steady state.
 
 
<img width="1175" height="139" alt="image"
src="https://github.com/user-attachments/assets/80e7d228-8a8e-4316-8e04-b02c2be30f04"
/>

The model also loads about 12 seconds faster.

Example of ConstantFolding being one of the top contributors where we
duplicate memory for higher peak before Resolve takes care of no longer
used initializers.
<img width="1100" height="558" alt="Sanpshot 3 Peak on ConstantFolding
Transpose Optimizer"
src="https://github.com/user-attachments/assets/95545abd-3f99-46d9-862e-bbf27cbb5b40"
/>

<img width="1060" height="600" alt="Snapshot 4 Peak AddInitializer from
ConstantFolding"
src="https://github.com/user-attachments/assets/dd457ec6-23ee-4efd-8c60-625d5faad61e"
/>

<img width="325" height="160" alt="image"
src="https://github.com/user-attachments/assets/37c1194d-f683-49a7-afb1-073dfbb9bbfc"
/>


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Reduce memory usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants