Convert graph initializers into OrtValue Phase I #23979
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // can not trace string tensor | ||
| ORT_ENFORCE(entry->second->data_type() != ONNX_NAMESPACE::TensorProto_DataType_STRING, "Can not trace string tensor"); | ||
| ORT_RETURN_IF_ERROR(planner.Trace(entry->first, entry->second)); | ||
| const auto [_, tensor_proto] = *entry; |
Check notice
Code scanning / CodeQL
Unused local variable
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 10 months ago
To fix the problem, we need to remove the unused variable _ from the code. This can be done by modifying the line where _ is declared and removing it. The best way to fix this without changing existing functionality is to directly destructure the entry variable to only extract the tensor_proto part, which is actually used.
Suggested changeset
1
onnxruntime/core/framework/session_state_utils.cc
| @@ -269,3 +269,3 @@ | ||
| "OrtValue index: ", ort_value_index, " from initializer_allocation_order not found among initialized tensors"); | ||
| const auto [_, tensor_proto] = *entry; | ||
| const auto& tensor_proto = entry->second; | ||
|
|
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
yuslepukhin
force-pushed
the
yuslepukhin/ort_initializers
branch
8 times, most recently
from
650c07f to
2c4f38f
Compare
yuslepukhin
force-pushed
the
yuslepukhin/ort_initializers
branch
3 times, most recently
from
dfcbe93 to
ac5bed7
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Member
Author
|
Cc: @ranjitshs pls, take a look and perhaps build as well. Thx! |
yuslepukhin
changed the title
yuslepukhin
changed the title
Contributor
Yes. I will try to build and let you know in 1-2 days. |
yuslepukhin
commented
May 5, 2025
yuslepukhin
commented
May 6, 2025
onnxruntime/core/providers/shared_library/provider_wrappedtypes.h
Outdated
Show resolved
Hide resolved
yuslepukhin
force-pushed
the
yuslepukhin/ort_initializers
branch
from
0610388 to
0bc68a1
Compare
yuslepukhin
changed the title
Contributor
|
Hi @yuslepukhin |
Member
Author
Which commit did you try? |
Contributor
|
@yuslepukhin |
Contributor
|
Below are some stack trace for failures:
This is crashed in onnxruntime::Graph::AddInitializedTensor . Let me know if you find any useful info from this . I will try to understand changes in this PR. :) |
snnn
reviewed
Jun 10, 2025
skottmckay
approved these changes
Jun 11, 2025
Contributor
|
@yuslepukhin BTW, any idea when we are going for next release ? |
adrianlizarraga
added a commit
that referenced
this pull request
Jun 26, 2025
#25159) ### Description Updates the `OrtGraph` implementation to take advantage of the work done in PR #23979, which sets the infrastructure to store initializers as `OrtValue` instances in the `onnxruntime::Graph`. There still needs to be second part to the [aforementioned PR](#23979) to ensure that all initializers are stored as `OrtValue`s in the Graph. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
adrianlizarraga
pushed a commit
that referenced
this pull request
Jul 23, 2025
### Description Make protobuf weights refer to OrtValues on load. Create OrtValues for initializers that are loaded from ORT format for uniformity. Create OrtValues for ORT format initializers. Adjust exporting Graph::ToGraphProto() so it does not export in memory references in external data. Make CoreML process external data including in memory references so it can copy it. ### Motivation and Context Follow up for #23979
wcy123
pushed a commit
to wcy123/onnxruntime
that referenced
this pull request
Aug 1, 2025
it is related to microsoft#25320 microsoft#23979
This was referenced Aug 1, 2025
carzh
pushed a commit
that referenced
this pull request
Aug 7, 2025
### Description Make protobuf weights refer to OrtValues on load. Create OrtValues for initializers that are loaded from ORT format for uniformity. Create OrtValues for ORT format initializers. Adjust exporting Graph::ToGraphProto() so it does not export in memory references in external data. Make CoreML process external data including in memory references so it can copy it. ### Motivation and Context Follow up for #23979
adrianlizarraga
pushed a commit
that referenced
this pull request
Aug 8, 2025
### Description It is related to #25320 #23979. Enable tensor raw data sharing for externalized tensor proto with kTensorProtoMemoryAddressTag ### Motivation and Context With #25320 #23979, all initialized tensor protos are associated with OrtValue, VitisiAI EP need to adapt to this change. Co-authored-by: mingyue <[email protected]>
adrianlizarraga
pushed a commit
that referenced
this pull request
Aug 9, 2025
### Description It is related to #25320 #23979. Enable tensor raw data sharing for externalized tensor proto with kTensorProtoMemoryAddressTag ### Motivation and Context With #25320 #23979, all initialized tensor protos are associated with OrtValue, VitisiAI EP need to adapt to this change. Co-authored-by: mingyue <[email protected]>
sanketkaleoss
pushed a commit
to sanketkaleoss/onnxruntime
that referenced
this pull request
Aug 11, 2025
### Description Make protobuf weights refer to OrtValues on load. Create OrtValues for initializers that are loaded from ORT format for uniformity. Create OrtValues for ORT format initializers. Adjust exporting Graph::ToGraphProto() so it does not export in memory references in external data. Make CoreML process external data including in memory references so it can copy it. ### Motivation and Context Follow up for microsoft#23979
sanketkaleoss
pushed a commit
to sanketkaleoss/onnxruntime
that referenced
this pull request
Aug 11, 2025
### Description It is related to microsoft#25320 microsoft#23979. Enable tensor raw data sharing for externalized tensor proto with kTensorProtoMemoryAddressTag ### Motivation and Context With microsoft#25320 microsoft#23979, all initialized tensor protos are associated with OrtValue, VitisiAI EP need to adapt to this change. Co-authored-by: mingyue <[email protected]>
gedoensmax
pushed a commit
to gedoensmax/onnxruntime
that referenced
this pull request
Sep 2, 2025
### Description It is related to microsoft#25320 microsoft#23979. Enable tensor raw data sharing for externalized tensor proto with kTensorProtoMemoryAddressTag ### Motivation and Context With microsoft#25320 microsoft#23979, all initialized tensor protos are associated with OrtValue, VitisiAI EP need to adapt to this change. Co-authored-by: mingyue <[email protected]>
yuslepukhin
added a commit
that referenced
this pull request
Oct 30, 2025
…lues early (#26345) ### Description Converts weights early and revert "Properly remove in-memory references (#25652)" This reverts commit 3ca49d8 and makes appropriate adjustments for the current state of the code. This PR is made possible and on the heels of: #26263 #25833. Previous history: #23979 #25320 #25626 #25652 The first change (#26263) allows us to convert initializers to OrtValues early and save lots of memory at model loading time. Specifically, for Phi-4-mini-instruct-INT4 model before and after looks like this: **Before** <img width="1204" height="124" alt="Before change DEBUG 2025-10-16 144819" src="https://github.com/user-attachments/assets/674ff75b-057f-498a-a906-0140d59d46e6" /> **After** <img width="997" height="114" alt="After change DEBUG 2025-10-16 144819" src="https://github.com/user-attachments/assets/df1783af-7f50-4cd2-b3ad-6868f23be53f" /> The two peaks represent memory usage at optimization time (8.1Gb before) and after weights memory mapping (6.5Gb) After this change corresponding numbers look 3.5Gb and 4.7Gb respectively. Most of the savings during optimization phase come from `ConstantFolding` where we are able to reuse the resulting OrtValues directly for the new initializers. This PR concludes a series of PRs converting initializers to OrtValues. Memory consumption before the conversion began was 9.3Gb and 6.7Gb respectively. We are saving almost 6Gb during optimization and 2Gb for the steady state. <img width="1175" height="139" alt="image" src="https://github.com/user-attachments/assets/80e7d228-8a8e-4316-8e04-b02c2be30f04" /> The model also loads about 12 seconds faster. Example of ConstantFolding being one of the top contributors where we duplicate memory for higher peak before Resolve takes care of no longer used initializers. <img width="1100" height="558" alt="Sanpshot 3 Peak on ConstantFolding Transpose Optimizer" src="https://github.com/user-attachments/assets/95545abd-3f99-46d9-862e-bbf27cbb5b40" /> <img width="1060" height="600" alt="Snapshot 4 Peak AddInitializer from ConstantFolding" src="https://github.com/user-attachments/assets/dd457ec6-23ee-4efd-8c60-625d5faad61e" /> <img width="325" height="160" alt="image" src="https://github.com/user-attachments/assets/37c1194d-f683-49a7-afb1-073dfbb9bbfc" /> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Reduce memory usage.
quic-ankus
pushed a commit
to CodeLinaro/onnxruntime
that referenced
this pull request
Nov 25, 2025
### Description This PR converts TensorProto graph initializers to TensorProto/OrtValue pairs. Currently, we only split the output for some optimizers to the above pairs. Eventually, we should be able to convert all initializers to OrtValues on load. Small weights will continue to be an exception as they are sometimes required by ONNX inference functions. Some graph API leaks to EPs so we are not able to remove it at present, and this constrains our ability to convert everything at once. ### Motivation and Context Lay Gound for proper layers separation. Eventually eliminate weights copies in the EPs.
quic-ankus
pushed a commit
to CodeLinaro/onnxruntime
that referenced
this pull request
Nov 25, 2025
microsoft#25159) ### Description Updates the `OrtGraph` implementation to take advantage of the work done in PR microsoft#23979, which sets the infrastructure to store initializers as `OrtValue` instances in the `onnxruntime::Graph`. There still needs to be second part to the [aforementioned PR](microsoft#23979) to ensure that all initializers are stored as `OrtValue`s in the Graph. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR converts TensorProto graph initializers to TensorProto/OrtValue pairs.
Currently, we only split the output for some optimizers to the above pairs.
Eventually, we should be able to convert all initializers to OrtValues on load.
Small weights will continue to be an exception as the are sometimes required by ONNX inference functions.
Some graph API leaks to EPs so we are not able to remove it at present, and this constrains our ability to convert everything at once.
Motivation and Context
Lay Gound for proper layers separation. Eventually eliminate weights copies in the EPs.