Skip to content

[EP ABI] API to get external initializer info + lazy load external OrtValues #25482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
adrianlizarraga merged 43 commits into from
Jul 24, 2025
Merged

[EP ABI] API to get external initializer info + lazy load external OrtValues #25482

adrianlizarraga merged 43 commits into from
Jul 24, 2025

Conversation

@adrianlizarraga
Copy link
Contributor

@adrianlizarraga adrianlizarraga commented Jul 21, 2025 •
edited
Loading

Description

  • Adds APIs to get information (file path, file offset, byte size) for initializers with data in external files. This allows EPs to do their own custom memory-mapping of initializer data. By default, EPs that don't have specific requirements can still use ValueInfo_GetInitializerValue to get an OrtValue with memory-mapped initializer data.
  • Updates OrtGraph to only load OrtValue for external initializers on demand. This prevents having to memory map all external initializers before the first call to OrtEp::GetCapability.

Follow up to #25320

New API functions:

Function Summary
ValueInfo_GetExternalInitializerInfo Get OrtExternalInitializerInfo from OrtValueInfo (or NULL). Must be released with ReleaseExternalInitializerInfo
ReleaseExternalInitializerInfo Releases the OrtExternalInitializerInfo instance
ExternalInitializerInfo_GetFilePath Returns the relative path to the file that stores the initializer's data
ExternalInitializerInfo_GetFileOffset Returns the byte offset within the file where the initializer's data is stored
ExternalInitializerInfo_GetByteSize Returns the size in bytes of the initializer's data within the file

Motivation and Context

yuslepukhin and others added 21 commits July 7, 2025 19:35
@yuslepukhin
Adjust InlineIfNode
e4c8244 
Fix reshaping for external weights
Fix Fusion Helper
@yuslepukhin
Merge branch 'main' into yusleoukhin/ort_initializers_ii
5dc217e
@yuslepukhin
Fix a bug in Initializer::ToProtoWithOrtValue
ffb9bbe
@yuslepukhin
Address type conversion
e19b938
@yuslepukhin
Adjust ToProto() handling of in external data in memory
564c29f
@yuslepukhin
Make CoreML accept external initializers
507eb56
@yuslepukhin
Fix handling of external data in ToProto*()
5e89498
@yuslepukhin
Fix compiler error
fdcdc12
@yuslepukhin
Merge branch 'yuslepukhin/ort_initializers_mac' into yusleoukhin/ort_…
6e29d73 
…initializers_ii
@yuslepukhin
Address compile error in Mac code
a0635b7
@yuslepukhin
Merge branch 'main' into yusleoukhin/ort_initializers_ii
e81b06c
@yuslepukhin
Adjust in memory references when saving optimized model
4cbddc9
@yuslepukhin
Merge branch 'main' into yusleoukhin/ort_initializers_ii
6ab9839
@yuslepukhin
Address ToGraphProto() issues
582f27c
@yuslepukhin
Fix ToGraphProto() and recreate test databases for test_embedlayer_fu…
a636579 
…sion.py
@yuslepukhin
GCC not happy about attr placement
83c313f
@yuslepukhin
Address build error
2c0f97e
@yuslepukhin
Address compiler error
5c15725
@yuslepukhin
Adjust test data for fastgelu fusion
a6b2576
@adrianlizarraga
[EP ABI] Load external initializer OrtValues on demand
bf4e81c
@adrianlizarraga
Add comment about unique_ptr<OrtValue>
7077654
yuslepukhin
yuslepukhin reviewed Jul 21, 2025
View reviewed changes
yuslepukhin
yuslepukhin reviewed Jul 22, 2025
View reviewed changes
yuslepukhin
yuslepukhin reviewed Jul 22, 2025
View reviewed changes
@adrianlizarraga adrianlizarraga changed the title [DRAFT] [EP ABI] Load external initializer OrtValues on demand [DRAFT] [EP ABI] API to get external initializer info + lazy load external OrtValues Jul 23, 2025
Base automatically changed from yusleoukhin/ort_initializers_ii to main July 23, 2025 06:40
@adrianlizarraga adrianlizarraga changed the title [DRAFT] [EP ABI] API to get external initializer info + lazy load external OrtValues [EP ABI] API to get external initializer info + lazy load external OrtValues Jul 23, 2025
@adrianlizarraga adrianlizarraga marked this pull request as ready for review July 23, 2025 06:50
skottmckay
skottmckay reviewed Jul 23, 2025
View reviewed changes
adrianlizarraga
adrianlizarraga commented Jul 23, 2025
View reviewed changes
yuslepukhin
yuslepukhin reviewed Jul 23, 2025
View reviewed changes
yuslepukhin
yuslepukhin reviewed Jul 23, 2025
View reviewed changes
yuslepukhin
yuslepukhin approved these changes Jul 23, 2025
View reviewed changes
Copy link
Member

@yuslepukhin yuslepukhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@adrianlizarraga adrianlizarraga merged commit d702978 into main Jul 24, 2025
106 of 113 checks passed
@adrianlizarraga adrianlizarraga deleted the adrianl/ep-plugin-ext-ort-value-load branch July 24, 2025 21:47
@snnn
Copy link
Contributor

snnn commented Jul 25, 2025

Hi there! We haven't cut the release branch for this version yet, so I'm removing the release:1.23.0 label for now to keep things tidy. Thanks so much for your contribution! We'll make sure this gets included when the release is prepared. 🤖

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skottmckay skottmckay skottmckay left review comments

@yuslepukhin yuslepukhin yuslepukhin approved these changes

+1 more reviewer

@javier-intel javier-intel javier-intel approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@adrianlizarraga @snnn @skottmckay @yuslepukhin @javier-intel