Skip to content

dedup _get_model_external_data_paths#2217

Merged
IlyasMoutawwakil merged 4 commits into
huggingface:mainfrom
xieofxie:xieofxie/dedup
Apr 15, 2025
Merged

dedup _get_model_external_data_paths#2217
IlyasMoutawwakil merged 4 commits into
huggingface:mainfrom
xieofxie:xieofxie/dedup

Conversation

@xieofxie

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #2216

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@IlyasMoutawwakil

Copy link
Copy Markdown
Member

can you provide a concrete case where this happens ?

@xieofxie

xieofxie commented Apr 7, 2025

Copy link
Copy Markdown
Contributor Author

when using Olive, I could get models like this
model.zip

sample code

import onnx
from onnx.external_data_helper import ExternalDataInfo, _get_initializer_tensors
from pathlib import Path
from collections import Counter

model_path = Path("./model.onnx")

# _get_model_external_data_paths 
onnx_model = onnx.load(str(model_path), load_external_data=False)
model_tensors = _get_initializer_tensors(onnx_model)
# filter out tensors that are not external data
model_tensors_ext = [
    ExternalDataInfo(tensor).location
    for tensor in model_tensors
    if tensor.HasField("data_location") and tensor.data_location == onnx.TensorProto.EXTERNAL
]
data = [model_path.parent / tensor_name for tensor_name in model_tensors_ext]

data_count = Counter(data)
print(data_count)

Got Counter({WindowsPath('./weights.pb'): 685})

@IlyasMoutawwakil

Copy link
Copy Markdown
Member

okay I see

Comment thread optimum/onnx/utils.py Outdated
Comment thread optimum/onnx/utils.py Outdated
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@IlyasMoutawwakil

Copy link
Copy Markdown
Member

@xieofxie please run pip install .[quality] and then make style

@xieofxie

Copy link
Copy Markdown
Contributor Author

Internal Error 。。

@IlyasMoutawwakil

Copy link
Copy Markdown
Member

@xieofxie yeah those are okay, merging !

@IlyasMoutawwakil
IlyasMoutawwakil merged commit 3239964 into huggingface:main Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dedup _get_model_external_data_paths

3 participants