Skip to content

Comments

fix: setting builder optimization level to TRT 8.6 default#15897

Merged
chilo-ms merged 3 commits intomicrosoft:mainfrom
gedoensmax:trt_opt_lvl_defaukt
May 12, 2023
Merged

fix: setting builder optimization level to TRT 8.6 default#15897
chilo-ms merged 3 commits intomicrosoft:mainfrom
gedoensmax:trt_opt_lvl_defaukt

Conversation

@gedoensmax
Copy link
Contributor

@gedoensmax gedoensmax commented May 10, 2023 •
edited
Loading

The actual released default level is 3 and not the previously used 2.

Just a small sample of the effects:
Screenshot 2023-05-10 at 15 49 55

@chilo-ms
Copy link
Contributor

chilo-ms commented May 10, 2023

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

@chilo-ms
Copy link
Contributor

chilo-ms commented May 10, 2023

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

@azure-pipelines
Copy link

azure-pipelines bot commented May 10, 2023

Azure Pipelines successfully started running 5 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented May 10, 2023

Azure Pipelines successfully started running 9 pipeline(s).

@jywu-msft
Copy link
Member

jywu-msft commented May 10, 2023

/azp run Linux QNN CI Pipeline, Windows ARM64 QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented May 10, 2023

Azure Pipelines successfully started running 2 pipeline(s).

@chilo-ms
Copy link
Contributor

chilo-ms commented May 11, 2023 •
edited
Loading

Could you help merge the main to pass "Linux GPU TensorRT CI Pipeline"?
There are some PRs related to ONNX 1.14 release were merged to main right after your PR.

This was referenced May 11, 2023
Closed
Merged
@jywu-msft
Copy link
Member

jywu-msft commented May 11, 2023

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

@jywu-msft
Copy link
Member

jywu-msft commented May 11, 2023

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

@jywu-msft
Copy link
Member

jywu-msft commented May 11, 2023

/azp run Linux QNN CI Pipeline, Windows ARM64 QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented May 11, 2023

Azure Pipelines successfully started running 9 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented May 11, 2023

Azure Pipelines successfully started running 5 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented May 11, 2023

Azure Pipelines successfully started running 2 pipeline(s).

@jywu-msft jywu-msft requested a review from chilo-ms May 11, 2023 14:50
chilo-ms
chilo-ms reviewed May 11, 2023
View reviewed changes
onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc
Copy link
Contributor

@chilo-ms chilo-ms May 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 2831 & 2892 need to be modified as well.

if (trt_state->builder_optimization_level != 2) {

Copy link
Member

@jywu-msft jywu-msft May 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll have to kick off the CI again after those lines are updated.

Copy link
Member

@jywu-msft jywu-msft May 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we really need to simplify option updates, too error prone currently due to the need to change too many places in code.

Copy link
Contributor Author

@gedoensmax gedoensmax May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Is this something your team is working on right now ?

Copy link
Member

@jywu-msft jywu-msft May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's on our radar and we would like to dedicate some time/resources on it.

jywu-msft
jywu-msft previously approved these changes May 11, 2023
View reviewed changes
@jywu-msft jywu-msft dismissed their stale review May 11, 2023 22:22

minor change needed

@jywu-msft
Copy link
Member

jywu-msft commented May 12, 2023

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

@jywu-msft
Copy link
Member

jywu-msft commented May 12, 2023

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

@jywu-msft
Copy link
Member

jywu-msft commented May 12, 2023

/azp run Linux QNN CI Pipeline, Windows ARM64 QNN CI Pipeline

@azure-pipelines
Copy link

azure-pipelines bot commented May 12, 2023

Azure Pipelines successfully started running 5 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented May 12, 2023

Azure Pipelines successfully started running 9 pipeline(s).

@azure-pipelines
Copy link

azure-pipelines bot commented May 12, 2023

Azure Pipelines successfully started running 2 pipeline(s).

@chilo-ms chilo-ms merged commit 1435510 into microsoft:main May 12, 2023
prathikr pushed a commit that referenced this pull request May 16, 2023
@gedoensmax
fix: setting builder optimization level to TRT 8.6 default (#15897)
d7b4b47 
The actual released default level is 3 and not the previously used 2.

Just a small sample of the effects:
![Screenshot 2023-05-10 at 15 49
55](https://github.com/microsoft/onnxruntime/assets/44298237/5a694446-22c0-4943-9ddf-80670781878f)
@snnn snnn added the triage:approved Approved for cherrypicks for release label May 18, 2023
snnn pushed a commit that referenced this pull request May 19, 2023
@gedoensmax
fix: setting builder optimization level to TRT 8.6 default (#15897)
a8ba9c9 
The actual released default level is 3 and not the previously used 2.

Just a small sample of the effects:
![Screenshot 2023-05-10 at 15 49
55](https://github.com/microsoft/onnxruntime/assets/44298237/5a694446-22c0-4943-9ddf-80670781878f)
snnn pushed a commit that referenced this pull request May 19, 2023
@gedoensmax
fix: setting builder optimization level to TRT 8.6 default (#15897)
ba38cec 
The actual released default level is 3 and not the previously used 2.

Just a small sample of the effects:
![Screenshot 2023-05-10 at 15 49
55](https://github.com/microsoft/onnxruntime/assets/44298237/5a694446-22c0-4943-9ddf-80670781878f)
snnn pushed a commit that referenced this pull request May 19, 2023
Cherry-picks to the release branch (#16017)
6cdf071 
### Description
Cherry-picks 26 commits to the release branch. 
Most cherry-picks are clean merges. Except:

1. When I got conflicts in cgmanifest.json and download-deps.yml, I
choose to ignore the conflicts and regenerate the two files
2. There were some conflicts in cmake/deps.txt, onnxruntime_c_api.cc


PR list:

[js/webgpu] fix Transpose with non-float tensor (#15819)
[js/web] fix terser reserved symbols for worker (#15864)
[JSEP] fix constructor for OrtDevice (#15805)
Bump engine.io from 6.4.1 to 6.4.2 in /js/web (#15799)
Bump engine.io from 6.4.0 to 6.4.2 in /onnxruntime/test/wasm (#15798)
[wasm] revert emsdk to v3.1.19 (#15793)
[wasm/JSEP] add threaded build to artifacts (#15777)
[js/web] add target ort.webgpu.min.js (#15780)
update ort extensions to 94142d8391c9791ec71c38336436319a2d4ac7a0 (#15688)
fix: setting builder optimization level to TRT 8.6 default (#15897)
Adust GetVersionString() GetBuildInfoString() signatures and move them to OrtApi (#15921)
Fix segfault for multiple GPU run (regression) (#15823)
android package fix (#15999)
[CoreML EP] Minor changes to allow CoreML EP to handle more nodes and models. (#15993)
Adding support for conv fp16 fusion on Resnet50v1 (#15474)
update onnx release 1.14 for docker files (#15680)
Avoid generating training documentation during packaging (#15795)
Update Conv-Add-Relu Fusion Transformation (#15834)
Fix symbolic shape infer empty value_info (#15842)
NhwcFusedConv: Add before Activation (#15837)
use __hmul2 instead of __hmul2_rn (#15852)
change the EP device to default OrtDevice() for memoryType equals CPU Input (#15903)
Fixing NhwcFusedConv fp16 (#15950)
fix topo sort in quantization tool (#16003)
[doc] add LeakyRelu to coreml supported ops (#15944)
[DML EP] Add frequent upload heap flushing (#15960)

Co-authored-by: Yulong Wang 
Co-authored-by: dependabot[bot] 
Co-authored-by: Guenther Schmuelling 
Co-authored-by: Shalva Mist 
Co-authored-by: Maximilian Müller 
Co-authored-by: Dmitri Smirnov 
Co-authored-by: pengwa 
Co-authored-by: Ashwini Khade 
Co-authored-by: Edward Chen 
Co-authored-by: Jian Chen 
Co-authored-by: liqun Fu 
Co-authored-by: Baiju Meswani 
Co-authored-by: Tianlei Wu 
Co-authored-by: Chen Fu 
Co-authored-by: Ye Wang 
Co-authored-by: cao lei 
Co-authored-by: Yufeng Li 
Co-authored-by: Rachel Guo 
Co-authored-by: Patrice Vignola
@snnn snnn removed triage:approved Approved for cherrypicks for release release:1.15 labels May 19, 2023
preetha-intel pushed a commit to intel/onnxruntime that referenced this pull request Jun 7, 2023
@preetha-intel
Cherry-picks to the release branch (microsoft#16017)
5380939 
### Description
Cherry-picks 26 commits to the release branch. 
Most cherry-picks are clean merges. Except:

1. When I got conflicts in cgmanifest.json and download-deps.yml, I
choose to ignore the conflicts and regenerate the two files
2. There were some conflicts in cmake/deps.txt, onnxruntime_c_api.cc


PR list:

[js/webgpu] fix Transpose with non-float tensor (microsoft#15819)
[js/web] fix terser reserved symbols for worker (microsoft#15864)
[JSEP] fix constructor for OrtDevice (microsoft#15805)
Bump engine.io from 6.4.1 to 6.4.2 in /js/web (microsoft#15799)
Bump engine.io from 6.4.0 to 6.4.2 in /onnxruntime/test/wasm (microsoft#15798)
[wasm] revert emsdk to v3.1.19 (microsoft#15793)
[wasm/JSEP] add threaded build to artifacts (microsoft#15777)
[js/web] add target ort.webgpu.min.js (microsoft#15780)
update ort extensions to 94142d8391c9791ec71c38336436319a2d4ac7a0 (microsoft#15688)
fix: setting builder optimization level to TRT 8.6 default (microsoft#15897)
Adust GetVersionString() GetBuildInfoString() signatures and move them to OrtApi (microsoft#15921)
Fix segfault for multiple GPU run (regression) (microsoft#15823)
android package fix (microsoft#15999)
[CoreML EP] Minor changes to allow CoreML EP to handle more nodes and models. (microsoft#15993)
Adding support for conv fp16 fusion on Resnet50v1 (microsoft#15474)
update onnx release 1.14 for docker files (microsoft#15680)
Avoid generating training documentation during packaging (microsoft#15795)
Update Conv-Add-Relu Fusion Transformation (microsoft#15834)
Fix symbolic shape infer empty value_info (microsoft#15842)
NhwcFusedConv: Add before Activation (microsoft#15837)
use __hmul2 instead of __hmul2_rn (microsoft#15852)
change the EP device to default OrtDevice() for memoryType equals CPU Input (microsoft#15903)
Fixing NhwcFusedConv fp16 (microsoft#15950)
fix topo sort in quantization tool (microsoft#16003)
[doc] add LeakyRelu to coreml supported ops (microsoft#15944)
[DML EP] Add frequent upload heap flushing (microsoft#15960)

Co-authored-by: Yulong Wang 
Co-authored-by: dependabot[bot] 
Co-authored-by: Guenther Schmuelling 
Co-authored-by: Shalva Mist 
Co-authored-by: Maximilian Müller 
Co-authored-by: Dmitri Smirnov 
Co-authored-by: pengwa 
Co-authored-by: Ashwini Khade 
Co-authored-by: Edward Chen 
Co-authored-by: Jian Chen 
Co-authored-by: liqun Fu 
Co-authored-by: Baiju Meswani 
Co-authored-by: Tianlei Wu 
Co-authored-by: Chen Fu 
Co-authored-by: Ye Wang 
Co-authored-by: cao lei 
Co-authored-by: Yufeng Li 
Co-authored-by: Rachel Guo 
Co-authored-by: Patrice Vignola
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jywu-msft jywu-msft jywu-msft left review comments

@chilo-ms chilo-ms chilo-ms approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@gedoensmax @chilo-ms @jywu-msft @snnn