Skip to content

Add TensorRT optimization profiles - #36660

Merged
tensorflow-copybara merged 16 commits into
tensorflow:masterfrom
tfeher:trt_add_optimization_profiles
Feb 19, 2020
Merged

tensorflow-copybara merged 16 commits into
tensorflow:masterfrom
tfeher:trt_add_optimization_profiles

Conversation

@tfeher

@tfeher tfeher commented Feb 11, 2020

Copy link
Copy Markdown
Contributor

This PR adds optimization profile handling to TF-TRT and enables inference using the experimental dynamic shapes mode (where dynamic shapes mode = explicit batch mode with dynamic input shapes).

This PR is not supposed to change the existing behavior in implicit batch mode.

For each unknown input dimension, a shape optimization profile describes what is the expected [min, max] range for that dimension, and also define an optimum value. "The builder selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are valid for all input tensor sizes in the valid range between minimum and maximum dimensions."

To create an engine with dynamic shapes one must define at least one optimization profile. In the current PR we define only a single optimization profile, using the shape of the concrete input data during the first inference call.

The real advantage of dynamic shapes is to have a single engine that can execute the network with different input shapes. Unfortunately it is not possible to demonstrate this with the current PR. Although the TrtShapeOptimizationProfile class is designed to manage multiple profiles, we still need to enable profile generation mode during build phase, to take the full advatage of dynamic shapes mode. That feature is implemented in PR #36080.

@tensorflow-bot tensorflow-bot Bot added the size:XL CL Change Size:Extra Large label Feb 11, 2020
@tfeher
tfeher requested a review from bixia1 February 11, 2020 12:51
@gbaned gbaned self-assigned this Feb 12, 2020
@gbaned gbaned added the comp:gpu:tensorrt Issues specific to TensorRT label Feb 12, 2020
@gbaned

gbaned commented Feb 12, 2020

Copy link
Copy Markdown
Contributor

@tfeher Can you please resolve conflicts? Thanks!

@tfeher tfeher left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bixia1 for the comments, I have addressed the issues.

@bixia1 bixia1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this so I have a convenient way to get it to my local tree to do some further checking.

@tensorflow-bot tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 14, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 14, 2020
@tensorflow-bot tensorflow-bot Bot removed the ready to pull PR ready for merge process label Feb 17, 2020

@tfeher tfeher left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bixia1 for the comments! I have addressed the issues. I have confirmed that it works with TRT5, TRT6, and TRT7.

@bixia1 bixia1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving it temporarily for myself to get it to my local tree.

@tensorflow-bot tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 18, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 18, 2020
@tensorflow-bot tensorflow-bot Bot removed the ready to pull PR ready for merge process label Feb 18, 2020
@tensorflow-bot tensorflow-bot Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 18, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 18, 2020
tensorflow-copybara pushed a commit that referenced this pull request Feb 19, 2020
PiperOrigin-RevId: 295843707
Change-Id: I5130c598bcc230748e8d3c6235f279a1b0ec1e9d
@tensorflow-copybara
tensorflow-copybara merged commit 003afb8 into tensorflow:master Feb 19, 2020
@tfeher
tfeher deleted the trt_add_optimization_profiles branch May 16, 2022 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes comp:gpu:tensorrt Issues specific to TensorRT ready to pull PR ready for merge process size:XL CL Change Size:Extra Large

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants