Repository navigation
Add TensorRT optimization profiles - #36660
Merged
tensorflow-copybara merged 16 commits intoFeb 19, 2020
Merged
tensorflow-copybara merged 16 commits into
tensorflow-copybara merged 16 commits into
Conversation
Contributor
|
@tfeher Can you please resolve conflicts? Thanks! |
bixia1
reviewed
Feb 14, 2020
tfeher
commented
Feb 14, 2020
bixia1
approved these changes
Feb 14, 2020
bixia1
left a comment
Contributor
There was a problem hiding this comment.
Approving this so I have a convenient way to get it to my local tree to do some further checking.
bixia1
requested changes
Feb 14, 2020
tfeher
commented
Feb 17, 2020
bixia1
approved these changes
Feb 18, 2020
bixia1
left a comment
Contributor
There was a problem hiding this comment.
Approving it temporarily for myself to get it to my local tree.
bixia1
requested changes
Feb 18, 2020
bixia1
approved these changes
Feb 18, 2020
tensorflow-copybara
pushed a commit
that referenced
this pull request
Feb 19, 2020
PiperOrigin-RevId: 295843707 Change-Id: I5130c598bcc230748e8d3c6235f279a1b0ec1e9d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds optimization profile handling to TF-TRT and enables inference using the experimental dynamic shapes mode (where dynamic shapes mode = explicit batch mode with dynamic input shapes).
This PR is not supposed to change the existing behavior in implicit batch mode.
For each unknown input dimension, a shape optimization profile describes what is the expected [min, max] range for that dimension, and also define an optimum value. "The builder selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are valid for all input tensor sizes in the valid range between minimum and maximum dimensions."
To create an engine with dynamic shapes one must define at least one optimization profile. In the current PR we define only a single optimization profile, using the shape of the concrete input data during the first inference call.
The real advantage of dynamic shapes is to have a single engine that can execute the network with different input shapes. Unfortunately it is not possible to demonstrate this with the current PR. Although the
TrtShapeOptimizationProfileclass is designed to manage multiple profiles, we still need to enable profile generation mode during build phase, to take the full advatage of dynamic shapes mode. That feature is implemented in PR #36080.