Skip to content

Load and select optimization profiles for static TRT engine - #52047

Merged
copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
tfeher:trt_static_engine_profile_handling
Sep 22, 2021
Merged

Load and select optimization profiles for static TRT engine#52047
copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
tfeher:trt_static_engine_profile_handling

Conversation

@tfeher

@tfeher tfeher commented Sep 17, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR implements TRT optimization profile handling in case of static engine:

  • Initializes the optimization profile helper class from ICudaEngine
  • Select the optimization profile for the input shape

This is PR is necessary to use dynamic shape mode with static engines.

Tracker #52012

Tagging @bixia1 for review.

@google-ml-butler google-ml-butler Bot added the size:S CL Change Size: Small label Sep 17, 2021
@google-ml-butler google-ml-butler Bot added the awaiting review Pull request awaiting review label Sep 17, 2021
@google-cla google-cla Bot added the cla: yes label Sep 17, 2021
@tfeher tfeher changed the title Load and select optimization profiles for static Load and select optimization profiles for static TRT engine Sep 17, 2021
@tfeher tfeher mentioned this pull request Sep 17, 2021
Merged
@bixia1 bixia1 assigned bixia1 and unassigned bixia1 Sep 17, 2021
@bixia1
bixia1 self-requested a review September 17, 2021 21:41
@bixia1 bixia1 added the comp:gpu:tensorrt Issues specific to TensorRT label Sep 17, 2021
@gbaned gbaned self-assigned this Sep 20, 2021
@tfeher
tfeher force-pushed the trt_static_engine_profile_handling branch from 61a05c6 to a4a0bb3 Compare September 20, 2021 08:25
@tfeher

tfeher commented Sep 20, 2021

Copy link
Copy Markdown
Contributor Author

Improved trt_engine_op_test to cover the static engine path (including profile handling).

The logic of handling and creating engines in TrtEngineOp::GetEngine needs some simplification. I propose to do that in a follow up PR, so that here we have a clear overview of the changes.

@bixia1 ready for review

@tfeher
tfeher marked this pull request as ready for review September 20, 2021 08:32
@tfeher
tfeher force-pushed the trt_static_engine_profile_handling branch 2 times, most recently from 4e84324 to 5c25d06 Compare September 20, 2021 10:25
bixia1
bixia1 reviewed Sep 20, 2021
View reviewed changes
Comment thread tensorflow/compiler/tf2tensorrt/convert/convert_graph.h Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op_test.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op_test.cc Resolved
@sanjoy
sanjoy removed their request for review September 20, 2021 20:29
@tfeher
tfeher force-pushed the trt_static_engine_profile_handling branch from 5c25d06 to 0ea8aeb Compare September 21, 2021 06:32
@tfeher
tfeher force-pushed the trt_static_engine_profile_handling branch from 0ea8aeb to 6cd3e3c Compare September 21, 2021 06:44
tfeher
tfeher commented Sep 21, 2021
View reviewed changes

@tfeher tfeher left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thansk @bixia1 for the comments, I have fixed the issues!

Comment thread tensorflow/compiler/tf2tensorrt/convert/convert_graph.h Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op_test.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op_test.cc Resolved
@google-ml-butler google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Sep 21, 2021
@gbaned gbaned removed awaiting review Pull request awaiting review kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Sep 22, 2021
@gbaned gbaned added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Sep 22, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Sep 22, 2021
@copybara-service
copybara-service Bot merged commit f46b34f into tensorflow:master Sep 22, 2021
@google-ml-butler google-ml-butler Bot removed the ready to pull PR ready for merge process label Sep 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bixia1 bixia1 bixia1 approved these changes
@joker-eph joker-eph Awaiting requested review from joker-eph
@sherhut sherhut Awaiting requested review from sherhut

Assignees

@gbaned gbaned

Labels

cla: yes comp:gpu:tensorrt Issues specific to TensorRT size:S CL Change Size: Small

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tfeher @bixia1 @kokoro-team @gbaned