Add TF-TRT optimization profiles for testing by tfeher · Pull Request #45588 · tensorflow/tensorflow

tfeher · 2020-12-10T23:44:27Z

TF-TRT dynamic shape feature requires the creation of optimization profiles. Currently just a basic optimization profile is created where we set min = opt = max = input_shape. Let's call this optimal profile.

This PR implements an additional optimization profile generation strategies:

kImplicitBatchLike: the batch dimension has min = 1, opt = max = input_batch_size, other dims agree with the input dim

For any shape tensor compatible input the profile is adjusted to have max > min. The goal here is to enable unit testing of converters that have shape input. This requires optimization profile where max != min. Otherwise the shape input tensor is known at build time (equals with min=opt=max), and TRT7 drops it from the list of input bindings. (This problem will probably be corrected in an upcoming version of TRT).

Additionally, profile handling improved in dynamic shape mode. Previously it required to run profile_generation_mode, otherwise engine creation failed. This condition was relaxed: if profile generation mode was not used, then we create a single implicit batch like profile using the current input shape. This improves testing of the trt_engine_op, additionally this will provide an implicit batch mode compatible fall-back for users that do not wish to build the engine in advance.

A follow up PR shall implement additional optimization profile strategies.

Additionally, two bugs are fixed:

The profile index in case of newly created engine was not set,
Error was not handled while setting the input binding size.

Tagging @bixia1 for review and @DEKHTIARJonathan for visibility.

tfeher

Thanks @bixia1 for the review. There is a misunderstanding where the accumulation for min/max profiles should take place: that should go to InitProfiles(), or subroutines called from initprofiles. OptimizationProfileConfig::SetDimensions just enforces consistency with the network input shape. I have updated the docstring for the latter, please have a look.

gbaned · 2020-12-21T17:37:52Z

@tfeher Can you please check @DEKHTIARJonathan's comments and keep us posted ? Thanks!

tensorflowbutler · 2021-01-07T01:05:57Z

It has been 15 days with no activity and the awaiting response label was assigned. Is this PR still valid? Assigning the stalled label. Please comment to reassure me that this is still being worked on.

tfeher

@bixia1 I have updated the PR according our offline discussion. Please have a look.

bixia1 · 2021-01-14T00:26:17Z

One more thing: with this PR, now tensorflow/python/compiler/tensorrt:trt_mode_test fails in TF2 with cuda 11 and TensorRT 7 (--test_env=TF2_BEHAVIOR=1). Here is one subtest and error message:
ExplicitBatchTest.testTfTrtV2_OfflineConversion_DynamicEngine_FP16_NoCalibration

google3.third_party.tensorflow.python.framework.errors_impl.AbortedError: User disallowed engine native segment execution
[[node TRTEngineOp_3_0 (defined at /third_party/py/absl/testing/absltest.py:2494) ]] [Op:__inference_pruned_2404]

tfeher

Thanks @bixia1 for the the review. I have addressed the issues on the C++ side, please have a look. I still have to investigate the failing python test.

gbaned · 2021-01-19T16:52:12Z

@tfeher Can you please resolve conflicts? Thanks!

bixia1 · 2021-01-19T17:03:44Z

We are having some offline discussion on the solution to workaround the TensorRT bugs mentioned in the PR.

tfeher · 2021-02-01T13:44:21Z

@bixia1, I have updated the PR based on our offline discussion: the TRT bug workaround is handled now in any optimization profile strategy by adjusting profile parameters for input tensors that are shape value compatible. Additionally, the error in the python test is fixed. Please have a look.

tfeher

Hi @bixia1, sorry missing the test error, I have corrected that together with the remaining issues.

I have also found an issue with trt_engine_op_test. While from fixing that, I have modifed trt_engine_op not to throw on error in dynamic shape mode if shape information for optimization profiles were not collected. This is achieved by adding a default profile for the current input shape if no profiles were collected.

Additionally, two bugs are fixed:

The profile index in case of newly created engine was not set,
Error was not handled while setting the input binding size.

I have updated the PR description accordingly.

Additionally, profile handling improved in dynamic shape mode: if profile generation mode was not used, then we create a a single implicit batch like profile using the current input shape.

tfeher · 2021-02-09T12:18:00Z

@bixia1, this seems to be stuck. Can you have a look?

bixia1 · 2021-02-09T16:10:54Z

I couldn't figure out what is going on either. I tried to do it manually, and got this error. Trying to seek help internally.
ERROR: Unexpected error (please file a bug against copybara): Multiple entries with same key: tfeher=CONTRIBUTOR and tfeher=CONTRIBUTOR (java.lang.IllegalArgumentException: Multiple entries with same key: tfeher=CONTRIBUTOR and tfeher=CONTRIBUTOR)
210209 08:05:40.487:I 19 [Thread-3] [com.google.chubby.svelte.WaiterThread.endSession:718] shutting down svelte session

google-ml-butler Bot added the size:M CL Change Size: Medium label Dec 10, 2020

google-ml-butler Bot requested a review from joker-eph December 10, 2020 23:44

google-cla Bot added the cla: yes label Dec 10, 2020

tfeher mentioned this pull request Dec 10, 2020

TF-TRT Dynamic Shapes Feature Tracker #45481

Closed

bixia1 reviewed Dec 11, 2020

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

gbaned self-assigned this Dec 11, 2020

gbaned added the comp:gpu:tensorrt Issues specific to TensorRT label Dec 11, 2020

tfeher commented Dec 11, 2020

View reviewed changes

tfeher force-pushed the trt_ut_opt_strategy branch from 33d8ce0 to 40243d8 Compare December 11, 2020 14:54

bixia1 reviewed Dec 11, 2020

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

DEKHTIARJonathan reviewed Dec 12, 2020

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

gbaned added the stat:awaiting response Status - Awaiting response from author label Dec 21, 2020

tensorflowbutler added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jan 7, 2021

tfeher force-pushed the trt_ut_opt_strategy branch from 40243d8 to b768e0f Compare January 13, 2021 15:30

tfeher commented Jan 13, 2021

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved

tfeher force-pushed the trt_ut_opt_strategy branch from b768e0f to d702b80 Compare January 13, 2021 15:36

bixia1 reviewed Jan 14, 2021

View reviewed changes

tfeher commented Jan 14, 2021

View reviewed changes

tfeher force-pushed the trt_ut_opt_strategy branch from d702b80 to bd38eb1 Compare January 14, 2021 21:04

tfeher mentioned this pull request Jan 22, 2021

TF-TRT Improve reshape op converter #40545

Merged

tfeher force-pushed the trt_ut_opt_strategy branch 2 times, most recently from 0a6a7f8 to 1b0b5aa Compare February 1, 2021 13:40

google-ml-butler Bot removed the stale This label marks the issue/pr stale - to be closed automatically if no activity label Feb 1, 2021

google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 1, 2021

kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 1, 2021

gbaned added ready to pull PR ready for merge process and removed ready to pull PR ready for merge process stat:awaiting response Status - Awaiting response from author labels Feb 2, 2021

bixia1 requested changes Feb 2, 2021

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved

bixia1 reviewed Feb 2, 2021

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles_test.cc Resolved

tfeher force-pushed the trt_ut_opt_strategy branch from 1bfb148 to 00903b4 Compare February 3, 2021 17:55

google-ml-butler Bot removed the ready to pull PR ready for merge process label Feb 3, 2021

tfeher force-pushed the trt_ut_opt_strategy branch from 00903b4 to 0e9bc4d Compare February 3, 2021 19:32

tfeher commented Feb 3, 2021

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved

bixia1 reviewed Feb 3, 2021

View reviewed changes

Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved

tfeher force-pushed the trt_ut_opt_strategy branch from 0e9bc4d to 5af4a62 Compare February 4, 2021 09:41

gbaned requested a review from bixia1 February 4, 2021 14:06

bixia1 approved these changes Feb 4, 2021

View reviewed changes

google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 4, 2021

kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 4, 2021

Add TF-TRT optimization profiles for testing

f5c929c

Additionally, profile handling improved in dynamic shape mode: if profile generation mode was not used, then we create a a single implicit batch like profile using the current input shape.

tfeher force-pushed the trt_ut_opt_strategy branch from 5af4a62 to f5c929c Compare February 9, 2021 21:48

google-ml-butler Bot removed the ready to pull PR ready for merge process label Feb 9, 2021

bixia1 approved these changes Feb 9, 2021

View reviewed changes

google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Feb 9, 2021

kokoro-team removed the kokoro:force-run Tests on submitted change label Feb 9, 2021

copybara-service Bot merged commit 702d7ae into tensorflow:master Feb 10, 2021

tfeher mentioned this pull request Feb 11, 2021

[TFTRT] Add Dynamic Shape Testing for ConvertConv3D #46940

Merged

Conversation

tfeher commented Dec 10, 2020 • edited by bixia1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tfeher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbaned commented Dec 21, 2020

Uh oh!

tensorflowbutler commented Jan 7, 2021

Uh oh!

tfeher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bixia1 commented Jan 14, 2021

Uh oh!

tfeher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbaned commented Jan 19, 2021

Uh oh!

bixia1 commented Jan 19, 2021

Uh oh!

tfeher commented Feb 1, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tfeher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tfeher commented Feb 9, 2021

Uh oh!

bixia1 commented Feb 9, 2021

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

tfeher commented Dec 10, 2020 •

edited by bixia1

Loading