Skip to content

[TF:TRT] INT8 calibration in dynamic shape mode - #55166

Merged
copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
tfeher:calibration_in_dynamic_shape
Apr 26, 2022
Merged

[TF:TRT] INT8 calibration in dynamic shape mode#55166
copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
tfeher:calibration_in_dynamic_shape

Conversation

@tfeher

@tfeher tfeher commented Mar 8, 2022

Copy link
Copy Markdown
Contributor

This PR enables INT8 conversion with calibration in dynamic shape mode.

  • The profile collection logic is moved before the calibration step in TRTEngineOP, because TRT optimization profiles have to be defined before we can calibrate an engine.
  • INT8 test are now enabled in V2 mode both with and without dynamic shapes.

Tracker: #45481

@google-ml-butler google-ml-butler Bot added the size:L CL Change Size: Large label Mar 8, 2022
@google-cla

google-cla Bot commented Mar 8, 2022

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

For more information, open the CLA check for this pull request.

@tfeher

tfeher commented Mar 8, 2022

Copy link
Copy Markdown
Contributor Author

Tagging @bixia1 for review. Thanks for @vict-guo for the initial implementation.

A follow up PR is planned to refactor trt_engine_op.cc: there are repetitive steps in GetEngine / BuildEngine / AllocateCalibrationResources. These shall be factored out into helper functions.

@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from dc67e08 to 1bea265 Compare March 8, 2022 19:49
@gbaned
gbaned requested a review from bixia1 March 9, 2022 11:15
@google-ml-butler google-ml-butler Bot added the awaiting review Pull request awaiting review label Mar 9, 2022
bixia1
bixia1 reviewed Mar 11, 2022
View reviewed changes
Comment thread tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc Resolved
Comment thread tensorflow/python/compiler/tensorrt/test/trt_engine_op_shape_test.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
@tensorflowbutler tensorflowbutler removed the awaiting review Pull request awaiting review label Mar 13, 2022
@gbaned

gbaned commented Mar 14, 2022

Copy link
Copy Markdown
Contributor

@tfeher Can you please check @bixia1's comments and keep us posted ? Thanks!

@gbaned gbaned added the stat:awaiting response Status - Awaiting response from author label Mar 14, 2022
@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from 1bea265 to 0af5b8e Compare March 15, 2022 16:06
tfeher
tfeher commented Mar 15, 2022
View reviewed changes

@tfeher tfeher left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bixia1 for the review, I have addressed the issues.

Comment thread tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc Resolved
Comment thread tensorflow/python/compiler/tensorrt/test/trt_engine_op_shape_test.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
Comment thread tensorflow/python/compiler/tensorrt/trt_convert.py Resolved
bixia1
bixia1 reviewed Mar 16, 2022
View reviewed changes
Comment thread tensorflow/python/compiler/tensorrt/test/tf_trt_integration_test_base.py Resolved
Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from 0af5b8e to b55e628 Compare March 17, 2022 18:56
tfeher
tfeher commented Mar 17, 2022
View reviewed changes

@tfeher tfeher left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bixia1 for the additional comments, I have addressed those.

We have discussed offline about the possibility of adding the calibration_input_fn arg to the build() method. I propose to start a separate PR for that, and continue the discussion there.

Comment thread tensorflow/python/compiler/tensorrt/test/tf_trt_integration_test_base.py Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc Resolved
Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.cc Resolved
bixia1
bixia1 reviewed Mar 17, 2022
View reviewed changes

@bixia1 bixia1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! Just one minor comment and then it should be ready for landing.

Comment thread tensorflow/compiler/tf2tensorrt/utils/trt_shape_optimization_profiles.h Resolved
@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from b55e628 to 934cc10 Compare March 18, 2022 15:35
@tfeher

tfeher commented Mar 18, 2022

Copy link
Copy Markdown
Contributor Author

@bixia1 I have fixed the issues.

@google-ml-butler google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Mar 18, 2022
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 18, 2022
@tensorflowbutler tensorflowbutler removed the stat:awaiting response Status - Awaiting response from author label Mar 20, 2022
@gbaned gbaned added comp:gpu:tensorrt Issues specific to TensorRT ready to pull PR ready for merge process and removed ready to pull PR ready for merge process labels Mar 21, 2022
@bixia1

bixia1 commented Mar 22, 2022

Copy link
Copy Markdown
Contributor

There are failing tests, here are the log
fail1.log
fail2.log
.

@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from 934cc10 to 3f2216d Compare March 25, 2022 19:12
@google-ml-butler google-ml-butler Bot removed the ready to pull PR ready for merge process label Mar 25, 2022
@tensorflowbutler tensorflowbutler removed the awaiting review Pull request awaiting review label Apr 1, 2022
@tfeher tfeher mentioned this pull request Apr 5, 2022
Closed
@google-ml-butler google-ml-butler Bot added the kokoro:force-run Tests on submitted change label Apr 7, 2022
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Apr 7, 2022
@gbaned gbaned added ready to pull PR ready for merge process and removed ready to pull PR ready for merge process labels Apr 18, 2022
@gbaned

gbaned commented Apr 19, 2022

Copy link
Copy Markdown
Contributor

@tfeher Can you please resolve conflicts? Thank you!

@gbaned gbaned added stat:awaiting response Status - Awaiting response from author and removed ready to pull PR ready for merge process labels Apr 19, 2022
@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from e7d82de to 49cb98c Compare April 20, 2022 08:59
@gbaned
gbaned requested a review from bixia1 April 20, 2022 12:01
@google-ml-butler google-ml-butler Bot added the awaiting review Pull request awaiting review label Apr 20, 2022
@gbaned gbaned removed stat:awaiting response Status - Awaiting response from author awaiting review Pull request awaiting review labels Apr 20, 2022
@google-ml-butler google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Apr 20, 2022
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Apr 20, 2022
@tfeher
tfeher force-pushed the calibration_in_dynamic_shape branch from 49cb98c to f9467f4 Compare April 21, 2022 21:48
@google-ml-butler google-ml-butler Bot removed the ready to pull PR ready for merge process label Apr 21, 2022
@tfeher

tfeher commented Apr 21, 2022

Copy link
Copy Markdown
Contributor Author

@bixia1 The CI failure was caused by an issue in the algorithm selector in TRT 7.x. We had a workaround in place, but it was only active for TRT 7.1. I have adjusted it make it work with TRT 7.2 as well

@google-ml-butler google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Apr 22, 2022
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Apr 22, 2022
@copybara-service
copybara-service Bot merged commit 987cd87 into tensorflow:master Apr 26, 2022
@meena-at-work meena-at-work mentioned this pull request Apr 26, 2022
Merged
@tilakrayal tilakrayal mentioned this pull request Aug 20, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bixia1 bixia1 bixia1 approved these changes

Assignees

@gbaned gbaned

Labels

comp:gpu:tensorrt Issues specific to TensorRT ready to pull PR ready for merge process size:L CL Change Size: Large

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tfeher @gbaned @bixia1 @kokoro-team @tensorflowbutler