Enable explicit batch mode with optimization profiles by pooyadavoodi · Pull Request #36080 · tensorflow/tensorflow

pooyadavoodi · 2020-01-21T01:12:07Z

This PR adds a new experimental feature to TF-TRT.
This features enables the explicit batch mode in TF-TRT which comes with two main advantages: 1) Batch dimension is not a special dimension in this mode, which means a single TRT network can have layers with different batch sizes and more importantly reshape layers can change the batch dimension dynamically, 2) TRT networks can be created with dynamic shapes (-1 dims), while optimization profiles are generated automatically out of the users data to hint TRT for optimizing the built engines.
Currently this feature does not support building TRT engines during runtime, which means the only single engine that is built during build() can be used during runtime. This makes it important for users to feed data to build() with all interesting shapes so that the single engine can execute those shapes.
Currently this feature does not support INT8 calibration.
This feature is not supposed to change any of the existing behavior of TF-TRT in any of the various modes.
It is expected to use this feature for elementwise ops. We are soon going to enable all the existing converters to work with this feature.

An example on how to use this feature:

    root = _model()
    save.save(root, input_saved_model_dir,
              {signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: root.run})

    rewriter_config_with_trt = rewriter_config_pb2.RewriterConfig()
    rewriter_config_with_trt.optimizers.extend(
        ["constfold", "layout", "constfold"])
    rewriter_config_with_trt.meta_optimizer_iterations = (
        rewriter_config_pb2.RewriterConfig.ONE)
    optimizer = rewriter_config_with_trt.custom_optimizers.add()
    rewriter_config_with_trt.custom_optimizers.add().name = "constfold"
    optimizer.name = "TensorRTOptimizer"
    optimizer.parameter_map["minimum_segment_size"].i = 1
    optimizer.parameter_map["is_dynamic_op"].b = True
    optimizer.parameter_map["maximum_cached_engines"].i = 1
    optimizer.parameter_map["use_implicit_batch"].b = False

    conversion_params = trt_convert.DEFAULT_TRT_CONVERSION_PARAMS._replace(
        rewriter_config_template=rewriter_config_with_trt)
    converter = trt_convert.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir,
                                    conversion_params=conversion_params)
    converter.convert()

    input_shapes = [(2, 4), (3, 9), (64, 128)]
    def my_input_fn():
      for x in input_shapes:
        yield (np.random.normal(size=x).astype(np.float32),)

    converter.build(input_fn=my_input_fn)

    output_saved_model_dir = self.mkdtemp()
    converter.save(output_saved_model_dir=output_saved_model_dir)

The EngineContext in TRT cache resource now handles a vector of contexts, one for each profile.

…g.bzl TF doesn't compile without this commit.

gbaned · 2020-02-11T11:21:58Z

@pooyadavoodi Can you please resolve conflicts? Thanks!

…nto add_profiles_merged

googlebot · 2020-02-11T15:17:38Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

tfeher · 2020-02-11T15:19:58Z

Thanks @gbaned for the message, I have updated the PR. Please note that PRs #36660 and #36664 have to merged before this one.

gbaned · 2020-02-12T04:08:31Z

@pooyadavoodi, @tfeher Can you please sign cla ? Thank you!

tfeher · 2020-02-12T18:40:17Z

@googlebot I consent

tfeher · 2020-02-13T13:53:05Z

The CLA is failing because the original PR author has left the company and is not on the corporate CLA anymore. Since this PR is anyways broken into smaller pieces, I have taken the last piece still contained here and moved into a separate PR #36729.

This PR can be closed.

For completeness, here is a list of PRs that were created out of this one:
1 Enable explicit batch mode #36379
2. Improve binding index query #36434
3. Add binding size specification #36435
4. Define networks with dynamic shapes #36439
5. Add optimizaton profiles #36660
6. Execution context managment #36664
7. ~~Test deserialization of opt profiles~~ (will be added independently of the rest in a separate PR).
8. TensorRT profile generation mode #36729

gbaned · 2020-02-14T08:25:58Z

Thanks @tfeher
Closing this PR, as its change has been moved to #36729

Pooya Davoodi and others added 30 commits January 20, 2020 15:17

Add profile_generation_mode attribute and adjust build() accordingly

95b638f

Move TRT VectorTensorShapeHasher trt_lru_cache.h to convert/utils.h

d28d6e7

Introduce TrtShapeOptimizationProfiles class

5faea55

Add DebugString functions for vectors of shapes

48b3a5c

Correct include file names and initProfiles args

34299a2

Add profile collection mode to TRTEngineOp.

10165f6

The EngineContext in TRT cache resource now handles a vector of contexts, one for each profile.

Use profiles in ConvertGraphDefToEngine

2e89ee1

Revert to platform:build_config.bzl from platform:default/build_confi…

a4c7f5f

…g.bzl TF doesn't compile without this commit.

Add missing optimization profile args to convert_nodes_test

314ec0b

Use enqueuev2 only with explicit batch mode

71bfdaa

Use explicit batch mode API only if use_implicit_batch=false

bf393d7

Fix typos

4c1a172

Configure profiles only in explicit batch mode

01ead47

Add test for TrtShapeOptimizationProfile

5655f19

Fix shape_optimization_profiles test by clearing exec_context

97c8cd9

Keep logger alive to avoid segfault in trt_engine_resource_ops test

20d1b9a

Add static mode test for TrtShapeOptimizationProfile

a964e32

Rename opt_profile_idx to trt_context_idx

fe54b29

Enable profiles in python and organize profile creation

d8b037a

Fix failing test for TrtShopeOptimizationProfiles

108fd51

Test switching optimization profile

f4ed782

Build TRT network with dynamic shapes

b255634

Remove profile_map_

931b494

Restore missing closing bracket

7c2d7f4

Add utility function te query number of inputs for a TRT engine

e50ad6b

Add RestoreProfiles method to TrtShapeOptimizationProfiles

1e0a0c7

Restore profiles and ExecutionContexts after deserialization

6c5cb4b

Correcty typo in function name

5acbfbf

Correcty typo in function name

ba9d942

Only init profiles once

204bcc5

tfeher added 10 commits February 7, 2020 11:21

Fix style

fb390ee

Simplify storing input shapes TRT node

ff4f45d

Update is_dynamic check and its comments in SqueezeTensor

28c600b

Improve comments log/error messages

ec82b4c

Allow scalar input in explicit batch mode

d036896

Improve comments

b850ea0

Use boolean mask to define unknown shapes for TF-TRT integration test

3aae93a

Skip only the base class in trt_mode_test.py

f7eb8f8

Improve comments

2370086

Introduce BuildParamsWithMask to specify dynamic shapes tests

b75a622

tfeher added 5 commits February 11, 2020 12:22

Introduce TrtShapeOptimizationProfile class and use it in TRTEngineOp

fd0d5ad

Restore profiles and ExecutionContexts after deserialization

7ed3021

Add GetNumberOfEngineInputs function

a5ac44a

Update TRT dynamic shape tests

449d2e0

Improve comments and style

1ab2281

tfeher mentioned this pull request Feb 11, 2020

Add TensorRT optimization profiles #36660

Merged

tfeher added 3 commits February 11, 2020 14:36

Merge remote-tracking branch 'tfeher/trt_add_optimization_profiles' i…

d584740

…nto add_profiles_merged

Edit style and delete unused code

08dd9a6

Remove profile deserialization test (will be added in separate PR).

5e33341

tfeher mentioned this pull request Feb 11, 2020

Execution context management for TensorRT profiles #36664

Merged

googlebot added cla: no and removed cla: yes labels Feb 11, 2020

gbaned closed this Feb 14, 2020

Navigation Menu

Repository navigation

Enable explicit batch mode with optimization profiles - #36080

Enable explicit batch mode with optimization profiles#36080
pooyadavoodi wants to merge 95 commits into
tensorflow:masterfrom
pooyadavoodi:add_profiles

pooyadavoodi commented Jan 21, 2020 •

edited

Loading

Uh oh!

gbaned commented Feb 11, 2020

Uh oh!

googlebot commented Feb 11, 2020

Uh oh!

tfeher commented Feb 11, 2020 •

edited

Loading

Uh oh!

gbaned commented Feb 12, 2020

Uh oh!

tfeher commented Feb 12, 2020

Uh oh!

tfeher commented Feb 13, 2020

Uh oh!

gbaned commented Feb 14, 2020

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

pooyadavoodi commented Jan 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaned commented Feb 11, 2020

Uh oh!

googlebot commented Feb 11, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

tfeher commented Feb 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbaned commented Feb 12, 2020

Uh oh!

tfeher commented Feb 12, 2020

Uh oh!

tfeher commented Feb 13, 2020

Uh oh!

gbaned commented Feb 14, 2020

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pooyadavoodi commented Jan 21, 2020 •

edited

Loading

tfeher commented Feb 11, 2020 •

edited

Loading