Define TensorRT network with dynamic shapes - #36439

This PR builds enables TRT network creation using dynamic shapes.

Dynamic shapes can be only used in the explicit batch mode (experimental). This PR only implements TensorRT network definition using dynamic shapes. To actually build the engine and run inference, one needs to specify optimization profiles. That is implemented in the next PR. Therefore TRT Engine creation is expected to fail if explicit batch mode is selected and the input tensors have unknown dimensions. This is tested in trt_engine_op_test.

This PR is not supposed to change any existing behavior: inference in implicit batch mode, or in explicit mode with known (static) shapes should still work as before.

Note
With the introduction of dynamic shapes, TF-TRT will have two different strategies to handle input with different shapes. (Confusingly the doc sometimes refers both as dynamic shapes). From the user point of view they are similar: the aim is to use TensorRT inference for inputs whose shape is unknown at the time when we call trt.convert().

• The existing strategy is dynamic engine creation. It means that TensorRT engines can be created during runtime to handle different input shapes.
  

  tensorflow/tensorflow/python/compiler/tensorrt/trt_convert.py

  Lines 1210 to 1221 in 97b4c2c

  	If is_dynamic_op is True, each TRTEngineOp will contain a serialized
  	subgraph GraphDef, which will be converted to a TRT engine at execution time
  	and the TRT engine will be cached for future usage. A new TRT engine will be
  	created each time when none of the cached engines match the input shapes. If
  	it fails to execute the TRT engine or the number of cached engines reaches
  	maximum_cached_engines, the op will fall back to call the corresponding TF
  	function.
  	If is_dynamic_op is False, each TRTEngineOp will contain a serialized TRT
  	engine created from the corresponding subgraph. No more engines will be
  	created on the fly, and the op will fall back to call the corresponding TF
  	function when it fails to execute the engine.

  
  

  tensorflow/tensorflow/python/compiler/tensorrt/trt_convert.py

  Lines 136 to 142 in 97b4c2c

  	# Whether to generate dynamic TRT ops which will build the TRT network
  	# and engine at run time.
  	# i.e. Since TensorRT version < 6.0 does not support dynamic dimensions
  	# other than the batch dimension, when the TensorFlow graph has a
  	# non-batch dimension of dynamic size, we would need to enable this
  	# option. This option should be set to True in TF 2.0.
  	"is_dynamic_op",

  
  Dynamic engine creation is the only supported ongine creation in TF v2. See the relevant doc:
  

  tensorflow/tensorflow/python/compiler/tensorrt/trt_convert.py

  Lines 858 to 859 in 97b4c2c

  	Note that in V2, is_dynamic_op=False is not supported, meaning TRT engines
  	will be built only when the corresponding TRTEngineOp is executed. But we

• This PR introduces dynamic input shapes (supported since TensorRT version 6), which is another way to deal with input data that has a shape not known at conversion time: we can define the TensorRT network with unknown input shapes. This way a single engine can handle input with different sizes. We still need to know the expected range of each input dimension (shape optimization profiles) at engine creation time.