Describe the feature request
It would be great to have the option to provide pre-optimised TensorRT engine plans to ORT.
Describe scenario use case
Using TensorRT in standalone, e.g. trtexec, allows for much more precise control over engine compilation.
Currently, only a very small amount of TRT features is accessible through the ORT API, but most importantly, there is no way to ship precompiled engines with an application; Engine compilations need to be done on the user machine, resulting in a bad user experience.
Also, how (TRT) caching is handled internally is not very transparent, in my opinion, and might lead to unnecessary engine recompilations.
Describe the feature request
It would be great to have the option to provide pre-optimised TensorRT engine plans to ORT.
Describe scenario use case
Using TensorRT in standalone, e.g. trtexec, allows for much more precise control over engine compilation.
Currently, only a very small amount of TRT features is accessible through the ORT API, but most importantly, there is no way to ship precompiled engines with an application; Engine compilations need to be done on the user machine, resulting in a bad user experience.
Also, how (TRT) caching is handled internally is not very transparent, in my opinion, and might lead to unnecessary engine recompilations.