First of all, thank you very much for making our lives easier with the work you do at huggingface, congratulations!
We have a model based on the encoder-decoder architecture, made up of 2 RoBERTa. The model works quite well, but unfortunately its inference time is quite high (about 400ms to generate a sentence of about 7 tokens). We would like to reduce that time, and have opted for ONNX and Optimum.
We have managed to export the model to ONNX, generating an encoder_model.onnx, a decoder_model.onnx, and a decoder_with_past_model.onnx. We can load this exported model using ORTModelForSeq2SeqLM.from_pretrained(·), and therefore, it allows us to use the generate(·) method. The problem is that the model in ONNX format is 2 times slower than the model without exporting.
We have seen some similar issues: #365, #362. We believe that the exported model is simply in another format, and therefore does not necessarily have to be faster than the base model. Because of all this, our last step has been to use ORTOptimizer.from_pretrained(·) to apply graph optimization (operator fusion and constant folding) to speed up latency and inference. Unfortunately, we have not achieved the latter. The ORTOptimizer.from_pretrained(·) method expects, in addition to the ONNX model, a config.json file. We do not have this file, since it is not generated when exporting the model to ONNX. We have made several attempts to generate and save configuration files, even with the base model's config.json, but not even crossing our fingers we have succeeded 😢
We would appreciate if someone (@lewtun?) could point us to some lines so we can continue...
Thank you very much. Best,
Jesus.
First of all, thank you very much for making our lives easier with the work you do at huggingface, congratulations!
We have a model based on the encoder-decoder architecture, made up of 2 RoBERTa. The model works quite well, but unfortunately its inference time is quite high (about 400ms to generate a sentence of about 7 tokens). We would like to reduce that time, and have opted for ONNX and Optimum.
We have managed to export the model to ONNX, generating an encoder_model.onnx, a decoder_model.onnx, and a decoder_with_past_model.onnx. We can load this exported model using
ORTModelForSeq2SeqLM.from_pretrained(·), and therefore, it allows us to use thegenerate(·)method. The problem is that the model in ONNX format is 2 times slower than the model without exporting.We have seen some similar issues: #365, #362. We believe that the exported model is simply in another format, and therefore does not necessarily have to be faster than the base model. Because of all this, our last step has been to use
ORTOptimizer.from_pretrained(·)to apply graph optimization (operator fusion and constant folding) to speed up latency and inference. Unfortunately, we have not achieved the latter. TheORTOptimizer.from_pretrained(·)method expects, in addition to the ONNX model, a config.json file. We do not have this file, since it is not generated when exporting the model to ONNX. We have made several attempts to generate and save configuration files, even with the base model's config.json, but not even crossing our fingers we have succeeded 😢We would appreciate if someone (@lewtun?) could point us to some lines so we can continue...
Thank you very much. Best,
Jesus.