Feature request
Currently, the merged decoder in ORTModelForCausalLM only output a List of 25 OrtNDArrays, 1 for the output logits + 24 from past keys and values.
Could you add the output of hidden_states? It doesn't look like much work; only need to set the key output_hidden_states=True in GPT2LMHeadModel. But this is a very important feature, which enables one of the SOTA contrastive search: https://huggingface.co/blog/introducing-csearch
Thanks!
Motivation
This is a very important feature, which enables the contrastive search: https://huggingface.co/blog/introducing-csearch.
Your contribution
I don't know how this works yet. But it doesn't seems hard.
In the existing contrastive search implementation, this output of hidden_states is simply enabled by setting a key output_hidden_states=True in
/transformers/generation/utils.py L1834
Relavant past PRs https://github.com/huggingface/optimum/pull/647/files, https://github.com/huggingface/optimum/pull/930/files.
Feature request
Currently, the merged decoder in ORTModelForCausalLM only output a List of 25 OrtNDArrays, 1 for the output logits + 24 from past keys and values.
Could you add the output of
hidden_states? It doesn't look like much work; only need to set the keyoutput_hidden_states=TrueinGPT2LMHeadModel. But this is a very important feature, which enables one of the SOTA contrastive search: https://huggingface.co/blog/introducing-csearchThanks!
Motivation
This is a very important feature, which enables the contrastive search: https://huggingface.co/blog/introducing-csearch.
Your contribution
I don't know how this works yet. But it doesn't seems hard.
In the existing contrastive search implementation, this output of hidden_states is simply enabled by setting a key output_hidden_states=True in
/transformers/generation/utils.py L1834
Relavant past PRs https://github.com/huggingface/optimum/pull/647/files, https://github.com/huggingface/optimum/pull/930/files.