[Feature]: miniCOIL: no way to control input sequence length (and VRAM usage) natively #544

What feature would you like to request?

As I understand it, the interface of the miniCOIL model does not allow users to control the length of tokens passed to the base transformer model. As miniCOIL uses an 8k context model as a backbone, users will be suddenly running out of GPU memory when processing large texts.

Adding a max_sequence_length or similar parameter would be nice.

An easy option to avoid OOMs is to truncate the input string, but it is wrong on many levels.

Is there any additional information you would like to provide?

No response