Fix vision encoder decoder attention implementation pick#31203
Conversation
fxmarty
changed the title
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
This highlights a wider issue with how attention is selected for models through configs. Having things autoset breaks a lot of the "don't do things by magic" convention that we have in transformers and this has already caused quite a few issues / surpising behaviour within the library. Instead, I think we should completely re-think how attention is passed through and selected. |
|
agreed |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
3 participants
Noticed this bug while making the CI pass for #30138.
Essentially,
VisionEncoderDecoderModel.from_pretrained(xxx)does not correctly load with SDPA (e.g. with bert, deit) while it should be. The reason is thatVisionEncoderDecoderModeldoes not have a class attribute_supports_sdpa, and we can not generally set_supports_sdpa = True(as multiple archs are supported). Thus, infrom_pretrainedlogic to pick the attention implementation, SDPA is never picked.We need to rely on the encoder/decoder classes to rightfully pick the attention implementation.