Skip to content

VLMs: major clean up 🧼 #34502

Merged
zucchini-nlp merged 1 commit into
huggingface:mainfrom
zucchini-nlp:llavas
Jan 8, 2025
Merged

VLMs: major clean up 🧼 #34502
zucchini-nlp merged 1 commit into
huggingface:mainfrom
zucchini-nlp:llavas

Conversation

@zucchini-nlp

@zucchini-nlp zucchini-nlp commented Oct 30, 2024

Copy link
Copy Markdown
Member

What does this PR do?

We have updated all the configs for VLMs on the hub so this PR removes legacy path for models, as it has been there for already 3 releases from v4.44. Also it fixes some stuff that broke on the way, like generating from only text input in LLaVA models

For Video-LLaVA the hub configs cannot be updated as the hub owner has been silent for several mmonths already. And since there is only one model with such architecture, we can hardcode the default values for patch_num and also remove the legacy path

fixes #34824, fixes #35169 and fixes #35450, fixes #35424

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this, we deprecated the legacy path, we can just remove it now no?
I don't remember what we said for for 4.46 but better to go with non legacy now if we can!

@zucchini-nlp

zucchini-nlp commented Oct 30, 2024

Copy link
Copy Markdown
Member Author

We can remove it after updating the files on the hub and that mean we also need to change warning to error so users have chance to see what is the reason for failure.

I think the earliest we can remove is next release, because the blocking PR will prob be merged next week. After that I will take time to update all hub configs. Maybe then we'll wait for the blocking PR and remove all deprecation warnings?

@zucchini-nlp zucchini-nlp changed the title Fix llava tests VLMs: major clean up 🧼 Nov 24, 2024
@ArthurZucker

Copy link
Copy Markdown
Collaborator

Sounds good, let's wait a bit!

@zucchini-nlp

Copy link
Copy Markdown
Member Author

@ArthurZucker i think this can be review now :)

@lcxrocks

Copy link
Copy Markdown

Any updates on this PR? Looking forward to getting llava-hf/llava-v1.6-mistral-7b-hf running for inference without images.
Related to this issue on HF.

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most welcome, related to #35534 were the same is done!

Comment thread src/transformers/models/llava_next_video/modeling_llava_next_video.py Outdated
Comment thread src/transformers/models/llava_next_video/modular_llava_next_video.py Outdated
@ArthurZucker

Copy link
Copy Markdown
Collaborator

O o 👁️ 👁️

@zucchini-nlp

Copy link
Copy Markdown
Member Author

Triggered slow tests on some models, will merge when those pass (or at least it doesn't add up more failing tests...)

@Rocketknight1

Copy link
Copy Markdown
Member

This is amazing! Let me know when it's merged so I can rebase Pixtral onto it

@zucchini-nlp

Copy link
Copy Markdown
Member Author

Oke, so BLIP models apparently have one extra model class which was not modified for some reason, and their official checkpoints on the hub are also not updated. Therefore, I am not adding BLIP to this PR, it will only remove legacy from LLaVAs

Slow tests are passing on my end if compared to the main branch, there are tests failing on main due to tiny logit inconsistencies. I believe it might also be my setup/hardware, we are trying to match the runner's outputs usually

I will merge this in a one hour, just last time slow test runs and done

@zucchini-nlp
zucchini-nlp merged commit d1681ec into huggingface:main Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

5 participants