Skip to content

CLIP score improvements#2978

Merged
SkafteNicki merged 37 commits into
masterfrom
feature/clip_improvement
Mar 12, 2025
Merged

CLIP score improvements#2978
SkafteNicki merged 37 commits into
masterfrom
feature/clip_improvement

Conversation

@SkafteNicki

@SkafteNicki SkafteNicki commented Feb 27, 2025

Copy link
Copy Markdown
Collaborator

What does this PR do?

Fixes #2935
Fixes #2906
Fixes #2883

Add the possibility for more multimodel models that supports longer captions than the original clip model.

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?
PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃


📚 Documentation preview 📚: https://torchmetrics--2978.org.readthedocs.build/en/2978/

@SkafteNicki SkafteNicki added the enhancement New feature or request label Feb 27, 2025
@SkafteNicki SkafteNicki added this to the v1.6.0 milestone Feb 27, 2025
@SkafteNicki SkafteNicki self-assigned this Feb 27, 2025
@SkafteNicki SkafteNicki modified the milestones: v1.6.0, v1.7.0 Feb 28, 2025
@SkafteNicki
SkafteNicki marked this pull request as ready for review March 3, 2025 12:34
@mergify mergify Bot added the has conflicts label Mar 3, 2025
@mergify mergify Bot removed the ready label Mar 5, 2025
@Borda

Borda commented Mar 5, 2025

Copy link
Copy Markdown
Collaborator

seems like just the docs are failing with warnings?

@mergify mergify Bot added the ready label Mar 5, 2025
@SkafteNicki

Copy link
Copy Markdown
Collaborator Author

@Borda no there are also failing gpu tests (the test for clip score only runs on gpu, else it would take too long on cpu)

@Borda

Borda commented Mar 5, 2025

Copy link
Copy Markdown
Collaborator

no there are also failing gpu tests (the test for clip score only runs on gpu, else it would take too long on cpu)

GPU testing is fine :)

>           raise ValueError(f"pic should not have > 4 channels. Got {pic.shape[-1]} channels.")
E           ValueError: pic should not have > 4 channels. Got 64 channels.

Let's try to pass it as PIL image instead?
Or it has reversed order of dimensions width x height x channels?

Comment thread src/torchmetrics/functional/multimodal/clip_score.py Outdated
@SkafteNicki

Copy link
Copy Markdown
Collaborator Author

@Borda is there a way to get more info from azure? it is hard to debug without what is going wrong

@mergify mergify Bot removed the ready label Mar 11, 2025
@mergify mergify Bot added the ready label Mar 11, 2025
@SkafteNicki
SkafteNicki merged commit d332ae3 into master Mar 12, 2025
@SkafteNicki
SkafteNicki deleted the feature/clip_improvement branch March 12, 2025 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

3 participants