System Info
Problem's root cause is in ImageTextToTextPipeline class in the image_text_to_text.py pipeline.
Line 438
model_inputs = self.processor(
images=images, text=text, return_tensors=self.framework, legacy=False, **processing_kwargs
).to(dtype=self.torch_dtype)Notice how legacy is always specified as False?
If you use this model (llava-hf/LLaVA-NeXT-Video-7B-32K-hf) on transfomers==4.47.1 you will get this error because its config specifies to use the class: LlavaNextVideoProcessor from processing_llava_next_video.py and it's __call__ method is not expecting that kwarg.
The quick fix is this:
Modify __call__ (line 101) in processing_llava_next_video.py
from this:
def __call__(
self,
text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]],
images: ImageInput = None,
videos: VideoInput = None,
padding: Union[bool, str, PaddingStrategy] = False,
truncation: Union[bool, str, TruncationStrategy] = None,
max_length: int = None,
return_tensors: Optional[Union[str, TensorType]] = TensorType.PYTORCH,
) -> BatchFeature:to this:
def __call__(
self,
text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]],
images: ImageInput = None,
videos: VideoInput = None,
padding: Union[bool, str, PaddingStrategy] = False,
truncation: Union[bool, str, TruncationStrategy] = None,
max_length: int = None,
return_tensors: Optional[Union[str, TensorType]] = TensorType.PYTORCH,
**kwargs, # <-- this guy
) -> BatchFeature:Notice the unused kwargs at the end. This reflects the pattern used for __init__
which looks like this:
def __init__(
self,
video_processor=None,
image_processor=None,
tokenizer=None,
chat_template=None,
patch_size=None,
vision_feature_select_strategy=None,
video_token="<video>",
image_token="<image>",
num_additional_image_tokens=0,
**kwargs, # <-- this guy
):I ain't got time to step through the PR process, so I hope this helps the HF staff either make this quick patch, or solve the problem at a higher level in the code for image_text_to_text.py.
Who can help?
HF staff
Information
Tasks
- [x ] An officially supported task in the
examples folder (such as GLUE/SQuAD, ...) (image-to-text-to-text)
Reproduction
pipe = pipeline("image-text-to-text", model="llava-hf/LLaVA-NeXT-Video-7B-32K-hf")
messages = {'role': 'user', 'content': [{'type': 'text', 'text': "What's in this image?"}, {'type': 'video'}]}
videos = ["https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"]
out = pipe(text=messages, videos=videos)Expected behavior
No exception raised due to an unexpected kwarg.
System Info
Problem's root cause is in
ImageTextToTextPipelineclass in theimage_text_to_text.pypipeline.Line
438Notice how legacy is always specified as False?
If you use this model (
llava-hf/LLaVA-NeXT-Video-7B-32K-hf) ontransfomers==4.47.1you will get this error because its config specifies to use the class:LlavaNextVideoProcessorfromprocessing_llava_next_video.pyand it's__call__method is not expecting that kwarg.The quick fix is this:
Modify
__call__(line101) inprocessing_llava_next_video.pyfrom this:
to this:
Notice the unused kwargs at the end. This reflects the pattern used for
__init__which looks like this:
I ain't got time to step through the PR process, so I hope this helps the HF staff either make this quick patch, or solve the problem at a higher level in the code for
image_text_to_text.py.Who can help?
HF staff
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...) (image-to-text-to-text)Reproduction
Expected behavior
No exception raised due to an unexpected kwarg.