Skip to content

Fuyu processor: box coordinates#27083

Closed
pcuenca wants to merge 2 commits into
huggingface:fuyu_follow_up_image_processingfrom
pcuenca:fuyu-processor-coordinates
Closed

Fuyu processor: box coordinates#27083
pcuenca wants to merge 2 commits into
huggingface:fuyu_follow_up_image_processingfrom
pcuenca:fuyu-processor-coordinates

Conversation

@pcuenca

@pcuenca pcuenca commented Oct 26, 2023

Copy link
Copy Markdown
Member

What does this PR do?

PoC to post-process box coordinates returned by the model. The following should work:

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = FuyuForCausalLM.from_pretrained(model_id, device_map=device, torch_dtype=dtype)
processor = FuyuProcessor(image_processor=FuyuImageProcessor(), tokenizer=tokenizer)

# Prompt appropriate for bounding box detection
text = "statistics"
prompt = f"When presented with a box, perform OCR to extract text contained within it. If provided with text, generate the corresponding bounding box.\n{text}"
image = Image.open("screen2words_ui_example.png")

model_inputs = processor(text=prompt, images=[image]).to(device)    
generation_output = model.generate(**model_inputs, max_new_tokens=40)

results = processor.post_process_box_coordinates(generation_output, target_sizes=torch.Tensor([image.size[::-1]]))

# TODO: maybe unbox the <box> here as well??
decoded = processor.decode(results[0], skip_special_tokens=True)
print(decoded)
# <box>60, 124, 100, 268</box>

I'd like to validate whether this approach is appropriate, what do you think @amyeroberts? If it is, then we can:

  • Support point coordinates too.
  • Perform the reverse transformations on input prompts. There's already code in the processor for that purpose, I think we could maybe simplify it a bit.
  • Maybe provide an optional resizing + padding pre-processing step for images, only for the bounding box detection task. According to our conversations with the original authors (and our tests), this task only works properly when the input image size is close to (1080, 1920). The correct approach is to downscale larger images, and then pad to match that size.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts, @molbap

@pcuenca
pcuenca requested review from amyeroberts and molbap October 26, 2023 14:00
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@amyeroberts amyeroberts left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pcuenca Nice! LGTM :)

My vote would be not to add unboxing the <box> in decode, as it's a very common method with a standard API and traditionally used as the inverse to encoding with the tokenizer.

except:
return tokens

if bbox_end_pos != bbox_start_pos + 5:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the 5 come from here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, should have explained!

The model returns coordinates in the following format:

  • Beginning of bbox delimiter, which is a single token id.
  • 4 token ids corresponding to the scaled coordinate numbers, without any delimiters.
  • End of bbox delimiter, another single token id.

So we find the begin and end delimiters, and verify that there are exactly 4 token ids in-between.

The same approach is taken for 2d point coordinates, I'll incorporate them now as well as the reverse pre-processing transformation.

Thanks a lot for the quick review and comments @amyeroberts!

@adhikjoshi

Copy link
Copy Markdown

Can I use AutoModelForCausalLM and AutoProcessor instead of using Fuyu-specific pipelines?

@molbap
molbap deleted the branch huggingface:fuyu_follow_up_image_processing November 2, 2023 11:25
@molbap molbap closed this Nov 2, 2023
@amyeroberts

Copy link
Copy Markdown
Contributor

Hi @adhikjoshi, yes, you can load both the Fuyu model and its processor using AutoModelForCausalLM and AutoProcessor respectively

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants