Using Hugging Face Object Detection Models with Kangas

In this guide we will demonstrate how you can use Kangas to evaluate models from the Hugging Face hub.

We'll to use the OWL-ViT object detection model to make predictions on the on 500 examples from the validation split of the fashionpedia 4 categories dataset.

The dataset has 4 classes. ['accessories', 'clothing', 'bags', 'shoes'].

For each image, we'll log bounding boxes and scores for the individual classes in the dataset. Learn more about annotating images with Kangas here

Install Dependencies

pip install -r requirements.txt

Run the detection script

python detect.py

Start the Kangas Server

kangas server

Exploring Model Predictions

Let's take a look at some of the things we can do with Kangas.

Find examples where the model fails to predict the presence of an object

Here we're going to filter for example images that contain accessories that were not detected by the model.

To do this, we'll run the following query in the filter section.

{"accessories"} != 0 and {"score_accessories"} == 0

object-detection-filtering.mov

Sort by highest and lowest mAP score to find our best and worst predictions

Next, let's sort the images based on their mAP score. This lets us find some examples images where our model is doing well and where it is having difficulty.

object-detection-sorting.mov

Group by an object class to see how it affects mAP

It is likley that our model is detecting some objects well and missing others. We want to see if the mAP score per image is being affected by the presence of an object in that image.

object-detect-groupby.mov

Here we see that the average mAP score is much better in images that contain shoes.

Filtering based on Image metadata

We have logged annotations and labels as metadata in our Images. Kangas lets you filter examples based on this metadata.

Let's take a look at images that contain bags and shoes. Simply add the following line to the filter section.

"gt_bags" in {"Image"}.labels.keys() and "gt_shoes" in {"Image"}.labels.keys()

object-detect-image-metadata-filtering.mov