Feature request

Some models like Deformable DETR rely on custom CUDA kernels to be compiled as seen here.

Currently these are compiled when importing the Transformers library, but this needs to happen later, when initializing the models.

All custom CUDA kernels are defined here: https://github.com/huggingface/transformers/tree/main/src/transformers/kernels

Motivation

This is pretty important to fix actually as currently running make fixup on machines that have ninja installed will compile all these kernels before running the quality checks, making it super slow.. thanks @younesbelkada for the info

Your contribution

Not sure I can help with this, the current workaround is simply removing ninja from the environment