Store cached models in location compatible with `huggingface-cli`

Reference: https://github.com/huggingface/huggingface_hub/issues/1259

`huggingface-cli scan-cache` doesn't see cached diffusers models. Are there any drawbacks to changing the cache folder?