The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LGTM! Let me know when it's out of draft and you want a final review

It would be good to add a test for compiled models alongside this change

Thanks @amyeroberts ! This is ready for review !
Note currently QLoRA + torch.compile is not really supported as we need to make some changes to make 4bit layers from bnb compatible with torch.compile . Therefore I added a test that simply checks if the trainer is correctly initialized so that we don't get the unappropriate error from #29033 . Once compile + QLoRA will be supported, the integration will be seemless as users will simply have to update bitsandbytes to make it work

Thanks @amyeroberts for the review! I just adapted the PR according to your suggestion, one concern I have with this new approach is that if bnb supports torch.compile in the next months we'll need to do some version check on trainer's init and make the code slightly bloated - though I am happy with this approach as well as I don't think this is that bad. What do you think? 🙏

@younesbelkada How soon and with what confidence do you think bnb will support this?

I agree we want to avoid bloat as much as possible, but I think it's better overall to be explict and have well defined behaviour for our users. The easiest way to avoid many version checks is to only support the latest version of bnb.

OK sounds good! I agree with that approach!
Currently we don't have any ETA :/ it might take a bit long, and it will also depend if the community will ask for it

@younesbelkada I would certainly be interested if bnb could support torch.compile!