Current only rms prop is supported by trl for DPO training. But bitsandbytes contains 8bit RMSprop that could be used to save on vram, making DPO QLORA on 7b’s fit better without going into shared memory, on 24 gig cards. Could we work on an implementation of 8bit rms prop into trl?
Current only rms prop is supported by trl for DPO training. But bitsandbytes contains 8bit RMSprop that could be used to save on vram, making DPO QLORA on 7b’s fit better without going into shared memory, on 24 gig cards. Could we work on an implementation of 8bit rms prop into trl?