New PR in at llama.cpp for full CUDA GPU acceleration!
github.com/ggerganov/llam…
This is huge! For the first time GGML is beating GPTQ speed.
On a 4090 + i9-13900K I'm getting 109.29 tokens/s on 7B and 29.11 tokens/s on 30B.
AutoGPTQ is: 98 t/s for 7B, 35 t/s for 30B.
