Current situation: I've got a desktop with 16 GB of DDR4 RAM, a 1st gen Ryzen CPU from 2017, and an AMD RX 6800 XT GPU with 16 GB VRAM. I can 7 - 13b models extremely quickly using ollama with ROCm (19+ tokens/sec). I can run Beyonder 4x7b Q6 at around 3 tokens/second.
I want to get to a point where I can run Mixtral 8x7b at Q4 quant at an acceptable token speed (5+/sec). I can run Mixtral Q3 quant at about 2 to 3 tokens per second. Q4 takes an hour to load, and assuming I don't run out of memory, it also runs at about 2 tokens per second.
What's the easiest/cheapest way to get my system to be able to run the higher quants of Mixtral effectively? I know that I need more RAM Another 16 GB should help. Should I upgrade the CPU?
As an aside, I also have an older Nvidia GTX 970 lying around that I might be able to stick in the machine. Not sure if ollama can split across different brand GPUs yet, but I know this capability is in llama.cpp now.
Thanks for any pointers!
I hate to bear the bad news, but as long as the model is too large to fit entirely in VRAM getting 5 t/s on a 8x7b is going to be difficult. You can throw another 16gb RAM in the system which could help with caching and context length, but since the model is still having to juggle data in and out of VRAM the speeds will remain low.
I wouldn't upgrade the CPU personally, focus on adding beefier GPU. And it's probably not worth adding the 970 to the mix, the 4GB isn't providing much room and will likely slow down the 6800 XT more.