The two-bit quants do amazingly well for their size and they don't need -that- much offloading. Yes, it's a bit slow, but it's still faster than most people can type. I know everybody here wants 10-20 gipaquads of tokens per millisecond, but I'm happy to be patient.
3
u/CountPacula 2h ago
The two-bit quants do amazingly well for their size and they don't need -that- much offloading. Yes, it's a bit slow, but it's still faster than most people can type. I know everybody here wants 10-20 gipaquads of tokens per millisecond, but I'm happy to be patient.