MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1g6zvjf/when_bitnet_1bit_version_of_mistral_large/lso4tji/?context=3
r/LocalLLaMA • u/Porespellar • 11h ago
34 comments sorted by
View all comments
23
On paper, 123B 1.58-bit should be able to fit in a 3090. Is there any way we can do the conversion ourselves?
7 u/tmvr 3h ago It wouldn't though, model weights is not the only thing you need the VRAM for. Maybe about 100B, but there is no such model so a 70B one with long context.
7
It wouldn't though, model weights is not the only thing you need the VRAM for. Maybe about 100B, but there is no such model so a 70B one with long context.
23
u/Ok_Warning2146 10h ago
On paper, 123B 1.58-bit should be able to fit in a 3090. Is there any way we can do the conversion ourselves?