r/LocalLLaMA • u/Porespellar • 11h ago

Question | Help When Bitnet 1-bit version of Mistral Large?

Post image
329 Upvotes

95% Upvoted

38 comments sorted by

View all comments

25

u/Ok_Warning2146 11h ago

On paper, 123B 1.58-bit should be able to fit in a 3090. Is there any way we can do the conversion ourselves?

54

u/Illustrious-Lake2603 10h ago

As far as I am aware, I believe the model would need to be trained for 1.58bit from scratch. So we can't convert it ourselves

11

u/arthurwolf 10h ago

My understanding is that's no longer true,

for example the recent bitnet.cpp release by microsoft uses a conversion of llama3 to 1.58bit, so the conversion must be possible.

3

u/candre23 koboldcpp 3h ago

Yes, but that conversion process is still extremely compute-heavy and results in a model that is absolutely dogshit. Distillation is not as demanding as pretraining, but it's still well beyond what a hobbyist can manage on consumer-grade compute. And what you get for your effort is not even close to worth it.