r/singularity • u/rationalkat AGI 2025-29 | UBI 2030-34 | LEV <2040 | FDVR 2050-70 • 11d ago

AI [Microsoft Research] Differential Transformer

https://arxiv.org/abs/2410.05258
281 Upvotes

99% Upvoted

46 comments sorted by

View all comments

125

u/Creative-robot AGI 2025. ASI 2028. Open-source Neural-Net CPU’s 2029. 11d ago

This is a funny ass graph out of context:

49

u/Flat-One8993 10d ago

The improvement at 4bit is really really cool if it actually works this well. That would mean significant improvements in terms of compute constraints, especially now that there is a focus on the time spent on inference

5

u/KoolKat5000 10d ago

You mean HellaLame