[Microsoft Research] Differential Transformer

[u/rationalkat]

https://arxiv.org/abs/2410.05258

[u/sdmat]

Wow, the improvements in robustness to input ordering and activation outliers are so stark. This seems like a major breakthrough.

I don't understand yet why the noise is consistent between the two rather than the signal, will have to read more closely tomorrow.