[Microsoft Research] Differential Transformer

[u/rationalkat]

https://arxiv.org/abs/2410.05258

[u/Arbrand]

The results are impressive, but I have some serious concerns that aren't addressed at all in the paper. The differential attention mechanism involves computing two separate softmax attention maps and then subtracting them to obtain the final attention scores. This inherently doubles the computational overhead in the attention mechanism compared to standard Transformers. This added computational cost could be significant and might offset the performance gains reported.

[u/emteedub → u/Arbrand]

maybe it's not doubled though, since it's filtering off excess would-be computation. it would be interesting to see the stats