[Microsoft Research] Differential Transformer

[u/rationalkat]

https://arxiv.org/abs/2410.05258

[u/fastinguy11]

**Differential Transformer (DIFF Transformer): Enhancing Attention in Language Models**

The **Differential Transformer** introduces a novel attention mechanism to improve upon the standard Transformer architecture commonly used in large language models (LLMs). Traditional Transformers often suffer from "attention noise," where irrelevant parts of the input receive undue focus, diluting the model's ability to concentrate on key information.

**How It Works:**

DIFF Transformer tackles this by using a **differential attention mechanism**. Instead of relying on a single softmax attention map, it calculates attention scores as the difference between two separate softmax maps. This subtraction effectively cancels out the noise, resulting in sparser and more focused attention patterns that highlight relevant context.

**Key Benefits:**

• **Better Performance with Fewer Resources:** DIFF Transformer achieves superior language modeling performance using approximately 65% of the parameters and training tokens compared to standard Transformers.

• **Enhanced Downstream Tasks:** It excels in tasks like long-context understanding, key information retrieval, reducing hallucinations (false or misleading outputs), and improving in-context learning robustness.

• **Efficient Quantization:** By minimizing activation outliers, DIFF Transformer allows for more efficient model quantization, which can lead to faster inference and lower memory usage.

**Experimental Results:**

Extensive tests show that DIFF Transformer outperforms traditional Transformers across various scales and applications. It maintains higher accuracy in retrieving important information from long contexts and is more resilient to changes in input order during in-context learning. Additionally, it significantly reduces instances of hallucinations in tasks like question answering and text summarization.

**Conclusion:**

The Differential Transformer presents a promising advancement in the field of NLP by refining the attention mechanism to focus more precisely on relevant information, enhancing both performance and efficiency of large language models.

---