System 2 Attention (is something you might need too)

The paper: http://arxiv.org/abs/2311.11829

## Purpose 
The paper introduces System 2 Attention (S2A), a technique aimed at improving the attention mechanism in Transformer-based Large Language Models (LLMs). S2A addresses the problem of LLMs incorporating irrelevant information from their input context, which adversely affects their output generation.

## Methods 
- S2A leverages LLMs' natural language reasoning and instruction-following abilities.
- Two-step process: 
  1. Regenerate the input context to include only relevant parts (x' ∼ S2A(x)).
  2. Use the regenerated context to produce the final response (y ∼ LLM(x')).

## Key Findings 
1. S2A outperforms standard attention-based LLMs in tasks prone to opinion or irrelevant information.
2. Improves factuality and objectivity in responses, reducing sycophancy.
3. Effective in various settings, including factual QA, longform generation, and math word problems.

## Discussion 
S2A demonstrates its effectiveness in generating more accurate and objective responses by refining the attention process. It shows potential for enhancing LLMs' ability to focus on relevant information and resist misleading context.

## Critiques 
1. S2A doesn't always succeed in removing irrelevant context.
2. More computationally intensive than standard LLM regeneration.
3. The method's success depends on the prompt's design, which could be further optimized.

## Tags
#System2Attention #LLMs #AI #NaturalLanguageProcessing #AttentionMechanisms #AIAlignment #AIResearch.