Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-500
GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B solve multi-step retrieval problems without fine-tuning, achieving near-perfect accuracy for chains of up to 5 steps