Lines of Thought in Large Language Models

2025-01-22 · ICLR 2025 Poster · anchor

Findings

• IC-608Token trajectories in GPT-2, Llama 2 7B, Mistral 7B, and Llama 3.2 models cluster on a low-dimensional manifold and follow a linear drift plus Gaussian noise dynamics
• IC-609The last transformer layer of Mistral 7B, Llama 3.2 1B, and Llama 3.2 3B shows anomalous trajectory statistics inconsistent with the linear drift-plus-noise pattern of intermediate layers