Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-721
The 72-head entity tracking circuit identified in Llama-7b achieves high faithfulness in Vicuna-7b and Goat-7b without any modification to the circuit graph
IC-722
Entity tracking in Llama-7b is implemented by detecting and transmitting the positional information of the correct entity, with distinct head groups for position detection, transmission, and value fetching
IC-723
The entity tracking performance gap between Goat-7b and Llama-7b is primarily attributable to enhanced positional information in the value fetcher and position transmitter heads