GOAT-7B

anchor

Findings

• IC-721The 72-head entity tracking circuit identified in Llama-7b achieves high faithfulness in Vicuna-7b and Goat-7b without any modification to the circuit graph
• IC-722Entity tracking in Llama-7b is implemented by detecting and transmitting the positional information of the correct entity, with distinct head groups for position detection, transmission, and value fetching
• IC-723The entity tracking performance gap between Goat-7b and Llama-7b is primarily attributable to enhanced positional information in the value fetcher and position transmitter heads