Path patching on GPT-2-small (English-only) and CPM-distilled (Chinese-only), which share the same architecture and parameter count but were trained on entirely different data, reveals that both implement the same high-level IOI algorithm: duplicate token detection, s-inhibition, and name copying. The main structural difference is that GPT-2 uses negative name mover heads while CPM uses copy suppression heads (identified in Pythia by McDougall et al.) to suppress the repeated token. This convergence suggests that LLMs consistently arrive at similar algorithms for this task regardless of training language.
The authors note the CPM model has substantially lower accuracy and zero-rank rate, and that the copy suppression head divergence could be a starting point for understanding when specific circuitry emerges during training.