SourceJoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention
The paper demonstrates that a single neuron in the MLP hidden layers of Pythia-70M and Pythia-160M can be activated by multiple unrelated token combinations. For example, the same neuron fires for both 'every morning' and 'in the realm of physics', which are semantically unrelated phrases. This 'pattern superposition' arises because the neuron's weight vector can encode multiple orthogonal query-pattern pairs simultaneously, as predicted by the JOMA framework's implicit bias mechanism.