Sparse autoencoders trained on the residual stream of Pythia-160m (12 layers) and Mamba-130m (24 layers) yield features whose activation patterns are highly correlated across architectures. The mean max pairwise Pearson correlation (MPPC) is 0.681 for the cross-architecture comparison, close to the model-seed skyline (0.725) and SAE-seed skyline (0.806), and far above the random baseline. Over 55% of SAE features exceed MPPC 0.7, and more than 25% of Pythia features match a Mamba feature with MPPC above 0.95. The matched features are organised by depth: a feature at layer l in Pythia most often matches a feature near layer 2l in Mamba, consistent with Mamba having twice the depth.
Evidence
correlational
Key metric
mean MPPC 0.681 (cross-arch), 0.725 (model seed variant), 0.806 (SAE seed variant); >55% SAE features with MPPC > 0.7; >25% features with MPPC > 0.95; 90% of features in MPPC-difference interval [-0.23, 0.07]; >60% with |difference| < 0.05
Caveat
SAEs capture only a subset of true features; the superposition hypothesis underlying SAEs is still an immature research area. Conclusions rely on specific SAE hyperparameters (expansion factor 32, L1=2e-4) and a single tokenizer shared by both models.