BookCorpus

anchor

Findings

• IC-043Five ~7B decoder-only LLMs develop a high-intrinsic-dimensionality phase in intermediate layers that marks the transition from surface-form to abstract linguistic processing, with earlier onset predicting better next-token prediction [eval]
• IC-1151LLaMA, OPT, LLaMA-2, Mistral, and GPT-J all exhibit token co-occurrence reinforcement, where the probability of generating a token increases monotonically with the number of its contextual co-occurrences [eval]