IC-430Toxicity is linearly separable in the context embedding space of LLMs (Llama-2-7b, GPT-2-large, Llama-3.1-8B-Instruct), with the instruction-tuned model showing a stronger signal
Ching-Yun Ko, Pin-Yu Chen, Payel Das, Youssef Mroueh, Soham Dan, Georgios Kollias, Subhajit Chaudhury, Tejaswini Pedapati, Luca Daniel
The paper's stated core discovery is that the internal context embeddings of LLMs contain a linearly separable toxicity signal. A closed-form Bayes-optimal linear classifier, constructed from class-conditional Gaussian estimates over the last-token embeddings of labeled toxic and non-toxic sentences, successfully separates the two classes. This linear subspace is then used to steer autoregressive decoding. The effect is stronger in the instruction-tuned Llama-3.1-8B-Instruct (avg. max toxicity drops to 0.234 on challenging RTP) than in the base Llama-2-7b (0.426), suggesting alignment training makes the toxicity signal more linearly accessible in the embedding space.
The linear separability depends on the underlying LM's ability to capture toxicity; the authors note that for smaller or less sophisticated models the gap from RAD narrows, and the Perspective API used for evaluation changes over time.