SourceImproving Instruction-Following in Language Models through Activation Steering
When the paper computes steering vectors for word-exclusion instructions (difference in activations between 'include word w' and 'do not include word w' inputs) and projects them onto the vocabulary space via the unembedding matrix, the top tokens are the excluded word itself (e.g., 'books' → 'books, books, books, book'; 'congress' → 'congress, parlament, legisl'). Adding these vectors to the residual stream increases rather than decreases the probability of generating the excluded word. The paper confirms this by showing that subtracting the word-inclusion vector consistently yields higher accuracy and a better accuracy-to-low-perplexity trade-off than adding the word-exclusion vector (Figure 19). The authors adopt subtraction of inclusion vectors as the effective approach for word exclusion.