IC-493A linear direction in the input embedding space of Llama-2-7B-Chat, Llama-2-13B-Chat, Mistral-7B-Instruct-v0.3, and Phi-3-mini-128k predicts instruction-following success, generalizes across tasks but not instruction types, and can be used to improve adherence via representation engineering
Juyeon Heo, Christina Heinze-Deml, Oussama Elachqar, Kwan Ho Ryan Chan, Shirley You Ren, Andrew Miller, Udhyakumar Nallasamy, Jaya Narain
The authors trained linear probes on frozen model representations to identify a direction in input embedding space that separates instruction-following successes from failures. This direction generalizes across unseen tasks (AUROC 0.70–0.88 for the first token) but not across unseen instruction types (AUROC 0.50–0.55, near chance), even when the training set is expanded to 23 instruction types. The information is available from the first token, before any response generation begins. When representations in the last layer are shifted along this direction (representation engineering), instruction-following success rates improve over both original outputs and random-direction shifts, while response quality is maintained or improved.
Analysis limited to 5 simple instruction types (or 23 in the expanded experiment) and 100 tasks; the authors note that unambiguously interpreting the meaning of the dimension remains an open question. Results are consistent across the four models studied but may not generalize to other architectures.