SourceDo LLMs ``know'' internally when they follow instructions?
The authors conducted a sensitivity analysis on 20 prompts from the 'forbidden keyword' instruction type, creating five modified versions for each of three perturbation types: changing the task to one with lower perplexity (task familiarity), simplifying the instruction by reducing constraints (instruction difficulty), and rephrasing while preserving meaning (phrasing). They computed the cosine similarity between the perturbation-induced representation change and the instruction-following dimension. In both models, phrasing modifications showed the strongest alignment with the dimension, suggesting the model's internal signal for instruction-following is driven by surface phrasing rather than the semantic difficulty of the task or instruction.