IC-1585LLaMA-7B and GPT-J-6B exhibit positional bias in instruction following: zero-shot performance varies significantly when the instruction is moved from after to before the context
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, Tuo Zhao
The paper tests three prompt templates that convey the same instruction but place it in different positions relative to the context. For GPT-J-6B on JSON formatting, placing the instruction before the context (rephrased) yields 63.3 format accuracy versus 28.83 when it follows the context (original). For LLaMA-7B, the shortened prompt (instruction at end, example removed) drops accuracy to 36.0 from 60.0. The authors note this sensitivity is well-known and show that PASTA mitigates it, but the underlying positional dependence is a property of the models themselves.
Evidence
correlational
Key metric
GPT-J-6B JSON format f.acc: 28.83 (original, instruction after context) vs 63.3 (rephrased, instruction before context); LLaMA-7B: 60.0 (original) vs 36.0 (shortened) vs 57.9 (rephrased)
Caveat
The rephrased prompt also changes wording, not only position, so the effect is a combination of positional and lexical variation; the authors do not isolate position alone.