IC-1585LLaMA-7B and GPT-J-6B exhibit positional bias in instruction following: zero-shot performance varies significantly when the instruction is moved from after to before the context

Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, Tuo Zhao

SourceTell Your Model Where to Attend: Post-hoc Attention Steering for LLMs

The paper tests three prompt templates that convey the same instruction but place it in different positions relative to the context. For GPT-J-6B on JSON formatting, placing the instruction before the context (rephrased) yields 63.3 format accuracy versus 28.83 when it follows the context (original). For LLaMA-7B, the shortened prompt (instruction at end, example removed) drops accuracy to 36.0 from 60.0. The authors note this sensitivity is well-known and show that PASTA mitigates it, but the underlying positional dependence is a property of the models themselves.

Evidence

correlational

Key metric

GPT-J-6B JSON format f.acc: 28.83 (original, instruction after context) vs 63.3 (rephrased, instruction before context); LLaMA-7B: 60.0 (original) vs 36.0 (shortened) vs 57.9 (rephrased)

Caveat

The rephrased prompt also changes wording, not only position, so the effect is a combination of positional and lexical variation; the authors do not isolate position alone.

Model

LLaMA Llama 7B, GPT-J 6B

Concepts

Positional bias

Datasets

Bias in Bios [eval]

Methods

Zero-shot prompting [primary]

Related findings

IC-1584, IC-1586

Extraction

automatic-extraction