In a preliminary experiment, the paper tests GPT-4o-mini on incorrect contexts with two prompt orderings: question before context (QC) and context before question (CQ). When the question precedes the context, accuracy on wrong contexts is 48.0 (TriviaQA), 48.3 (ConflictQA), and 78.8 (RedditQA). When the context precedes the question, accuracy drops to 33.3, 33.1, and 46.0 respectively. The same incorrect evidence is more likely to mislead the model when it appears earlier in the input, demonstrating a positional bias in the model's reliance on external information.
Evidence
correlational
Key metric
GPT-4o-mini acc given wrong contexts: QC 48.0/48.3/78.8 vs CQ 33.3/33.1/46.0 (TriviaQA/ConflictQA/RedditQA)
Caveat
Described as a preliminary experiment; tested on only three datasets and one model variant (GPT-4o-mini). The effect size varies by dataset (largest on TriviaQA and ConflictQA, smaller on RedditQA).