IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35%
On the inconsistent context task, two documents are concatenated, each supporting a different answer, and the model is instructed to respond 'conflict' when it detects multiple answers. Performance varies dramatically across families: the phi-3 series averages only 5.8% accuracy, while the GPT-4 series averages 89.35%. Open-source models lag behind proprietary models on this task, unlike the unanswerable task where all models struggle. The top three models are all proprietary. When the new (conflicting) context is presented alone rather than concatenated with the original, most models do not find it harder, indicating the difficulty lies in detecting the conflict between two sources.
Evidence
correlational
Key metric
phi-3 series average accuracy 5.8%; GPT-4 series average accuracy 89.35%; command r: 88% on new context alone vs 81% on original
Caveat
The inconsistent contexts were constructed by an LLM generating a new answer and modifying the context, then concatenating with the original. The new context is designed to be highly similar to the original, which may make conflict detection harder than in natural retrieval scenarios.