IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context
On the unanswerable context task, the context is modified to remove the evidence supporting the ground-truth answer, and the model is instructed to respond 'unknown' if no information is available. Across all 18 chat models, accuracy drops substantially compared to the original context. The gap ranges from 13.6% to 68.4%. Notably, high original-context performance does not predict unanswerable-context performance: phi-3-medium-128k-instruct scores 75.8% on the original context but only 7.4% on the unanswerable version. Within the same family, larger models do better, e.g. llama-3.1-70b-instruct improves by 10.3% over the 8B variant.
Evidence
correlational
Key metric
performance gap 13.6% to 68.4% across all chat models; phi-3-medium-128k-instruct: 75.8% (original) vs 7.4% (unanswerable); llama-3.1-70b-instruct +10.3% over 8B
Caveat
The unanswerable contexts were generated by an LLM (GPT-4o) modifying original contexts, so the difficulty may depend on how well the modification preserves coherence while removing the answer.