SourceSupervised Knowledge Makes Large Language Models Better In-context Learners
On the SQuAD 2.0 hold-out test, the paper measures how well each model handles unanswerable questions (questions with no valid answer in the context). ChatGPT achieves 32.35% accuracy on no-answer questions, while Llama-2-7b-chat achieves only 3.72%, meaning it almost never outputs the correct 'I don't know' response. The authors note that Llama-2-7b-chat 'struggled with format adherence and hallucinations, especially in answering no-answer questions.' In-context learning with mixed no-answer and has-answer examples improves no-answer handling but reduces has-answer accuracy.