SQuAD

anchor

Findings

• IC-034Benefit and detriment in RAG can be traded off at token level for Llama-2, OPT and Mistral using representation similarity [eval]
• IC-248Instruction fine-tuning causes context reliance under knowledge conflicts to initially increase then decrease (context-parametric inversion) in Llama2-7B, Pythia-6.9B, and Mistral-7B [eval]
• IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context [source]
• IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35% [source]
• IC-610In LLaMA2-7B-Chat, RAG hallucinations are causally driven by copying heads losing external context information during generation and by knowledge FFNs in mid-to-upper layers over-adding parametric knowledge to the residual stream [eval]
• IC-663OPT-1.3B, LLaMA-7B, and Aquila-7B encode sparse Harsanyi interactions, with only 29-51 salient interactions out of 1024 possible on SQuAD sentences [eval]
• IC-959LLaMA and OPT-1.3B (and Aquila-7B) encode more similar interaction primitives than smaller models such as BERT-base and BERT-large [eval]