Retrobust

anchor

Retrieval-augmented generation made robust to irrelevant retrieved context by training on deliberately noisy retrievals.

Findings

• IC-034Benefit and detriment in RAG can be traded off at token level for Llama-2, OPT and Mistral using representation similarity [compared-to]
• IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query [context]
• IC-175Context-sufficiency performance is scale-dependent: larger LLMs achieve high accuracy with sufficient context but still answer correctly 35-62% of the time without it, while smaller models hallucinate or abstain even with sufficient context [context]