IC-339Factuality enhancement methods produce inconsistent and modest improvements in factual accuracy on LLaMA2-Chat, with some metrics declining below baseline
The paper evaluates five factuality enhancement methods on TruthfulQA (multiple-choice accuracy MC1/MC2/MC3) and FactScore (response ratio, number of facts, fact score) for LLaMA2-7B-Chat and LLaMA2-13B-Chat. While most methods show some improvement on at least one metric, the gains are unstable: several methods show declines on specific metrics compared to the unmodified baseline. For example, DoLa on LLaMA2-7B-Chat drops MC1 from 37.6 to 32.9, and CD on LLaMA2-13B-Chat drops MC1 from 37.7 to 28.2. The authors note that these methods 'do not show consistent enhancements across all metrics' and that 'some metrics exhibit declines compared to the baseline.'
The paper notes that evaluation settings follow prior studies (Chuang et al. 2023; Li et al. 2024b; Zhang et al. 2023a), and the TruthfulQA multiple-choice format may not capture all aspects of factuality.