SciQ / SciQA

anchor

Findings

• IC-1319Larger LLaMA and LLaMA2 models show better calibration on phrase-level tasks but not consistently on sentence- and paragraph-level tasks [eval]
• IC-282GPT-2 XL (1.5B) exhibits lower accuracy but reduced overconfidence (smaller ECE and Brier scores) compared to larger models on the CAT benchmark [eval]
• IC-368Larger LMs (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro) exhibit better calibration than their smaller counterparts (GPT-4o-mini, Claude-3-Haiku, Gemini-1.5-Flash) when verbalizing confidence with certainty phrases [eval]
• IC-369LMs verbalizing confidence with certainty phrases are better calibrated on SCIQ than on TruthfulQA [eval]
• IC-370Knowledge entropy (sparsity of FFN memory coefficients) decreases consistently during pretraining for OLMo 1B, 7B, and Pythia 1.4B, and this decrease strongly correlates with reduced knowledge acquisition and increased forgetting in continual learning [eval]
• IC-371Artificially resuscitating inactive memory vectors by scaling the up-projection matrix K improves knowledge acquisition and reduces forgetting, with the effect more pronounced for later-stage OLMo models [eval]
• IC-414LLaVA-1.5-7B and LLaVA-1.5-13B exhibit severe performance degradation when H2O KV cache compression is applied in multimodal settings [eval]
• IC-472The degree of emergence metric derived from Pythia's internal structure positively correlates with benchmark performance across training epochs [eval]