Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension ability
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-395
Most mainstream LLMs exhibit positive ADCE across five tasks, indicating reliance on deep structure for problem-solving, with ADCE strongly correlated with accuracy (r² > 0.7)
IC-396
Closed-source LLMs (GPT, Claude) rely more on deep structure than open-source LLMs (Llama, Mistral), and open-source models' surface sensitivity decreases with model scale