IC-395Most mainstream LLMs exhibit positive ADCE across five tasks, indicating reliance on deep structure for problem-solving, with ADCE strongly correlated with accuracy (r² > 0.7)
Yujin Han, Lei Xu, Sirui Chen, Difan Zou, Chaochao Lu
The paper measures the Approximated Direct Causal Effect (ADCE) of deep structure on model outputs for 12 released LLMs across five tasks (2-digit multiplication, GSM8K, word unscrambling, analytic entailment, CommonsenseQA). ADCE is positive for most models, in contrast to the zero ADCE of a random-weights baseline, indicating that intervening on core semantics causes previously correct answers to become incorrect. A strong linear correlation exists between accuracy and ADCE across all tasks, with r² values ranging from 0.784 to 0.977, suggesting that higher-accuracy models depend more on deep structure.
ADCE is an approximation of the true direct causal effect; the approximation relies on intervention strategies that minimize but do not eliminate the gap between the true ICE and AICE. Some model-task pairs (e.g., Llama-3-8B on analytic entailment) show negative ADCE.