IC-395Most mainstream LLMs exhibit positive ADCE across five tasks, indicating reliance on deep structure for problem-solving, with ADCE strongly correlated with accuracy (r² > 0.7)
The paper measures the Approximated Direct Causal Effect (ADCE) of deep structure on model outputs for 12 released LLMs across five tasks (2-digit multiplication, GSM8K, word unscrambling, analytic entailment, CommonsenseQA). ADCE is positive for most models, in contrast to the zero ADCE of a random-weights baseline, indicating that intervening on core semantics causes previously correct answers to become incorrect. A strong linear correlation exists between accuracy and ADCE across all tasks, with r² values ranging from 0.784 to 0.977, suggesting that higher-accuracy models depend more on deep structure.
ADCE is an approximation of the true direct causal effect; the approximation relies on intervention strategies that minimize but do not eliminate the gap between the true ICE and AICE. Some model-task pairs (e.g., Llama-3-8B on analytic entailment) show negative ADCE.