IC-396Closed-source LLMs (GPT, Claude) rely more on deep structure than open-source LLMs (Llama, Mistral), and open-source models' surface sensitivity decreases with model scale
Yujin Han, Lei Xu, Sirui Chen, Difan Zou, Chaochao Lu
By comparing ADCE (deep structure effect) and AICE (surface structure effect) for 12 released LLMs across five tasks, the paper finds that closed-source models (GPT-3.5-Turbo, GPT-4o, Claude-3-Sonnet, Claude-3.5-Sonnet) show a larger ADCE-minus-AICE gap, indicating greater reliance on core semantics. Open-source models (Llama-2, Llama-3, Mistral, Mixtral) show higher AICE, meaning their outputs are more sensitive to surface-level presentation changes. Within open-source families, this surface sensitivity diminishes as model size increases (e.g., Llama-2-7B to Llama-2-70B, Llama-3-8B to Llama-3-70B, Mixtral-8x7B to Mixtral-8x22B).
Evidence
correlational
Caveat
The comparison is based on the ADCE-AICE difference shown in Figure 7; specific per-model values are not printed in the text. The finding is limited to the four model families tested and five tasks evaluated.