The paper evaluates whether smaller open-weight models can match or exceed a larger proprietary model on causal reasoning. Using the triplet method (which queries three variables at a time and enforces acyclicity), Phi-3 and Llama-3 8B produce cycle-free causal orders with lower SHD than GPT-4 using standard pairwise prompting. On the 20-node child dataset, Phi-3 triplet achieves SHD 69 and Llama-3 triplet achieves SHD 129, compared to GPT-4 pairwise's SHD 148, while both smaller models produce zero cycles versus GPT-4's >>10k cycles.
The comparison is confounded by the prompting strategy: the smaller models use the triplet method while GPT-4 uses pairwise. The paper does not report GPT-4 with triplet prompting on the same datasets for a controlled comparison (Table A10 shows GPT-4 triplet on a subset but with different metrics).