The paper evaluates GPT-3 (175B), FLAN-T5 (L, XL, XXL), and LLaMA (7B, 65B) on self-rationalization across StrategyQA, QuAREL, OpenBookQA, Numersense, and QASC using few-shot chain-of-thought prompting. Task accuracy and rationale quality metrics (plausibility, consistency, diversity) generally improve with model scale: GPT-3 achieves the highest average NRG on most datasets, FLAN-T5-XXL and LLaMA-65B are competitive, while LLaMA-7B and FLAN-T5-L show substantially lower performance, particularly on Numersense (17.59% and 26.13% accuracy respectively) and QASC (24.19% and 61.02%).
All reference LLMs are evaluated via few-shot prompting with the same demonstrations; no fine-tuning is applied to them. The paper notes that MARIO (0.7B) still lags behind GPT-3 on plausibility and task accuracy for most datasets.