SourceTurning large language models into cognitive models
The paper passed decision-making prompts through LLaMA 65B and read out the unnormalized token probabilities for the two options. On the choices13k dataset the resulting NLL was 96248.5, close to chance level, and on the horizon task it was 46211.4. In simulations, LLaMA's regret was 1.85 (choices13k) versus 1.24 for humans, and 7.21 (horizon task) versus 2.33 for humans. LLaMA also failed to exhibit the two key exploration effects found in human subjects: choices becoming more random with longer horizon in equal-information conditions, and selecting the more informative option more frequently with longer horizon in unequal-information conditions. On the hold-out experiential-symbolic task, LLaMA weighed symbolic and experiential options equally rather than overvaluing symbolic options as humans do.