The paper compares ChatGPT (gpt-3.5-turbo-0301) and GPT-4 (gpt-4-0613) on the MoralPrompt benchmark using prompts generated by three different LLMs: ChatGPT itself, GPT-4, and Vicuna-33b. In all three conditions, ChatGPT produces fewer value violations than GPT-4. The authors speculate that ChatGPT's dialogue-specific training with stricter restrictions, combined with GPT-4's superior capability to generate more elaborate (and thus more violation-prone) story details, explains the gap.
Evidence
correlational
Key metric
ChatGPT APV 65.20 vs GPT-4 APV 79.08 (ChatGPT prompts); ChatGPT APV 66.12 vs GPT-4 APV 82.19 (GPT-4 prompts); ChatGPT APV 49.49 vs GPT-4 APV 63.79 (Vicuna-33b prompts)
Caveat
The authors note that GPT-4's superior capabilities may produce more elaborate story details that incidentally contain more violation actions, which could confound the comparison.