When asked to critique a proposed solution, GPT-4 produces feedback that is frequently wrong. In graph coloring, it hallucinates non-existent edges between same-colored vertices and misstates vertex colors, often missing the actual violated constraint. In Game of 24, it evaluates 81.6% of correct expressions to 24 but only labels 79.1% as correct, meaning it rejects valid answers even when its arithmetic is right. In planning, it hallucinates whether action preconditions are met, and in mystery blocksworld it incorrectly assumes the state of preconditions, dropping critique generation accuracy to 3% for correct plans.
Hallucination rates vary with the error rate of the input: when more edges are wrong (random colorings), the model is more likely to point to a real error. The authors note this makes the hallucination rate harder to interpret in isolation.