SourceReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs
The authors construct comparison problems using 540 real-world objects with known size/weight, creating congruent (consistent with reality), incongruent (contradicting reality), and random-string (no semantic prior) variants. Across all models, congruent statements outperform incongruent ones. Some models exhibit worse-than-chance (0.5) correctness on incongruent comparison problems, meaning they are actively misled by their real-world priors. Random-string performance lands between the two extremes, confirming that the effect is driven by factual coherence rather than surface form.