IC-998GPT-3.5's zero-shot accuracy on OGBN-ARXIV depends on the position of the title relative to the abstract in the prompt: 0.720 when abstract precedes title, 0.695 when title precedes abstract
Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, Bryan Hooi
The paper tests four prompt variants on OGBN-ARXIV. The default prompt places the abstract before the title and yields 0.720 zero-shot accuracy. When the title is moved before the abstract (same content, different order), accuracy drops to 0.695. Two other variants that add instructional text (focus on text content, chain of thought) also yield 0.695 and 0.705 respectively. The authors note this aligns with the principle that placing more critical information later in the prompt can be beneficial.
Evidence
correlational
Key metric
0.720 (abstract before title) vs 0.695 (title before abstract), zero-shot on OGBN-ARXIV
Caveat
The difference is 0.025 in absolute accuracy; the paper does not report statistical significance or multiple seeds for the prompt comparison.