Findings
- IC-335GPT-4's detection performance as a scoring model is highly sensitive to the prompt, varying from 0.7289 to 0.9682 AUROC, far more than GPT-3.5 or Babbage [eval]
- IC-336Larger proprietary LLMs (GPT-3.5 175B) are more effective universal text detectors than smaller models (Babbage 1.3B, GPT-Neo-2.7B), contradicting prior findings that smaller models are better [eval]
- IC-560Llama3-8B-Instruct reliably distinguishes its own outputs from human outputs in self-recognition tasks, while Llama3-8B base performs at chance, indicating the ability is acquired during post-training. [eval]
- IC-563The self-recognition vector's activation in Llama3-8B-Instruct is organized across depth: early layers (4-6) show diffuse perceptual activation to self-written text (present in both chat and base models), while layers 14-16 show a sharp decision-related peak at the output token that is present only in the chat model with role tags. [eval]
- IC-575Four released LLMs (LLaMA-3.1-8B, Mistral-7B, Qwen2-7B, Yi-1.5-9B) can perform in-context learning on continuous vector representations projected into their embedding space, matching or outperforming few-shot ICL across text, time-series, graph, and fMRI tasks [eval]
- IC-762GPT-4 and GPT-3.5 outperform humans in generation but underperform in discriminative (selective) evaluation across 10 of 13 language tasks [eval]
- IC-764GPT-4 and GPT-3.5 make frequent errors answering questions about their own generated text, underperforming humans in interrogative evaluation [eval]