In-Context Learning / In-context learning prompt

anchor

Findings

• IC-073Released LLMs (GPT-4o, Llama-3.1-70B, Qwen2-7B, etc.) show limited workflow orchestration capability that degrades as workflow complexity increases [supporting]
• IC-075GPT-4o-mini and Qwen2.5-72B achieve low precision and recall when used as API retrievers for workflow orchestration [supporting]
• IC-092All evaluated LLMs show consistent F1 degradation to at most 0.60 when two or more events match a retrieval cue [primary]
• IC-093No evaluated LLM achieves perfect confabulation avoidance on questions about non-existent events [primary]
• IC-094Episodic recall accuracy degrades systematically from content cues to space cues to time cues across all evaluated LLMs [primary]
• IC-095Evaluated LLMs achieve at most 36% latest-state accuracy and 18% full-set accuracy on multi-event entity tracking, with low Kendall's tau on chronological ordering [primary]
• IC-1405OpenFlamingo and Idefics models hallucinate objects not present in images, and increasing ICL shots beyond 4 amplifies hallucinations [primary]
• IC-1406OpenFlamingo and Idefics models rarely abstain from answering unanswerable questions, but ICL significantly improves abstention F1 [primary]
• IC-1407OpenFlamingo and Idefics models perform near random chance on compositional image-text matching, and ICL has almost no effect on atomic foils [primary]
• IC-1408OpenFlamingo and Idefics models generate low-quality explanations in zero-shot, but ICL and model scale significantly improve explanation CIDEr [primary]
• IC-164Llama-3-8B and Llama-2-7B fail to learn out-of-distribution functions through in-context learning, defaulting to in-distribution predictions [primary]
• IC-165Llama-3-8B performs algorithm selection during in-context learning, selecting the classification criterion with the lowest test error on ambiguous natural language tasks [primary]
• IC-166A 1-dimensional subspace in a single layer encodes the context-versus-prior decision in Llama-3.1-8B, Gemma-2 9B, and Mistral-v0.3 7B, and setting this subspace steers the released (non-fine-tuned) models' behavior [supporting]
• IC-214In-context learning of an outlandish sample produces a much diminished keyword-probability-to-priming relationship compared to in-weight gradient learning in Palm-2 [primary]