Vector-ICL: In-context Learning with Continuous Vector Representations

2025-01-22 · ICLR 2025 Poster · anchor

Findings

IC-575Four released LLMs (LLaMA-3.1-8B, Mistral-7B, Qwen2-7B, Yi-1.5-9B) can perform in-context learning on continuous vector representations projected into their embedding space, matching or outperforming few-shot ICL across text, time-series, graph, and fMRI tasks
IC-576For 10-digit numerical function regression, vector-ICL consistently outperforms few-shot ICL with raw number inputs across all four LLMs because continuous representations avoid multi-token splitting
IC-577An encoder's text reconstruction performance positively correlates with its effectiveness in downstream vector-ICL classification tasks across 15 encoder-LLM-dataset configurations