IC-576For 10-digit numerical function regression, vector-ICL consistently outperforms few-shot ICL with raw number inputs across all four LLMs because continuous representations avoid multi-token splitting

Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, Jianfeng Gao

SourceVector-ICL: In-context Learning with Continuous Vector Representations

The authors pretrain linear projectors on 10-digit number reconstruction and addition, then test the LLMs on a non-linear function regression task f(x,y)=sqrt(x)*sqrt(y). Because 10-digit numbers span multiple tokens in all four LLMs' tokenizers, the discrete-token baseline is handicapped. The projected 100-dimensional one-hot vector representation allows the LLMs to learn the function in-context more effectively, with vector-ICL consistently outperforming few-shot ICL with raw numbers.

Evidence

correlational

Caveat

The advantage is specific to numbers that span multiple tokens; 3-digit numbers (single token in LLaMA-3.1-8B) show a different pattern. The function is fixed (sqrt(x)*sqrt(y)), so generalization to other functions is not tested.

Model

Llama 3.1 8B, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Qwen 2 7B, Yi Yi-1.5-9B

Related findings

IC-575, IC-577

Extraction

automatic-extraction