IC-576For 10-digit numerical function regression, vector-ICL consistently outperforms few-shot ICL with raw number inputs across all four LLMs because continuous representations avoid multi-token splitting
The authors pretrain linear projectors on 10-digit number reconstruction and addition, then test the LLMs on a non-linear function regression task f(x,y)=sqrt(x)*sqrt(y). Because 10-digit numbers span multiple tokens in all four LLMs' tokenizers, the discrete-token baseline is handicapped. The projected 100-dimensional one-hot vector representation allows the LLMs to learn the function in-context more effectively, with vector-ICL consistently outperforming few-shot ICL with raw numbers.
Evidence
correlational
Caveat
The advantage is specific to numbers that span multiple tokens; 3-digit numbers (single token in LLaMA-3.1-8B) show a different pattern. The function is fixed (sqrt(x)*sqrt(y)), so generalization to other functions is not tested.