Language Models Are Implicitly Continuous

2025-01-22 · ICLR 2025 Poster · anchor

Findings

• IC-245Pretrained LLMs produce duration-dependent outputs that are incompatible with a discrete token interpretation
• IC-246Pretrained LLMs assign coherent semantic meaning to linear interpolations between token embeddings, extending the linear embedding hypothesis to the output space
• IC-247Pretrained LLMs are invariant to positional shifts but sensitive to duration scaling of the input