Yi

2024-03 · text · generative · anchor

Open-weight bilingual language model family, cited in the corpus at 6B and 34B including the 200K long-context checkpoints.

Variants: Yi 6B, Yi 6B 200K, Yi 34B 200K, Yi-1.5-34B-Chat, Yi-1.5-9B

Findings

IC-004Retrieval heads are sparse, universal, and causally responsible for long-context retrieval in LLMs
IC-207Open LLMs (Llama-3-8B-Inst, Yi-1.5-34B-Chat) show weaker performance on coding and math tasks compared to proprietary models (GPT-4-turbo-0409, Claude 3 Opus) which perform well across all task categories
IC-299Instruction tuning preserves the weight-matrix structure of LLMs, with DOCS scores exceeding 0.7 across all matrices
IC-301Yi-1.5-9B-chat exhibits a layer-repetition pattern where a section of layers is duplicated at a later depth
IC-351GPT-4-turbo-2024-04-09, Qwen2-72b-instruct, and Llama-3.1-70b-instruct show distinct performance profiles across ultra-long, 32k, and 4k context benchmarks
IC-427Newer base models (post-November 2023) outperform older ones by 7.3 points on MMLU and 19.1 points on GSM8K controlling for pretraining compute, but this gap vanishes after fine-tuning all models on the same task-relevant data
IC-43256 LLMs from 19 families exhibit u-shaped scaling on hard questions and inverted-U scaling on easy questions, with the opposing trends explaining emergent ability stagnation
IC-575Four released LLMs (LLaMA-3.1-8B, Mistral-7B, Qwen2-7B, Yi-1.5-9B) can perform in-context learning on continuous vector representations projected into their embedding space, matching or outperforming few-shot ICL across text, time-series, graph, and fMRI tasks
IC-576For 10-digit numerical function regression, vector-ICL consistently outperforms few-shot ICL with raw number inputs across all four LLMs because continuous representations avoid multi-token splitting

Shared mechanisms