Prefix Tuning / Li & Liang 2021 (Prefix-tuning)

anchor

Findings

• IC-866A prefix applied to Llama-7B's first attention layer preserves the relative attention distribution over content positions and only adds a constant-direction bias to the attention block output [builds-on]
• IC-866A prefix applied to Llama-7B's first attention layer preserves the relative attention distribution over content positions and only adds a constant-direction bias to the attention block output [primary]
• IC-867In GPT-2 prefix-tuned on the emotion dataset, attention over prefix positions is nearly constant across inputs, collapsing the effective bias subspace to a single direction in most layers [builds-on]
• IC-867In GPT-2 prefix-tuned on the emotion dataset, attention over prefix positions is nearly constant across inputs, collapsing the effective bias subspace to a single direction in most layers [primary]