IC-745Function vectors for simple list-oriented tasks can be algebraically combined via addition and subtraction to produce new vectors that trigger composed tasks, sometimes outperforming 10-shot ICL.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, David Bau
The authors extract FVs for three decomposable tasks (first-copy, first-capital, last-copy) and form a composed vector v*_bd = v_ad + v_bc - v_ac, hypothesizing it triggers the 'last-capital' task. Testing on GPT-J, Llama 2 (13B), and Llama 2 (70B), the composed vector outperforms both the directly extracted FV and 10-shot ICL on some tasks (e.g., last-country-capital: 0.60 vs 0.15 vs 0.32 on GPT-J; last-capitalize-first-letter: 0.95 vs 0.76 vs 0.75). Other tasks resist composition, suggesting the algebraic structure holds for certain task decompositions but not universally.
Composition works for some tasks but not others; the authors note it may hinge on 'word-selection' and 'word-transformation' being separable components. Results are on a small set of list-oriented tasks.