Do Function Vectors and Task Vectors Generalize?

Abstract

A language model can learn a task from examples in its prompt. Function vectors (FVs) and task vectors (TVs) aim to transfer that behavior to a new prompt by modifying the model's activations. We reproduce both methods on word-pair tasks across GPT-J, Llama-3.1, and Gemma-2. In held-out tests, FVs outperform random controls on GPT-J and Llama-3.1-8B. TVs outperform shuffled-label controls on GPT-J and both base and instruction-tuned Gemma; the Llama-8B comparison is inconclusive. Our FV implementation achieves zero accuracy on both Gemma checkpoints in a two-task follow-up. Sentiment classification shows a further limitation: in-context learning (ICL) achieves 73.5–94.0% accuracy with arbitrary labels, while TVs achieve only 49.6–52.7%. Successful transfer on word pairs therefore does not extend to this sentiment dataset. Separating vector construction, layer selection, and testing also shows why gains over zero-shot accuracy can overstate the evidence for transfer.