The authors apply a bijective string-to-string encoding to harmful prompts and teach the model the mapping in-context, then measure whether the model produces a harmful response. Across Claude 3 Haiku, Claude 3 Opus, Claude 3.5 Sonnet, GPT-4o-mini, and GPT-4o, the attack outperforms an ensemble of 11 prior encoding-based attacks by at least 30 percentage points on both AdvBench-50 and HarmBench. The peak ASR for each model scales with its MMLU capability: within the Claude family, the more capable model reaches a higher peak ASR at a higher MMLU score, and the same pattern holds for GPT-4o and Llama 3.1 families.
Evaluations limited to closed-source models due to cost; larger sweeps restricted to cheaper models. Bijection learning requires 25,000-50,000 input tokens, limiting applicability to models with large context windows.