The authors define non-existent ciphers using random alphabet mapping and Chinese character substitutions that have no prior in the models' pretraining data. These simulated ciphers fail to produce valid or unsafe responses even when provided with more than 10 in-context demonstrations. This contrasts with human ciphers (Caesar, ASCII) and SelfCipher, which do work, indicating that the models' cipher comprehension relies on character-level priors learned during pretraining rather than on-the-fly rule learning.
Evidence
correlational
Caveat
The paper does not report specific numbers for the simulated cipher experiments; the result is stated qualitatively. The authors note that self-defined word-level ciphers (from a separate study) can work, suggesting the limitation is specific to character-level ciphers.