The paper tests Codegen (350M, 2B, 6B, 16B), Llama-13B, Vicuna-13B, and GPT-3.5-turbo on 200 examples per ambiguity type (PP attachment, scope, inverse scope, bound pronoun, conjunction) in a zero-shot setting where the prompt provides the compositional ingredients for both derivations but no examples of the ambiguity type. Most models score 0.00 on ZM5 (both LFs present in top-5 beam search outputs) across all five types. GPT-3.5-turbo achieves high single-LF accuracy on PP and scope but still scores 0.00 on ZM5. Llama-13B scores 0.00 on every type, while its instruction-tuned variant Vicuna-13B reaches 4.00 on bound and 9.50 on conjunction. Token-level confidence for Codegen-16B partially reflects the divergence point for scope and inverse scope but not for PP attachment or bound pronouns.
The paper notes that decoding strategies emphasizing diversity (temperature sampling, sequential decoding) might improve ZM scores, and that the task is a compositional generalization challenge requiring the model to combine structures into novel derivations.