SourceZero and Few-shot Semantic Parsing with Ambiguous Inputs
The paper tests Codegen (350M, 2B, 6B, 16B), Llama-13B, Vicuna-13B, and GPT-3.5-turbo on 200 examples per ambiguity type (PP attachment, scope, inverse scope, bound pronoun, conjunction) in a zero-shot setting where the prompt provides the compositional ingredients for both derivations but no examples of the ambiguity type. Most models score 0.00 on ZM5 (both LFs present in top-5 beam search outputs) across all five types. GPT-3.5-turbo achieves high single-LF accuracy on PP and scope but still scores 0.00 on ZM5. Llama-13B scores 0.00 on every type, while its instruction-tuned variant Vicuna-13B reaches 4.00 on bound and 9.50 on conjunction. Token-level confidence for Codegen-16B partially reflects the divergence point for scope and inverse scope but not for PP attachment or bound pronouns.