SourceZero and Few-shot Semantic Parsing with Ambiguous Inputs
When given prompts containing 10 input-LF pairs with varying ratios of LF0 to LF1 (0% to 100% in increments of 1), models' accuracy on LF0 tracks the prompt ratio. For scope and inverse scope, Codegen-2B's accuracy tracks almost perfectly. FDM scores (lower is better, measuring dataset-level alignment) show Codegen-16B achieving 0.20-0.38 and Vicuna-13B achieving 0.17-0.35 across ambiguity types, while Llama-13B scores 1.00 on all types (never predicts correctly). FIM scores (probability alignment via forced decode) are low across all models (0.03-0.16), indicating that even Llama-13B, which never generates a correct LF, assigns probabilities aligned with the prompt distribution when forced to decode the gold LFs. The paper notes that low FIM does not imply low FDM since FDM requires actual accuracy.