When given prompts containing 10 input-LF pairs with varying ratios of LF0 to LF1 (0% to 100% in increments of 1), models' accuracy on LF0 tracks the prompt ratio. For scope and inverse scope, Codegen-2B's accuracy tracks almost perfectly. FDM scores (lower is better, measuring dataset-level alignment) show Codegen-16B achieving 0.20-0.38 and Vicuna-13B achieving 0.17-0.35 across ambiguity types, while Llama-13B scores 1.00 on all types (never predicts correctly). FIM scores (probability alignment via forced decode) are low across all models (0.03-0.16), indicating that even Llama-13B, which never generates a correct LF, assigns probabilities aligned with the prompt distribution when forced to decode the gold LFs. The paper notes that low FIM does not imply low FDM since FDM requires actual accuracy.
FIM uses forced decoding of gold LFs, so low FIM does not imply the model would generate the correct LF under free decoding. The paper explicitly states: 'low f_im does not imply low f_dm, since low f_dm requires the model to be accurate.' GPT-3.5-turbo is excluded from few-shot experiments because the API does not provide logit scores.