IC-848GPT-3.5 achieves 98.3% precision and 96.0% recall for automatic question-tuple matching but makes errors when questions differ in wording yet are semantically unique
The paper uses GPT-3.5 with in-context examples to automatically evaluate whether generated DSG questions match their source semantic tuples (precision/recall) and whether questions are unique (no duplicates). On the full TIFA160 set, GPT-3.5 achieves 98.3% precision and 96.0% recall for matching, and 90.5% uniqueness for DSG questions. However, the paper identifies a specific error pattern: GPT-3.5 is 'distracted by text overlap' and flags semantically distinct questions as duplicates when they share surface wording, or misses true duplicates that are phrased differently.
The automatic evaluation uses GPT-3.5 as both the system under test and the evaluator for some metrics, and the paper notes disagreement between automatic and human evaluation in specific cases (Figure 9).