The paper evaluates ChatGPT (gpt-3.5-turbo-0301) in a zero-shot setting on the UniversalNER benchmark, which comprises 43 NER datasets across 9 domains (general, biomedical, clinical, STEM, programming, social media, law, finance, transportation). Using strict entity-level micro-F1, ChatGPT attains 34.9% average F1. Performance varies widely by domain, from 51.0% in general to 3.8% in transportation. The paper uses the prompting template from Ye et al. (2023) for NER.
Evidence
correlational
Key metric
34.9% average F1 (zero-shot, strict entity-level micro-F1, 43 datasets, 9 domains); domain range 3.8% (transport) to 51.0% (general)
Caveat
Evaluation uses up to 200,000 sampled passage-query pairs per dataset due to size constraints; strict F1 requires exact entity type and boundary match, which the authors note may underestimate zero-shot capability.