Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
COLLIE: Systematic Construction of Constrained Text Generation Tasks
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1373
All five evaluated LLMs show a strong positional bias in constrained text generation, with first-position constraints nearly always satisfied but last- and arbitrary-position constraints causing major performance drops
IC-1374
Counting difficulty in constrained generation increases with text level and constraint strictness, with exact sentence-level character counts being the hardest condition for all five models
IC-1375
GPT-4's constraint satisfaction improves by approximately 20% after one round of automated feedback but plateaus at 66% even after three additional rounds