SourceCOLLIE: Systematic Construction of Constrained Text Generation Tasks
The paper decomposes constraint difficulty by counting level and constraint type. Counting characters within a word is easier than within a sentence for all models. Exact equality constraints (e.g., exactly 82 characters in a sentence) are harder than range constraints (e.g., between 10 and 15 words per sentence), which are harder than simple upper/lower bounds. Adding a second counting level via logical composition (e.g., at least 20 words AND each word under 6 characters) uniformly degrades performance compared to the single-level version. Tasks word02, sent01, sent02, and para03 remain very challenging across all models.