IC-1374Counting difficulty in constrained generation increases with text level and constraint strictness, with exact sentence-level character counts being the hardest condition for all five models
Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik R Narasimhan
The paper decomposes constraint difficulty by counting level and constraint type. Counting characters within a word is easier than within a sentence for all models. Exact equality constraints (e.g., exactly 82 characters in a sentence) are harder than range constraints (e.g., between 10 and 15 words per sentence), which are harder than simple upper/lower bounds. Adding a second counting level via logical composition (e.g., at least 20 words AND each word under 6 characters) uniformly degrades performance compared to the single-level version. Tasks word02, sent01, sent02, and para03 remain very challenging across all models.
Evidence
correlational
Key metric
Tasks word02, sent01, sent02, para03 remain very challenging across all models; sent03 uniformly trails word01 despite the added constraint being assumed manageable
Caveat
The paper reports these as qualitative trends illustrated in figures rather than giving per-task numeric tables in the main text; specific per-task rates are in figures 6-7.