SourceCOLLIE: Systematic Construction of Constrained Text Generation Tasks
The paper engages GPT-4 in a generation-feedback dialogue where automated natural language feedback (e.g., 'you generated a word with 3 character v and 4 character i') is provided after each round. GPT-4 shows a significant 20% improvement after the second round, but performance plateaus at 66% after three further rounds, comparable to pass@5 with i.i.d. sampling. Improvement varies by task: word03's rate changes from 62.1% to 10%, while word02, sent01, and sent02 remain challenging throughout.