SourceLLM-grounded Video Diffusion Models
The paper prompts GPT-4 (and GPT-3.5) to generate bounding-box sequences (dynamic scene layouts) for 6-frame videos from text captions, providing only 3 in-context examples illustrating gravity, elasticity, and perspective projection. GPT-4 achieves 98% average rule-based accuracy across 5 tasks (numeracy, attribution, visibility, dynamics, sequential), while GPT-3.5 reaches 77%. Critically, GPT-4 generalises to objects and properties not present in the examples: it correctly applies elasticity to a ball but not to a rock, infers air friction for a paper airplane, and applies perspective geometry to a car viewed from different angles. The authors argue this knowledge resides in the model's weights rather than being extracted from the few examples. Adding retrieval-based video examples does not improve performance (96% vs 98%), confirming the few-shot generalisation.