IC-037Removing locality from Diffusion Transformers improves image generation quality.

Duy Kien Nguyen, Mido Assran, Unnat Jain, Martin R. Oswald, Cees G. M. Snoek, Xinlei Chen

SourceAn Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

The paper applies the locality-free principle to a Diffusion Transformer (DiT) for class-conditional image generation on ImageNet. They compare their pixel-based variant (DiT/1) operating on latent tokens from a VQGAN to the standard patch-based DiT (DiT/2). The pixel-based DiT achieves better FID, sFID, and Inception Score, demonstrating that the removal of locality is beneficial even in a generative setting with a different input representation and architecture.

Evidence

correlational

Key metric

FID (↓): DiT-l/2 (4.16) vs. DiT-l/1 (4.05); sFID (↓): DiT-l/2 (4.97) vs. DiT-l/1 (4.66); IS (↑): DiT-l/2 (210.18) vs. DiT-l/1 (232.95).

Caveat

The extended training results (1400 epochs) show a larger gap, but the main comparison is at 400 epochs. The paper notes the high computational cost of the pixel-based approach.

Model

DiT

Datasets

ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]

Related work

DiT [builds-on]

Extraction

automatic-extraction