Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Re2Pix is a two-stage video prediction framework that first forecasts future semantic features, then renders realistic next frames from them.

It predicts future representations in a frozen vision foundation model’s feature space and uses them to guide a latent diffusion model.

Nested dropout and mixed supervision reduce the train-test gap between ground-truth and predicted representations.

The result is better visual realism and temporal semantic consistency, with improved robustness in complex dynamic settings such as autonomous driving.