UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Researchers introduced UltraViT, a vision encoder designed specifically to reduce on-device latency for large vision-language models.

It uses a pyramidal architecture with heterogeneous spatial mixers, plus a two-stage generative pre-training scheme.

The pre-training combines dense distillation and frozen-LLM supervision to improve multimodal alignment and efficiency.

In experiments, UltraViT beat encoder-focused baselines while running about 1.7× faster on device.