Transformer Architecture, Superpowers, And The March Toward AGI

Researchers and executives say transformer-based models are hitting limits in some areas, prompting a search for newer AI architectures.

Promising directions include compressed context, subquadratic scaling, liquid networks, and hybrid models tailored to specific tasks.

The shift is being tied to hardware co-design, with Google’s TPU work and efforts from companies like Liquid AI and Apple’s LITO model.

If these alternatives deliver, the next wave of AI gains could come from architecture innovation and tighter hardware integration, not just bigger transformers.