A looped transformer runs the same layers repeatedly rather than stacking more of them, which is appealing because it reuses parameters instead of multiplying them. The idea has faced a stubborn empirical problem: given N times more pre-training compute, making the model N times bigger has usually beaten looping it N times.

That comparison is the one that matters, because it is the choice an engineer actually faces with a fixed budget. Looping has kept losing it, which is why the approach stayed a curiosity.

Loopie is presented as the most powerful looped transformer so far, two Mixture-of-Experts models at 20B and 6B parameters with only 2B and 0.6B active. Combining looping with sparse expert activation is a coherent bet: both are ways of getting more computation out of fewer resident parameters, and they act on different axes.