Mixture-of-Experts models are efficient because only a few experts run per token, but every expert's weights still have to be somewhere the GPU can reach. In serving workloads that also carry a large KV cache, those two demands compete for the same memory, and the cache grows with every concurrent conversation.

Something has to yield. Evicting cache means recomputing context that was already processed; keeping weights resident means fewer conversations fit at once.

PagedWeight quantises expert weights dynamically at runtime and balances that against the cache, with the quality-aware part carrying the important nuance: experts are not equally sensitive to being compressed, so uniform quantisation either over-degrades the fragile ones or under-compresses the robust ones. Deciding per expert, while serving, is what turns a blunt trade into a manageable one.