Rendering new camera views of a live, moving scene from streaming video has a built-in tension: you want a long memory to fill in things that briefly get hidden, but you also have to keep up in real time. Test-Time Training gives you that memory, except the usual version demands a gradient update every single frame, which is slow and prone to drifting over long stretches.

The insight is that you don't need to refresh memory as often as you use it. Most video is repetitive, so they update memory only periodically while applying it every frame, using cross-view attention to handle motion between the stored state and the current view. Two safeguards keep it honest: an auxiliary loss that forces the scene to actually stick, and a caching trick that keeps the weights from wandering. They report real-time, state-of-the-art results on dynamic human motion and minute-long memorization.

This is the abstract's summary, so the paper will spell out the datasets and metrics behind those claims.