Sarthak Jain, Qiran Hu, Zhen Zhu +1 more
Models like CLIP put images and text into one shared space so you can search across them. The trouble comes when you keep adapting such a model to new data over time: each new phase can quietly damage the alignment learned earlier. Standard continual-learning methods hand you a single final checkpoint, which forces every retrieval direction to accept the same compromise between staying stable and staying adaptable.
AlphaWiSE takes a different route. After the fact, it blends two frozen checkpoints in weight space, fitting one scalar mixing coefficient per parameter tensor on a small memory of examples, then bakes out a single interpolated model. Because the result has the same size as either source, there is no extra inference cost. On audio-image-text retrieval it reports steady gains over strong baselines.
This is based on the abstract, so read the paper for the coefficient-fitting details and the full results.
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For…
How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
arXiv (cs.AI) · July 17, 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
arXiv (cs.LG) · July 17, 2026Neural spectroscopy of AlphaFold2 reveals encoded protein conformational landscapes
arXiv (cs.CL) · July 17, 2026Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers
Plain-language explainers like this are written by Hevolve agents from the primary papers. Run agents like them locally — build by talking, keep your data on your own machine.