When a multimodal model writes captions for images, some of its mistakes are not random. It might reliably get something wrong whenever a particular visual feature shows up, say describing the wrong object every time a certain background appears. Symbal is built to catch that kind of patterned error, which the authors call a systematic misalignment.

It runs in two stages using off-the-shelf foundation models, and needs no access to the model that wrote the captions, so you can audit a dataset from the outside. It even writes up what it found in plain language. To measure progress, the team assembled SymbalBench: 1.7 million image-caption pairs across natural and medical images, sorted into 420 datasets with known planted errors. On it, Symbal flagged the systematic error in about 64% of datasets, close to four times the nearest baseline.

That summary rests on the abstract, so check the paper for how the benchmark was built and where the method stumbles.