Vision-language models are increasingly asked to read charts and draw conclusions, which makes it worth knowing how they arrive at an answer. A model that reads the correct bar and one that guesses from the caption produce identical output and deserve very different amounts of trust.

This paper offers a lightweight diagnostic saliency method built for the actual setting, text generation over images with transformers, rather than adapted from classification where most interpretability tooling originated. That distinction matters: a chart answer is generated token by token, and which part of the image mattered can change from one token to the next.

The word literacy in the title is apt. The question is not only whether the model gets chart questions right, but whether it is reading the visualisation at all, and those come apart precisely on the questions where being right matters most.