If a BCI lets someone spell by detecting brain responses to flashing letters, what number should you report to say how well it works? Plain accuracy and loss don't quite capture it, because what users care about is the spelling rate: how many characters actually get picked correctly, which drives the information-transfer rate. On top of that, these ERP systems see far more non-targets than targets, so imbalance-blind metrics can mislead.

The authors line up 13 metrics against spelling rate across two datasets, one private and one public, to see which track real performance and how they respond to repeated trials. The winners were the Brier score, Matthews correlation coefficient, and imbalance-aware measures like ROC AUC and precision-recall AUC. Their advice is practical: report those in ERP BCI studies rather than raw accuracy.

Drawn from the abstract, so the paper will have the correlation details and how repetition affected each metric.