Item response theory was invented to make sense of tests: thousands of students answering a few dozen questions, so you can separate how able each student is from how hard each question is. AI researchers have started borrowing it to rank models and judge benchmark quality. The trouble is that the setup is flipped. You usually have only a handful of models but thousands of questions, and their ability levels can be lumpy or skewed rather than the tidy bell curve the classic tools assume.

This paper stress-tests that borrowing. Drawing item difficulties and ability profiles from six real LLM benchmarks, the authors ran 18,000 simulated evaluations across different models and estimation methods. The picture is sobering: traditional estimators can simply choke on large benchmarks, while the faster, scalable ones can produce unreliable rankings and item judgments when there are few models or their abilities are not nicely distributed.

This summarizes the abstract, so see the paper for which methods held up and the diagnostics it recommends.