2026-10-15 · ArXiv
arXiv:2609.33196v2 Announce Type: replace
Abstract: Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9\% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.
原文链接：https://arxiv.org/abs/2609.33196
查看更多资讯请见 AI 行业资讯。