2026-10-27 · ArXiv
arXiv:2608.00794v5 Announce Type: replace
Abstract: Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has characterized how validity degrades across these stages. We present a knowledge-based three-layer compounding validity model, $V_{\rm total} \leq V_1 \times V_2 \times V_3$, that captures multiplicative degradation across task generation ($V_1$), human-simulator calibration ($V_2$), and automated judgment ($V_3$). Applying it to five widely-used agentic benchmark configurations, we find $V_1$ and $V_2$ independently measurable and both degraded, but $V_3$ not currently quantifiable for any: three of the four benchmarks score with a single deterministic rule, so no second rater exists and reliability is undefined, and the fourth's partial LLM grader has no published grader-vs-human measurement.
A structured survey of 55 published agentic evaluation papers finds that 26 of them report no chance-corrected inter-rater statistic of any kind, a determination four independent raters recover at Krippendorff's $\alpha = 0.834$. On the broader judgment of whether a reported metric fits its rating design, approximately 80% of this purposive sample use structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics (four-rater range 60.7%-84.4%). Both are consistent with systematic $V_3$ collapse. We further document $V_1$ failures (task validity flaws in 7 of 10 benchmarks) and $V_2$ miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers).
We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (intraclass correlation $\geq 0.70$; $\alpha \geq 0.67/0.70/0.80$ by consequence level) that practitioners can apply immediately.
原文链接：https://arxiv.org/abs/2608.00794
查看更多资讯请见 AI 行业资讯。