2026-10-21 · ArXiv
arXiv:2608.27309v4 Announce Type: replace-cross
Abstract: LLM-judge audits assess bias by comparing ratings across matched conditions. Difference-in-differences designs compare two candidate responses within each item and then compare that contrast across a manipulated attribute, with an endpoint read off a bounded rating scale. Building on prior work on nonlinear difference-in-differences, we show in closed form that when ratings are bounded, a severity shift common to both responses produces an observed interaction whenever the scale bounds attenuate it unequally. We examine this problem in a pre-registered audit comprising 990 calls to a frozen pedagogy judge. The registered primary analysis did not detect an effect of a stated novice or advanced profile on scaffolding preference. The only nominally significant secondary interaction concerned the productive-struggle sub-score ($+0.378$ points; $p = 0.002$). A post-hoc construction with zero differential preference reproduced 79 to 85% of this interaction using the observed severity shift and the scale floor alone. The observed interaction therefore does not identify differential preference on the latent scale.
原文链接：https://arxiv.org/abs/2608.27309
查看更多资讯请见 AI 行业资讯。