Send the same request to the same AI grader, byte for byte, at temperature zero, a day later, and 22 times out of 100 its ranking of the answers comes back different. Replayed the same day, it changed just as often, and on all four providers tested, rankings moved. That grader, an LLM judge, is the same kind of instrument that helps pick what chatbots learn from and scores the benchmarks in launch posts. Since July 2026, one of them also casts votes on Arena's leaderboard, the one that was supposed to be human. The teach first: what a judge is, why the industry adopted one at about half a cent a grade despite flaws documented in 2023, and its four jobs: what a model reads, what it learns to say, the launch headline (Gemini 2.5 scored 79 percent on Arena-Hard with Gemini 2.5 as judge, and 49 with GPT-4.1), and the leaderboard. Then what a judge physically is: Thinking Machines Lab sent one prompt 1,000 times at temperature zero and got 80 different answers, because the batch a request lands in changes how the chip rounds the arithmetic. With batch-invariant kernels on, all 1,000 matched. Along the way, a correction to our episode How LLM Inference Actually Works, which called temperature zero fully deterministic. Then the audit, limits first: Haoyuan Zhu of the University of Sheffield and Jie Zhang of Ranplan Wireless, posted to arXiv on September 3rd, 2026, with no peer review. Their grader needed a ranking agreement of 0.9 and got 0.4, mostly because the answers were near-ties, as the authors say themselves. In a byte-for-byte replay, 78 of 100 rankings came back identical against a bar of 99. Running the grader yourself with the fix came close on a quiet machine but didn't reach it. Then who's exposed: OpenAI switches off two of the founding graders on October 23rd, 2026, and GPT-4.1 on January 14th, 2027, so their scores can't be rerun. And what you can do: replay 100 of your own evaluation requests, run a pilot before you set a pass mark, and ask the grader five times when a verdict matters. CHAPTERS 00:00 Same request, a day later 01:47 What an LLM judge is 02:48 Judging LLM-as-a-Judge, 2023 04:40 Four jobs, starting with what models read 05:35 The launch headline 06:34 The leaderboard that was supposed to be human 08:00 When the automated checks said fine 09:08 One prompt, a thousand times 09:50 Why the batch changes the arithmetic 11:30 A correction on temperature zero 12:00 Even zero isn't a promise 12:42 Same name, different behavior 13:18 Anthropic's August 2025 bugs 15:45 The audit, limits first 17:27 A grader that never passed 19:16 Close contests at the top 19:56 78 in 100 20:39 Four providers and the self-hosted fix 22:02 The sharpest criticism is in the study 22:22 Validity versus reliability 23:14 The cheap fixes 24:30 Who's exposed 26:01 What deterministic serving costs 26:47 What you can do 27:37 The grader was never a ruler SOURCES Zhu & Zhang, arXiv, Sep 3, 2026: https://arxiv.org/abs/2609.04198 Thinking Machines Lab, Sep 10, 2025: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ Zheng et al., Judging LLM-as-a-Judge, 2023: https://arxiv.org/abs/2306.05685 Anthropic postmortem, Sep 17, 2025: https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues OpenAI on GPT-4o sycophancy, May 2, 2025: https://openai.com/index/expanding-on-sycophancy/ Arena, AutoEval, Jul 30, 2026: https://arena.ai/blog/autoeval-scores Arena, judge study, Sep 30, 2026: https://arena.ai/blog/llm-judge-self-preference Chen, Zaharia & Zou, 2023: https://arxiv.org/abs/2307.09009 OpenAI API deprecations: https://developers.openai.com/api/docs/deprecations
