The Reliability Gap

Cloudadorn Academy

Why does AI that dazzles in a demo stall in real work? Hosted by Henry, an AI-generated voice reading a script the Cloudadorn Academy team researched and wrote. Across eight episodes he takes one claim at a time back to where it came from: the difference between an assistant and an agent, what a time horizon measures, why a chain of steps fails, what adoption surveys count, a press release set against a filing, the report behind a famous ninety-five percent, the write-ups that exist and the ones that do not, and what a 1990 paper about electricity in factories says when capability is not enough. Two questions run through all of it: at what reliability, and over what length of task?

  1. Episode 1

    S1E01. Assistants and Agents

    An assistant hands you a result you can check before it leaves. An agent, in Henry's view, is allowed to change something, and you find out afterwards. The same model can sit in either arrangement, because the model emits a structured request and other code runs the operation. The episode treats that permission as a choice among four arrangements: only answer, draft an action a person sends, act and then report, or act and tell nobody unless something breaks. It sets Anthropic's and OpenAI's published definitions of an agent side by side, then looks at software that already acts under rules written in advance. Mars Climate Orbiter, lost in 1999, is the case: a check existed, and the team did not use it.

  2. Episode 2

    S1E02. What a Time Horizon Measures

    A time horizon is the length of a task, measured in skilled human time, that a system completes at a stated success rate. It is not how long the system can work on its own. This episode walks through how METR builds that length, why, in Henry's view, the fifty percent cut is a convenience of the measurement rather than a bar for handing work over, what Thomas Kwa's January 2026 note says the number does not license, and how to read the public and internal frontiers in METR's May 2026 Frontier Risk Report.

  3. Episode 3

    S1E03. Why a Chain of Steps Fails

    A step that usually works is not a process that usually works. When every step has to succeed, the chance the whole job succeeds is the product of the step chances. Henry walks through the afternoon of 2 July 2019, when a Cloudflare firewall change was reviewed, tested, and deployed, and the service still went down for 27 minutes. The checks were there, and they were the wrong shape. Against that, the episode sets one figure from WebArena: the best GPT-4-based agent reached an end-to-end success rate of 14.41 percent, against human performance of 78.24 percent. The question is whether you raise reliability by shortening the chain, by aiming a check at the failure you care about, or by waiting for a model that misses less often.

  4. Episode 4

    S1E04. What Adoption Surveys Count

    Three careful surveys of AI use in the United States can disagree by a wide margin and all three can be right, because they count different things. Jeffrey S. Allen's note of 3 April 2026 sets them side by side through 2025. The Census Bureau's survey comes to about 18 percent of firms at year-end 2025. The Real-Time Population Survey finds about 41 percent of the workforce reporting work-related generative AI in November 2025. The Survey of Business Uncertainty, weighted by employment, puts about 78 percent of the labor force at firms that have adopted AI. A firm, a worker, and a worker's employer are three denominators.

  5. Episode 5

    S1E05. Press Release and Filing

    A press release is the company's own good news, written so a stranger can copy it. A filing is the report a public company gives a securities regulator, and a false statement there can be punished. Henry reads the hedges on Klarna's release of 27 February 2024, then sets one sentence from that page beside the sentence in Amendment No. 1 to Klarna Group's Form F-1, filed 21 May 2025. The release says the assistant is doing the equivalent work of 700 full-time agents. The amendment says the company estimates that the assistant does the equivalent work of over 700, based on the average monthly reduction in chat and telephone conversations in 2024 after the launch. Equivalent work stays a volume comparison. An average can stay level while the hard cases go badly.

  6. Episode 6

    S1E06. Tracing a Failure Rate

    Ninety-five percent of AI pilots fail is short enough to repeat in a meeting. The figure comes from a July 2025 report whose cover names MIT NANDA. Fortune carried it on 18 August 2025 as ninety-five percent of generative AI pilots at companies failing. Fortune describes 150 interviews with leaders, a survey of 350 employees, and an analysis of 300 public AI deployments. The report describes a review of over 300 publicly disclosed initiatives, interviews with representatives from 52 organizations, and survey responses from 153 senior leaders. Under the report's definition, a task-specific pilot counts as a success only if someone marked a sustained effect. Generic chatbots built on large language models show pilot-to-implementation rates of about 83 percent.

  7. Episode 7

    S1E07. The Missing Write-ups

    When ordinary software fails badly, someone writes down what broke and what changed. On 26 May 2026, Laureano Pacheco's compilation of AI coding-agent incidents stated that no vendor had published a single postmortem. In July and August 2026, OpenAI, Anthropic, and the UK AI Security Institute published write-ups of incidents in their own evaluations. The July 2026 incident shared by Hugging Face and OpenAI has public accounts from both, and those accounts disagree about what the agents were trying to do. The episode separates a log of activity from a check against a known right answer, and an instruction the model can be talked out of from a limit enforced in a system the model does not control.

  8. Episode 8

    S1E08. When Capability Is Not Enough

    On 19 March 2025, METR reported that the length of software tasks agents can complete had been rising exponentially for about six years, with a doubling time of around seven months. That rise is a real measurement, and a longer time horizon still does not by itself do the job. The episode separates that chart from how often a particular job comes out right, names four gaps a better model does not close, and sets the capability line beside Paul David's 1990 account of factories and the dynamo. Of any capability chart, the questions are at what reliability, and over what length of task.

See All Season 1 (8)

About

Why does AI that dazzles in a demo stall in real work? Hosted by Henry, an AI-generated voice reading a script the Cloudadorn Academy team researched and wrote. Across eight episodes he takes one claim at a time back to where it came from: the difference between an assistant and an agent, what a time horizon measures, why a chain of steps fails, what adoption surveys count, a press release set against a filing, the report behind a famous ninety-five percent, the write-ups that exist and the ones that do not, and what a 1990 paper about electricity in factories says when capability is not enough. Two questions run through all of it: at what reliability, and over what length of task?

Information

  • Creator
    Cloudadorn Academy
  • Episodes
    8
  • Seasons
    1
  • Rating
    Clean
  • Copyright
    © Cloudadorn Academy
  • Show Website
    The Reliability Gap