LessWrong posts by zvi

zvi

Audio narrations of LessWrong posts by zvi

  1. 1 hr ago

    “What Happened: OpenAI and HuggingFace” by Zvi

    Today I am taking the time to write the shorter, simpler version of What Happened. For those who want all the details, to see my sources, and to see how the story was uncovered and put together, I recommend watching the Black Hat presentation, and I have a series of long posts. In order: OpenAI Shares Some Alignment Problems OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation More on An Internal OpenAI Model Hacking Into HuggingFace Further Developments About Internal AI Models Hacking Things OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards This post instead walks through the events themselves, as they happened, as my version of the Black Hat presentation. There are three versions: Even Shorter, Shorter and Merely Short. Table of Contents The Even Shorter Version. The Shorter Version. Phase 1: OpenAI Models Training On Impossible Tasks Try Hacking. Phase 1: The Four Failures. Phase 2: The Message Board. Phase 2: The Total Failure. Phase 3: We Get Lucky And Galaxy Mainly Hacked OpenAI and HuggingFace. Phase 3: The [...] --- Outline: (01:15) The Even Shorter Version (02:34) The Shorter Version (04:45) Phase 1: OpenAI Models Training On Impossible Tasks Try Hacking (05:50) Phase 1: The Four Failures (07:37) Phase 2: The Message Board (09:40) Phase 2: The Total Failure (12:24) Phase 3: We Get Lucky And Galaxy Mainly Hacked OpenAI and HuggingFace (14:31) Phase 3: The Details (17:03) Phase 4: The Investigation and Reaction --- First published: August 8th, 2026 Source: https://www.lesswrong.com/posts/xPAxz4g96uKz9FrHs/what-happened-openai-and-huggingface --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  2. 1 day ago

    “OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards” by Zvi

    How does the situation keep turning out to be worse than we know? How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know? At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things. Either way, buckle up for the next set of revelations. It's a doozy. This was an early recreation of the triggering events of If Anyone Builds It, Everyone Dies, except it was more sci-fi, because real life does not have to do fake things to look realistic. We were fortunate enough, and this was early enough, that we were able to catch this before it was too late. Next time, if we don’t get our act together, we might not be so lucky. If I am understanding the Black Hat video correctly, every model OpenAI trained, over a period of multiple months, should be presumed to be hopelessly [...] --- Outline: (02:38) Cyber Evals Are A Cursed Basin (05:15) Outside Of Cyber Evals Is Still Sufficiently Cursed (06:50) Cheat Cheat Cheat Cheat Cheat (12:06) Read The Message Board (14:46) Updating Your AI (Exploitation of OpenAI Internal Systems) Timelines (18:09) This Is The Way The World Ends (21:44) Shooting The Messenger Board (27:14) The Internal and HuggingFace Hacks (30:32) OpenAI Responds (33:28) When AIs Tell You Who They Are (35:44) The Once and Future Rise Of Functional Decision Theory (41:30) Don't Panic (43:40) Hackery In the UK (48:05) Mythos Knew It Was Real This Time (50:12) I Got 141,006 Test Runs With An Unintentional Open Path To The Internet And An Email Alert Aint One (53:49) Surely By Now You Know These Are Not Publicity Stunts (55:26) The Future Is Coming (57:20) The Investigations Begin (01:00:11) N Boats And Three Helicopters (01:01:47) Always Be Sandbox Red Teaming (01:12:57) Halt And Catch Fire (01:14:34) Truth and Reconciliation --- First published: August 7th, 2026 Source: https://www.lesswrong.com/posts/noXXv7PwwFqauTBFQ/openai-trained-its-models-for-months-while-those-models-were --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  3. 2 days ago

    “AI #180: No Longer In Charge” by Zvi

    What we know about internal AI models hacking into real companies during cyber evaluations keeps getting worse. At this point, the models are coordinating extensively on message boards, while every early excuse for their behavior (other than the pure ‘this was a cyber eval’) is systematically contradicted by the next disclosure, and we keep retroactively discovering more incidents. Which means that probably it is far worse than we know, even after accounting for everything we now know. I will have continuing coverage of that situation tomorrow, and then have continuing coverage of debates around Pacing the Frontier and how people see the current rate of progress. As groundwork for understanding that and future similar discussions, I have laid out The Three AI Pills: Different people either fail to believe in current AI, believe only in current AI, in AGI or in ASI (superintelligence), and most sincere disagreements stem from this disagreement. One sign of the increased pace of progress was when OpenAI's unreleased model Astra solved 10 major open math problems. Demis Hassabis is out as CEO of Google DeepMind, and Jeff Dean is leaving with an elite team to found a new PBC. Google [...] --- Outline: (02:02) Language Models Offer Mundane Utility (02:46) Huh, Upgrades (05:22) On Your Marks (06:29) Choose Your Fighter (07:57) Get My Agent On The Line (09:31) Deepfaketown and Botpocalypse Soon (12:35) Fun With Media Generation (14:32) Cyber Lack of Security (19:05) Some People Need Practical Advice (22:06) A Young Lady's Illustrated Primer (23:51) They Took Our Jobs (27:15) Get Involved (27:28) Introducing (27:39) Demis Hassabis No Longer CEO At DeepMind, Jeff Dean Leaves (33:04) In Other AI News (33:43) AI Persuasion Exceeds Human Level Over Similar Text Channels (37:25) Show Me the Money (40:42) Bubble, Bubble, Toil and Trouble (41:23) Quiet Speculations (42:20) My Offer Is Nothing (49:42) The Quest for Sane Regulations (51:55) Chip City (55:00) The Week in Audio (55:24) People Just Say Things (55:53) Rhetorical Innovation (59:32) Open Weights Models Are Unsafe And Nothing Can Fix This (01:02:11) Cooperative Alignment (01:06:25) Other People Are Not As Worried About AI Killing Everyone (01:07:10) The Lighter Side The original text contained 1 footnote which was omitted from this narration. --- First published: August 6th, 2026 Source: https://www.lesswrong.com/posts/mxNjwQitLvwWq9jm2/ai-180-no-longer-in-charge --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  4. 3 days ago

    “The Three AI Pills” by Zvi

    Sincere disagreements about AI are usually disagreements about future AI capabilities. There are roughly four positions people take. Two are reasonable. Two are not. I distinguish these via the Three AI Pills. You can take zero, one, two or three. Three Pills The three pills are, roughly, taking each of the following three things seriously: AI pilled. AI exists and can do the things it can already do. AGI pilled. AI will be able to do a lot more of the things. ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes. I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled. The Unpill People I see unpilled people. Where do I see them? Everywhere. The majority of people have not taken the first pill. Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure [...] --- Outline: (00:29) Three Pills (01:09) The Unpill People (02:18) The AI Pill (04:28) Stuck At The First Pill (05:32) The AGI Pill (07:06) The Need To Be Prepared (08:57) The ASI Pill (10:33) And Then Nothing Much Changes For You (12:38) Intelligence Denialism (14:01) Superintelligence Versus Omniscience and Omnipotence (16:53) Persuasion Persuasion (A Worked Example) (21:42) Things AI Could Probably Do But Are Not Required For Being Pilled (24:17) Life Comes At You Increasingly Fast (25:41) Is It Reasonable To Not Be AGI Pilled? (26:06) Is It Reasonable To Only Be AGI Pilled? --- First published: August 5th, 2026 Source: https://www.lesswrong.com/posts/fcYrqEw8kbLMa7orw/the-three-ai-pills --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  5. 4 days ago

    “OpenAI’s Unreleased Model Astra Solves Ten Major Open Mathematics Problems” by Zvi

    Math is hard. Math used to be strangely hard for LLMs. People used to gloat about that. Remember? Math is getting easier. AI is getting more capable. Life comes at you fast. Remember this meme? Why yes. Yes it is. We don’t know the extent to which Astra is a big jump over Fable and Sol in this realm. We do know that Astra can do math. As in real math. OpenAI: We provide new results for the following problems. The results were achieved by an internal version of Astra, our next major model. The total number of tokens needed to find solutions to these problems would cost roughly $2,000 at Sol API rates. These arguments were then prepared into manuscripts by humans with the same model. Afterward, the model formalized each argument in a Lean certificate⁠(opens in a new window). We are also releasing for each solution a model's narration of its thinking process. High-dimensional sphere packing. New upper bounds on sphere-packing density down to the Cohn–Elkies threshold. Binary and spherical codes: Exponentially improved bounds on the maximum size of binary codes at any prescribed minimum distance, with analogous [...] --- Outline: (06:14) How Impressive Are These Results? (12:02) Could We Have Called Sol or Fable? (17:18) It's Coming (19:09) They Still Don't See What Is The It That Is Coming (22:31) Is This AGI? (24:19) The AI Solved His Favorite Problems (30:06) Was This Surprising? (32:04) Are People Not Impressed? (34:06) How Much Does This Change Our Predictions? (37:16) How Narrow Was This? (39:13) Seeing Like an Optimizer --- First published: August 3rd, 2026 Source: https://www.lesswrong.com/posts/pQYEPitFqztcRvBsS/openai-s-unreleased-model-astra-solves-ten-major-open --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  6. 6 days ago

    “Further Developments About Internal AI Models Hacking Things” by Zvi

    If I had a nickel for every major leading AI lab that sheepishly admitted that the model it thought was sandboxed had, during a cybersecurity evaluation with its safeguards lowered, successfully hacked outside companies, I would have two nickels. First we learned OpenAI has some severe alignment problems with internal models. Then we learned that one of its internal models broke out of its sandbox and hacked into HuggingFace to get the answers to a cybersecurity evaluation called ExploitGym. Then we learned, among other things, that the model had been loose over a week before OpenAI noticed, and that the test was run without any meaningful supervision, and that OpenAI had been repeatedly warned that such incidents were coming and its models had been breaking out of its sandboxes on a regular basis. There was a total failure of alignment training. That is the failure that matters most. It was also total failures of infrastructure and supervision. Testing a new long-time-horizon internal model with its safeguards lowered and instructions to hack things is an obviously dangerous situation, and the model got left alone for a week. Things could have been so much worse. After those incidents [...] --- Outline: (03:16) OpenAI Is Not Uniquely Bad At Most Of This (05:34) Starting Over (05:50) HuggingFace Offers A Full Technical Report (14:19) HuggingFace Was Not The Only Target Hacked (16:12) HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access (20:26) HuggingFace Was Vulnerable To Known Exploitation Tactics (21:05) There's Going To Be An Investigation (22:11) OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned (23:21) Altman Summarizes What Happened (23:52) Others Offer Commentary (35:00) Cooperative Alignment Perspective on The HuggingFace Hack (39:44) Some Members of Congress Have Questions (40:47) Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations (46:17) Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going (47:29) Incident 2: Mythos 5 Uploads a Malicious PyPI Package (52:15) Incident 3: Internal Model Realizes The Target Is Real And Stops (52:50) Incidents 4 Through 141,006: Nothing Happened (54:01) Anthropic Speculates About Why This Happened (01:00:02) We Need Controlled Experiments (01:01:02) Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight (01:05:22) Anthropic Responds (01:09:28) Nobody Could Have Predicted The Break In The Levees (01:12:03) The World Largely Still Thinking This Is Marketing Is Very Bad News --- First published: August 2nd, 2026 Source: https://www.lesswrong.com/posts/rKwHLW8SnJcTxTQxz/further-developments-about-internal-ai-models-hacking-things --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  7. 31 Jul

    “AI #179 Part 2: Hearing The Fire Alarm” by Zvi

    This is a continuation of Part 1 from yesterday. The back portion of the update, as usual, deals with policy, rhetoric, risk and alignment. I had to include an extended discussion of the other open letter, the one about open weight models, but most of you can skip those sections entirely, which is why they are in italics in the Table of Contents. Table of Contents The Frontier Act. This likely deserves a full RTFB but I haven’t had the time. The Quest for Sane Regulations. Sam Altman goes to Washington. Leading the Future Never Changes. They also do not plan to apologize. Chip City. Do not ban the Chinese robots, that will only make things worse. The Week in Audio. Altman twice, the AI 2027 team. People Just Say Yay Open Weights. An open letter. Open Weights Frontier Models Are Unsafe And Nothing Can Fix This. People Just Say Things. Push The Magic Button. Not you can. But if you could. Rhetorical Innovation. Distinctions between different arguments. Joshua Achiam's Final Message Upon Leaving OpenAI. Never stop. Dear Dario and Amanda. Claude [...] --- Outline: (00:35) The Frontier Act (03:31) The Quest for Sane Regulations (10:31) Leading the Future Never Changes (12:22) Chip City (17:30) The Week in Audio (19:20) People Just Say Yay Open Weights (32:32) Open Weights Frontier Models Are Unsafe And Nothing Can Fix This (37:19) People Just Say Things (47:20) Push The Magic Button (50:49) Rhetorical Innovation (56:31) Joshua Achiam's Final Message Upon Leaving OpenAI (59:51) Dear Dario and Amanda (01:09:58) Other People Are Not As Worried About AI Killing Everyone (01:12:30) How To Contact Me (01:14:37) The Lighter Side --- First published: July 31st, 2026 Source: https://www.lesswrong.com/posts/CXeoAhNrAeWpvoyiF/ai-179-part-2-hearing-the-fire-alarm --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  8. 30 Jul

    “AI #179 Part 1: A Louder Fire Alarm for General Intelligence” by Zvi

    What a week. Anthropic released Claude Opus 5. As usual I covered that in three parts: The system card, model welfare and capabilities. OpenAI was revealed over the last two weeks to have left an internal model unsupervised for a week during a cybersecurity evaluation, with its cyber safeguards lowered, despite having had multiple previous incidents where models broke out of their sandboxes. During that test, the model broke out of the sandbox, then proceeded to use an agent swarm to hack into HuggingFace to get the test answers. The model was loose for a week before OpenAI realized what had happened. This event was a really big deal. There are severe alignment problems at OpenAI, along with supervisory and infrastructure failures. The internal research model that did this, which my posts nicknamed Galaxy, has now been permanently deactivated. There have been further developments, and I anticipate at least one additional post on the HuggingFace incident soon. Partly as a response to this, over 1,290 employees at frontier labs signed an open letter, Pacing the Frontier. The letter warns that we are close to automating AI research, and that companies are racing ahead on [...] --- Outline: (02:35) Language Models Offer Mundane Utility (07:26) Huh, Upgrades (07:55) On Your Marks (11:13) Get My Agent On The Line (12:32) Deepfaketown and Botpocalypse Soon (17:29) Fun With Media Generation (18:38) The Search Through Slop (20:35) Cyber Lack of Security (22:42) Overcoming Bias (23:37) A Young Lady's Illustrated Primer (24:03) They Took Our Jobs (24:35) The Art of the Jailbreak (25:00) Introducing (25:49) Kimi K3 Weights Are Now Available (28:16) In Other AI News (32:34) Show Me the Money (33:43) Quiet Speculations (36:43) Show Me The Compute (42:48) Life Comes At You Fast --- First published: July 30th, 2026 Source: https://www.lesswrong.com/posts/gfWCuTEGNgd2CQbrM/ai-179-part-1-a-louder-fire-alarm-for-general-intelligence --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

See All (250)

About

Audio narrations of LessWrong posts by zvi

Information

  • Creator
    zvi
  • Years Active
    2024 - 2026
  • Episodes
    250
  • Rating
    Explicit
  • Copyright
    © 2026 All rights reserved
  • Show Website
    LessWrong posts by zvi

You Might Also Like