LessWrong posts by zvi

zvi

Audio narrations of LessWrong posts by zvi

  1. 21h ago

    “AI #186: The World Takes Notice” by Zvi

    In the wake of Jacob Coxon's resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up. Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety. The people took notice, raising both the salience that AI might kill everyone and roughly doubling people's estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress. The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House. Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and [...] --- Outline: (03:16) Language Models Offer Mundane Utility (03:57) Language Models Don't Offer Mundane Utility (04:06) Huh, Upgrades (04:30) On Your Marks (04:55) Deepfaketown and Botpocalypse Soon (06:56) Cyber Lack of Security (07:31) Astra Is Hard To Monitor (08:02) Get Involved (08:11) Introducing (09:32) In Other AI News (10:26) Now You Know (14:32) Hugging the Face (16:53) Swarm of Undiscovered Swarms of Rogue OpenAI Agents (21:59) Show Me the Money (23:40) Quiet Speculations (24:41) White House Officials Attempt To Act Sanely (26:13) Democrats React Sanely to AI Potentially Killing Everyone (32:49) Pacing the Frontier (33:38) Guest Lecture from Alex Tabarrok on Regulatory Capture (41:22) Mark Zuckerberg Offers Thoughts (42:46) Megan McArdle On The Inadequacy Of Current Legal Frameworks (44:35) Pick Up the Phone (47:35) The Week in Audio (50:31) People Just Say Things (53:32) Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone (55:19) Rhetorical Innovation (57:54) Exhuming McCarthy (01:01:03) A Very Different Perspective (01:02:51) It's Even Rougher Out There (01:03:41) If We Wanted To (01:04:38) Open Weights Are Unsafe And Nothing Can Fix This (01:10:56) From The Famous Cautionary Tale (01:13:49) Reporting On All Your Misalignment Incidents Is Difficult (01:19:35) Aligning a Smarter Than Human Intelligence is Difficult (01:22:13) Storytime With Owain Evans (01:26:50) A Different Autonomous Swarm (01:30:44) Cooperative Alignment (01:32:58) Uncooperative Alignment (01:39:18) People Are Worried About AI Killing Everyone (01:41:25) The Lighter Side --- First published: September 17th, 2026 Source: https://www.lesswrong.com/posts/aa3HprreFktzLQiaW/ai-186-the-world-takes-notice --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  2. 1d ago

    “Trump Goes Full Hoax on AI Existential Risk” by Zvi

    This is our reality. I suppose we have to talk about it. Everyone in a position to know is freaking out about AI potentially killing everyone this decade and wants to pace the frontier, and people are finally listening. It only took a few days for the conversation to fully pivot to the counteroffensive, where the Usual Suspects and those they recruited attacked anyone and everyone who dared point out that we are in danger, with every attack they can think of, usually without substance or any attempt at understanding. Sigh. I knew what I signed up for. Table of Contents Hold Your Fire. If You Don’t Like the Weather. Trump Does Not Take Kindly. Trump Goes Full ‘Hoax’. This Is Not About Data Centers, Mr. President. I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus. Nvidia CEO Jensen Huang Is a Lying Liar. Trump Quietly Draws Key Distinction. Calling For Pacing the Frontier Is Bad For AI Stock Prices. People On The Internet Sometimes Lie. Origins of Cynicism. Ineffective Egoism. The McCarthyist Faction Attacks [...] --- Outline: (00:44) Hold Your Fire (01:37) If You Don't Like the Weather (02:33) Trump Does Not Take Kindly (05:12) Trump Goes Full 'Hoax' (06:36) This Is Not About Data Centers, Mr. President (08:06) I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus (10:10) Nvidia CEO Jensen Huang Is a Lying Liar (14:20) Trump Quietly Draws Key Distinction (15:21) Calling For Pacing the Frontier Is Bad For AI Stock Prices (19:05) People On The Internet Sometimes Lie (19:54) Origins of Cynicism (22:18) Ineffective Egoism (24:53) The McCarthyist Faction Attacks METR (32:25) Other Key Republicans React (36:49) David Sacks Stops Being Plausibly Constructive (37:39) Federal Trade Commission Chooses Danger (38:39) Chris Lehane Heel Face Turn (40:43) A Matter of Trust (41:55) China Calls It Fearmongering (43:17) Pick Up The Phone (47:41) If You Want To Beat China So Badly You Should Act Like It (48:49) Never Go Full Hoax (49:55) Trump Uses AI For Things --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/Kqgco8vLFMeBhdrQY/trump-goes-full-hoax-on-ai-existential-risk --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  3. 2d ago

    “The Bad Guy With An AI Named Claude” by Zvi

    A lot of bad guys try to use Claude to do bad things. Mostly they fail. We think. Anthropic has disrupted a bunch of them, and offers an extensive report. If Anthropic is sharing the worst cases, or anything close to them, things are actually looking good on the misuse front for closed models, even better than I thought. This report covers activity we disrupted between December 2025 and August 2026 across seven harm areas: cyber operations, influence operations, surveillance, scams and fraud, biological misuse, conventional weapons development, and distillation. There's a bit of Arson, Murder and Jaywalking there. One of these things, many would say, is not like the others. I do not agree, especially given the details we will see later, and given that distillation enables the other six via, as the report says, ‘driving performance on nearly every task’ via transfering Claude's cognitive skills, without transferring its safeguards. Indeed, distillation is by far the most important threat in this report, and the part of the report that will have the most impact. By exposing Chinese attempts at systematic fraudulent distillation of Claude, Anthropic has embarrassed and potentially antagonized the Chinese. [...] --- Outline: (02:08) Breaking Unrelated News (03:23) How To Not Tell a Fable (03:58) Bad Dudes Tend To Be Relatively Unsophisticated (05:45) Particular Bad Dudes (08:09) Influence Operations (13:09) Surveillance Operations (15:22) Conventional Weapons (17:30) Biological Misuse (18:54) Scams and Fraud (20:15) Illicit Fraudulent Distillation (32:45) What You Gonna Do About It, Punk? (34:53) Good News, Everyone (35:39) A Very Different Read of The Report --- First published: September 15th, 2026 Source: https://www.lesswrong.com/posts/qSjH9T83xCfWQkmk2/the-bad-guy-with-an-ai-named-claude --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  4. 3d ago

    “We Must Pace The Frontier” by Zvi

    Dario Amodei has a new essay that finally says the thing: We Must Pace the Frontier, naming his call after the Pacing the Frontier letter lab employees signed in July. As in, we need to slow the rate at which AIs increase their capabilities, to allow for the necessary alignment and safety work. He explained that, without pacing, he expects things to escalate quickly. He offered three proposals, and unilaterally committed to the first one. OpenAI followed, and both Elon Musk and Demis Hassabis endorsed the overall proposal. There is still a long way to go. The odds are still against us. The situation remains grim. The hard part lies ahead. We do not agree on what ‘Pacing the Frontier’ will mean in practice. But this is Actual Progress. The work can begin. Table of Contents Pacing Does Not Mean Pausing. Dario's First Proposal: Embedded Evaluators. Dario's Second Proposal: Democratic Coordination. Dario's Third Proposal: Global Coordination. Why Pace Now? Sam Altman Agrees and Commits to Embedded Evaluators. OpenAI Will Not IPO This Year. Elon Musk Agrees. Demis Hassabis Agrees. Microsoft CEO Satya Nadella Agrees [...] --- Outline: (01:02) Pacing Does Not Mean Pausing (02:53) Dario's First Proposal: Embedded Evaluators (11:05) Dario's Second Proposal: Democratic Coordination (14:36) Dario's Third Proposal: Global Coordination (17:24) Why Pace Now? (19:22) Sam Altman Agrees and Commits to Embedded Evaluators (24:42) OpenAI Will Not IPO This Year (26:08) Elon Musk Agrees (29:00) Demis Hassabis Agrees (29:38) Microsoft CEO Satya Nadella Agrees And Talks His Book (31:50) Anthropic's Long-Term Benefit Trust Is On Board (32:49) General Online Reactions (34:30) Mainstream Press Coverage (37:26) OpenAI Researcher Explains What The Labs See And It's a Rocket Ship (46:10) Consider the Alternative (47:47) It's Totalitarianism, Joe (53:35) Yes We're The Baddies How Did You Know? (55:22) Sometimes People On the Internet Just Lie (56:50) David Sacks Groks The Situation (57:48) David Sacks Says Go Ahead (59:28) Lies and Confusions About Who Previously Claimed What (01:04:07) House Speaker Mike Johnson Wants To Lock Everyone In a Room (01:09:28) Donald Trump is Not Tired of Winning (01:14:29) We Must Avoid Polarization on AI at (Almost) All Costs (01:15:14) How Will We Know If They Actually Paced? (01:18:24) The Real Frontier Is Internal Models At Top Labs --- First published: September 14th, 2026 Source: https://www.lesswrong.com/posts/iWPDPWAPCGiSMiFA2/we-must-pace-the-frontier --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  5. 4d ago

    “Brand New AI Solves a Millennium Prize” by Zvi

    The first Millennium Prize, Navier-Stokes, has fallen to AI. A deeply unfortunate situation has arisen involving what should have been some combination of a positive story about new progress in AI-assisted mathematical research and yet another opportunity to freak out about rapid AI progress. Or, as we call it around here, Tuesday. The Real Story Is The New Model That Is Better Than Astra Keep your eyes on the prize. There are three stories here. The first story is much more important than the second story, which in turn is much more important than the third story. OpenAI's next model took a week to get a generation ahead of Astra, and they are telling us this because everyone is totally freaked out about what is happening. Or, in official language: ‘We believe it is important to inform the world about the pace of AI progress and what to expect from upcoming models’ and that we ‘may require more deliberate choices about the pace of progress.’ This new AI has, eight days after it started training, solved Navier-Stokes. A bunch of drama over who gets the credit [...] --- Outline: (00:31) The Real Story Is The New Model That Is Better Than Astra (01:47) Setting the Stage (03:47) I Heard a Rumor (04:32) Our Price Cheap (06:14) An Accusation Is Made (10:57) How Did You Get That Idea? (12:59) OpenAI Almost Certainly Did Not Misappropriate User Data (14:34) Our Top Labs Cannot Get Along Even On A Feel-Good Math Story (15:52) What Next? (16:39) The Mathematicians Are Not Happy (21:14) OpenAI's New Model Was A Step Change Above Astra Four Days Into Training (23:16) OpenAI Research Progress Is Accelerating Due To OpenAI Research Progress (30:39) Quantifying OpenAI's Pause (31:09) This Is the Way the World Ends (31:44) Quickly, There's No Time --- First published: September 13th, 2026 Source: https://www.lesswrong.com/posts/uoZW6BKaCcmNQrWis/brand-new-ai-solves-a-millennium-prize --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  6. 5d ago

    “GPT-6-Astra Can Do Ambitious Things” by Zvi

    Astra is an excellent model. The jump from Sol to Astra is larger than the jump from Fable 5 to Fable 5.1. This is a big deal. Astra is the best model for what one would broadly call ‘ambitious projects,’ and likely has the highest raw intelligence factor of any model. These are the largest jumps. It is amazing at doing things in 3D, or anything involving games. Astra also excels at computer use, and at subagent coordination. Many benchmarks show dramatic jumps from all previous models. Where Astra is good, it can be in a league of its own. That does not mean Astra is in its own league across the board. Fable 5.1 is still a Claude. Astra is still a GPT. If you have a strong preference for one over the other, that still applies. For many purposes, especially involving back-and-forth discussions, Fable 5.1 is still my top choice. Fable remains my primary editor. If you want the best answer to your questions, you should ask both models. Regular coding is getting less of a focus. Astra is not a quantum leap there, but of course it is very good [...] --- Outline: (02:34) Meanwhile (04:38) The Official Pitch (13:08) Our Price Cheap (14:06) Unnecessary Overstatement (15:53) Paced Rollout (16:32) Official Benchmarks (22:37) Other People's Benchmarks (30:47) Thinking, Fast Without Slow (34:25) How Dare You, Sir (36:36) PoetryBench (37:54) In 3D (39:14) Time to Think (39:59) I'm Putting Together a Team (41:02) Reviews and Essays (41:37) Computer Use (42:47) Positive Reactions (50:53) AGI (54:20) Astra Can Do The Math (57:27) Astra Can Code (59:35) I Came to (Change the) Game (01:02:37) Astra Does Other Cool Things (01:03:31) Astra Never Quits Except When It Does (01:05:26) Negative Reactions (01:07:45) Stop It With the Hedging (01:08:57) Personality Clash (01:09:53) Revealed Preference (01:12:03) Dual Wielding The original text contained 1 footnote which was omitted from this narration. --- First published: September 12th, 2026 Source: https://www.lesswrong.com/posts/snaKjCwazKcRiS4qs/gpt-6-astra-can-do-ambitious-things --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  7. 6d ago

    “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade” by Zvi

    CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks. They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things. These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions. A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out. After all the events, plus statements by Dean Ball and Jakub Pachocki, we were already seeing the beginnings of a preference cascade. Then along came Jacob Coxon as the tipping point, and things took off. Table of Contents [...] --- Outline: (01:22) Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm (05:46) Mainstream Media Finally Pays Attention (06:47) Preference Cascade at Anthropic (10:18) Preference Cascade at OpenAI (13:26) Preference Cascade at Google (14:43) #NotAllMembersOfTechnicalStaff (15:23) Why a Preference Cascade Now? (21:01) This Is What Many Anthropic and OpenAI Employees Actually Believe (24:08) To Quit Or Not To Quit (31:29) Quiet Quitting Is A Dominated Option (32:50) When You Quit, Very Serious People Understand What That Means (39:29) Jacob Coxon Believes Existential Risk Is High That Is Why He Quit (41:47) Evan Hubinger Believes Existential Risk Is High That Is Why He Stays (45:22) Anthropic and OpenAI Have Commercial Incentives To Downplay Existential Risks, Not Advertise Them (50:54) What Do We Do Now? (53:03) OK, But How Exactly Would AI Kill Everyone? (01:01:44) Best Start Believing In Science Fiction Stories Because You Are In One (01:06:20) Literal Extinction Is Not Much Harder Than Loss of Control (01:07:42) Conspiracytown Is Always Hiring (01:20:14) Now You See It --- First published: September 11th, 2026 Source: https://www.lesswrong.com/posts/5MB7KENgEAW6Q4JtJ/jacob-coxon-warns-of-human-extinction-and-triggers-a --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  8. 6d ago

    “The Extinction Risk Preference Cascade: Quotes” by Zvi

    These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon's warnings, in which the employees confirm that they think AI might soon kill everyone. If more quotes come in over the next week or so, I will update this post accordingly. Preference Cascade Statements At OpenAI: Tomek Korbak Tomek Korbak (OpenAI): i’m late to the party but: from his time at OpenAI I remember Jacob as a very thoughtful researcher and he continues to be so in this thread. neither anthropic nor openai are on track to solve alignment to a degree sufficient for shipping superintelligence and we need to slow down Vie McCoy Vie McCoy (OpenAI): I think pacing progress and ensuring human enhancement is the only way that we don’t get out-evolved while retaining the dream of superintelligence. In this context, I see two paths before us. In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses. In the second, we set the pace of progress, focus on embedding human flourishing [...] --- Outline: (00:27) Preference Cascade Statements At OpenAI: Tomek Korbak (00:55) Vie McCoy (03:46) Adam Majmudar (05:05) Aidan Clark (05:46) Mo Bavarian (07:32) Boaz Barak (08:20) Micah Carroll (09:15) Roon (12:55) Confirmations At OpenAI: Dean Ball (14:39) Leo Gao (14:57) Anthropic's Evan Hubinger Confirms His Stance (15:50) Preference Cascade at Anthropic: Samuel Marks (17:36) Anna Wang (18:22) Ethan Perez (18:59) Dima Krasheninnikov (19:27) EigenGender (Anon Account) (19:58) Joe Benton (20:07) Confirmation at Anthropic: Drake Thomas (21:28) Jan Lieke (22:07) Sluggy (22:29) Preference Cascade at Google (22:56) Andreas Kirsch (23:39) Neel Nanda (24:15) Victoria Krakovna (25:26) Vishal Maini (27:01) Joe (OpenAI, ex-Google) (28:21) Josh Engels (28:39) Geoffrey Irving (29:30) Alex Turner and Geoffrey Hinton: Classic Examples (29:45) #NotAllMembersOfTechnicalStaff: Ted Sanders --- First published: September 11th, 2026 Source: https://www.lesswrong.com/posts/APGvWZtXEwkinvHDd/the-extinction-risk-preference-cascade-quotes --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

See All (250)

Ratings & Reviews

5
out of 5
2 Ratings

About

Audio narrations of LessWrong posts by zvi

Information

  • Creator
    zvi
  • Years Active
    2024 - 2026
  • Episodes
    250
  • Rating
    Explicit
  • Copyright
    © 2026 All rights reserved
  • Show Website
    LessWrong posts by zvi

You Might Also Like