LessWrong (Curated & Popular)

LessWrong

Audio narrations of LessWrong posts. Includes all curated posts and all posts with 125+ karma.If you'd like more, subscribe to the “Lesswrong (30+ karma)” feed.

  1. 8h ago

    "LLMs are (still) mostly powered by imitative learning, not RL" by Steven Byrnes

    Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It's so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture. Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of: (1) Imitative learning, including pretraining and supervised fine-tuning (SFT) See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.(2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.[1] If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM's capabilities. And my claim is that it's way more (1) than (2). I'll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment. Note that I am not arguing that RLVR [...] --- Outline: (02:00) 1. Some relevant evidence (02:04) 1.1. Theoretically, each GPU-hour spent on RL should have orders of magnitude less contribution to LLM capabilities than a GPU-hour spent on imitative learning (03:06) 1.2. The chain-of-thought (CoT) is still obviously strongly influenced by imitative learning (04:34) 1.3. LLM companies still seem to care a lot about imitative learning (pretraining & SFT) data, not just RL environments (05:06) 1.4. Three papers claiming that non-RLVR'd models can get into the same ballpark of capabilities as RLVR'd models, although maybe we shouldn't trust those papers too much (06:56) 1.5. A paper suggesting that RLVR mostly refines the heuristics controlling which (already-known) reasoning strategy to use in which situation (09:02) 2. What am I actually claiming here? (11:30) 3. Why does any of this matter? (11:38) 3.1. Thinking about CoT legibility (both today and in the future) (15:08) 3.2. Thinking about LLM capabilities (both today and in the future) (16:33) 3.3. Thinking about LLM alignment (both today and in the future) --- First published: July 24th, 2026 Source: https://www.lesswrong.com/posts/wYpjXRLqbLbnmjbJP/llms-are-still-mostly-powered-by-imitative-learning-not-rl --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  2. 1d ago

    "Mathematicians are Feeling the Doom" by alkjash

    I think there's a critical opportunity for someone here. Mathematicians are feeling the doom (mostly in the "lose our jobs" sense). Academics are freaking out about the daily news that amateurs are asking GPT "prove career-defining theorem, make no mistakes" and it's just working. Senior researchers are leaving for frontier AI labs. Many are mentally spiralling or flailing about their life's work not mattering anymore. Last year, when I tried explaining IABIED to colleagues, I would be met with incredulous stares. This year, I'm met with incredulous stares and "So what should I do now?" (I don't have a good answer for them, which is part of why I'm posting this.) I've quarantined AI discussions on my research discord because otherwise it would overwhelm everything else. Top mathematicians, people on par in mathematical ability with Critch, Christiano, and Steinhardt, people who have been running leading-edge research groups for decades, people with enormous soft power in academic circles, are going to OpenAI without even having considered x-risk for five minutes. Many more will be leaving soon. If you want these folks to hear something at all, to consider some other option in the rest [...] --- First published: July 23rd, 2026 Source: https://www.lesswrong.com/posts/zKCGq2bbCQrpCNc3W/mathematicians-are-feeling-the-doom --- Narrated by TYPE III AUDIO.

  3. 1d ago

    "Not Pinning Your OpenRouter Provider Might Invalidate Your Research" by Matthew Khoriaty

    Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted. When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available provider.OpenRouter providers have variable quality. Ensuring that your provider is high quality is really difficult. There is precedent for an AI safety paper accepted to NeurIPS having its core results entirely overturned by these issues.A review of influential AI Safety research codebases that use OpenRouter for their reported results found that 31/32 (97%) of them use OpenRouter unsafely.[2]Researchers who wish to do research using OpenRouter or similar providers should take precautions to minimize the risks to their research,[3] though the current selection of providers is insufficient for ideal scientific reliability.Replicators should test whether results hold up when these bugs are fixed.Core AI Safety codebases should make fixes so that downstream users are able to implement best practices. This is a tangent I took for a few days during the Pivotal AI Safety Research Fellowship. I’m doing an AI Control project mentored by Adam Kaufman and James Lucassen of Redwood Research and partnered with Aniruddh [...] --- Outline: (02:04) OpenRouter Kills an AI Safety NeurIPS Paper [... 16 more sections] --- First published: July 23rd, 2026 Source: https://www.lesswrong.com/posts/KsyoSAyBRXtwzSugg/not-pinning-your-openrouter-provider-might-invalidate-your --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  4. 2d ago

    "Lightcone Commons" by habryka

    TLDR: Lightcone Commons is a new funding platform for coordinating large-scale ambitious philanthropy. We recruit thinkers with strong track records to make grant recommendations to funders. Anyone giving away 100 thousand dollars+ per year is welcome to join. We are facilitating ~20 million dollars of grants in our first round, and hopefully more every 3 months after that. Evaluators are paid 2% of recommendations, and we charge a 3% platform fee. We handle all logistics and due diligence for funding recommendations to charities, individuals, and for-profits. Funders maintain full control over their funds, and there are no vetoes or constraints on what recommendations we generate. Apply for funding here. I am launching Lightcone Commons, our software-first platform for distributing philanthropic funding. We connect funders with giving opportunities, coordinate splitting the bill with others who want to fund the same projects, and make it easy to defer to grant evaluators who vet applications and scout for new grantmaking opportunities. Funders can leave and join the platform at any time and without the need to commit any funds in advance. I and my team (together with SFC, Andrew Critch, and others) have been developing software used to distribute [...] --- Outline: (03:54) Funder FAQ (20:10) Applicant FAQ (26:17) Evaluator FAQ (34:14) Misc FAQ --- First published: July 23rd, 2026 Source: https://www.lesswrong.com/posts/tjeoLz2GzfFMysZhg/lightcone-commons --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  5. 2d ago

    "Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?" by Alex Mallen, Girish Gupta

    OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted[1]. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term. Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences. Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback. Background The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of motivations in which the AI poses risk because it intentionally [...] --- Outline: (01:17) Background (03:40) Implications (03:52) These AIs can't be trusted in an intelligence explosion (05:00) This misalignment poses direct takeover risk (07:29) What the incident tells us about takeover risk generally (08:45) The naive fixes likely make misalignment worse The original text contained 5 footnotes which were omitted from this narration. --- First published: July 23rd, 2026 Source: https://www.lesswrong.com/posts/H6DDSEvrtCk8Sehfd/are-we-existentially-threatened-by-the-type-of-ai --- Narrated by TYPE III AUDIO.

  6. 2d ago

    "We should push for no-fault liability for actions taken by AI" by Yair Halberstadt

    Before I start, I'll mention that I'm in contact with a world expert on legislation and regulation, who would be happy to help with this or similar work pro-bono. If you work in AI policy and believe this could help you, please reach out. OpenAI recently announced that one of their models successfully exploited multiple zero day vulnerabilities to gain secret information from Hugging Face. It has been pointed out that if a human undertook the same actions they could face multiple years in prison. It is clear that models are now reaching a level of capabilities that should be highly concerning regardless of whether you believe that AI represents an existential threat or not. Frontier AI models can and will be exploited by bad actors, but its now clear that they may cause undesirable outcomes even when their users are well intended. AI companies have until now been able to avoid taking responsibility for actions taken by their AI, including multiple cases where AIs were involved in murders and suicides. At the same time AI offers the potential for incredible good. While chatbots may have encouraged a number of suicides, they are almost certainly responsible for providing magnitudes [...] --- First published: July 22nd, 2026 Source: https://www.lesswrong.com/posts/Kj3YpqzhFySCjYcWi/we-should-push-for-no-fault-liability-for-actions-taken-by --- Narrated by TYPE III AUDIO.

  7. 3d ago

    "OpenAI Shares Some Alignment Problems" by Zvi

    Kudos to OpenAI for sharing their recent experiences with a misaligned internal model, where they encountered problems sufficiently severe they were forced to take the model offline to work on new mitigations and defense-to-depth. And also further kudos for actually taking the model offline for a time to build new safeguards. They gave us one hell of a candid report. The tone is professional throughout, whereas my reaction reading it was less professional and more this: With a mix of this: It was not shared on the official account because OpenAI worried about it being seen as self-promotional hype. It is crazy that one needs to worry about that, but also plausibly a real concern. So again, good decision. Not that any of the behaviors or failures here are unexpected, exactly. Not by the AIs and not by the humans. Yet there is something I would call a missing mood, a failure to realize the gravity of the situation. There are some who responded ‘what part of this was unexpected, exactly?’ And that is actually fair, but that is also the problem. We have become numb to all this. We expect the models to [...] --- Outline: (02:49) Good News Bad News [... 7 more sections] --- First published: July 21st, 2026 Source: https://www.lesswrong.com/posts/KctxwGKxm9fHtwh6u/openai-shares-some-alignment-problems --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  8. 4d ago

    "OpenAI Models Behind HuggingFace Cybersecurity Incident" by LawrenceC

    From the OpenAI blog post: Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities. We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete. --- First published: July 21st, 2026 Source: https://www.lesswrong.com/posts/WpuRdcMfFeiLeXkxL/openai-models-behind-huggingface-cybersecurity-incident --- Narrated by TYPE III AUDIO.

See All (915)

Ratings & Reviews

4.7
out of 5
14 Ratings

About

Audio narrations of LessWrong posts. Includes all curated posts and all posts with 125+ karma.If you'd like more, subscribe to the “Lesswrong (30+ karma)” feed.

Information

  • Creator
    LessWrong
  • Years Active
    2022 - 2026
  • Episodes
    915
  • Rating
    Explicit
  • Copyright
    © 2026 LessWrong (Curated & Popular)
  • Show Website
    LessWrong (Curated & Popular)

You Might Also Like