LessWrong (30+ Karma)

LessWrong
  • 0.0 (0)
  • Technology
  • Updated daily

Audio narrations of LessWrong posts.

  1. 2 hr ago

    “How robust are natural language autoencoders to initialization?” by michaelzhang, TurnTrout

    Natural language autoencoders are meant to take in an LLM's activation vector and describe in plain text what the model is thinking. However, its training data collection involves asking Claude to guess what a model might be thinking. How robust are NLAs to these guesses? We change Claude's guesses in various ways and measure the impact on the NLA's statements as well as on reconstruction accuracy. We show that Qwen2.5-7B NLAs have some robustness to irrelevant statements and prevailing sentiments in Claude's guesses. However, if an NLA is initialized with entirely implausible statements, it can nevertheless achieve nearly the same reconstruction accuracy as plausible-initialized NLAs while emitting 99.3% implausible statements. RL does train implausible-initialized NLAs to be slightly more plausible (increasing from 0.08% to 0.7%). But the plausibility of plausible-initialized NLAs decreases from 21% at initialization to 7.6% at the end of training. If our results scale, they cast doubt on the usefulness of NLAs. Produced as part of the MATS program in the summer 2026 cohort of team shard. Terminology A "plausible" explanation is an objectively true statement about the world. For example, given a passage about greyhounds, a plausible explanation of model [...] --- Outline: (02:16) Introduction (05:06) The experimental setup (06:36) The "Carthago delenda est" experiment (08:15) The "I love Carthage" experiment (11:24) The "confabulation" experiment (20:51) The outputs of plausible-initialized and implausible-initialized NLAs (23:41) Limitations (26:17) Conclusions --- First published: July 10th, 2026 Source: https://www.lesswrong.com/posts/LQXWiF8PyJ5ojNsEv/how-robust-are-natural-language-autoencoders-to --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    28 min

  2. 6 hr ago

    “Selective Optimism: a critique of AI 2040” by Richard_Ngo

    Some context for this post: I’ve been working part-time as a consultant for the AI Futures Project over the last year. Most of the work I’ve done for them has involved critiquing and suggesting improvements for their AI 2040 scenario—some of which were addressed, and some of which weren’t. To their credit, they asked me to write up my remaining critiques into a post that would accompany its launch. In the rest of this post I’ll discuss my three biggest high-level criticisms of AI 2040. Before doing so, I want to emphasize that there are many interesting and thought-provoking details in the scenario. I’ve focused on the high-level framing of the scenario because that's where my main disagreements lie; given the scope of these disagreements, it's hard to evaluate the details. Since the AI Futures Project paid me to develop and write this criticism, you shouldn’t take this as a fully unbiased perspective. However, they haven’t reviewed this piece, and in general have been open-minded about receiving criticism (as their request for me to post this today demonstrates). Finally: the preview image for the substack version of this post comes from this video of a dad shouting to his [...] --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/BBd2EJywf2xXftyFn/selective-optimism-a-critique-of-ai-2040 --- Narrated by TYPE III AUDIO.

  3. 9 hr ago

    “Debate with Self-Play Best-of-N Optimization” by Dewi Gould, Sam Martin, Alejandro Aristizabal, Simon Marshall, Jacob Pfau

    Debate is a proposed protocol for scalable oversight. As tasks outrun direct supervision, labs are increasingly likely to train against protocols like it. Our concern is that, for questions which are hard to verify, models will become more compelling more quickly than they will become more accurate – this could undermine alignment research and safe use. Whilst existing public empirical work mostly focuses on debate as an evaluation protocol (does debate help a judge reach better verdicts?), there is limited work using debate as a reward signal for training. This note is the first in a series aimed at building an open, empirical science of debate training. We show that inference time optimization, via best-of-N (BoN), can be used to iterate on debate protocols – de-risking training runs before committing to RL. By building up a careful, controlled understanding of how optimization pressure interacts with protocols, we lay the groundwork for tackling higher-level questions with confidence. We introduce an inference-time proxy for debate training. Studying debate protocols using BoN allows us to scale optimization on different players independently and identify which parts of the debate game are doing work. We believe that BoN provides sufficient optimization power to [...] The original text contained 9 footnotes which were omitted from this narration. --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/hb8pv3zyAHGJpwz9F/debate-with-self-play-best-of-n-optimization --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  4. 10 hr ago

    “How big is the Sun? How could you figure it out?” by Elliott Thornley

    I’m a few chapters into Our Mathematical Universe by Max Tegmark. By this point he's covered the ingenuities of the ancient Greeks, taking my knowledge of physics to within two and a half thousand years of the cutting edge. And what ingenuities they were. A whole series of them, strung together, and culminating in a pretty good estimate of the size of the Sun. I think it's a remarkable feat to even just ask the question ‘How big is the Sun?’ and recognize that it has an answer. The fact that the ancient Greeks actually managed to figure it out (more or less) is astonishing. I don’t know the exact path they took with their reasoning. History is messy. But here's a plausible route they could have taken, basically matching the one you’ll find charted in OMU. Step 1: Discover that the Earth is round. One way to discover that the Earth is round is to wait for a lunar eclipse, where the Earth casts its shadow on the Moon. If you look carefully, you’ll notice that the shadow is curved. Another sign of Earth's roundness is the way that departing ships disappear over the horizon: hull-first and mast-last. [...] --- Outline: (00:57) Step 1: Discover that the Earth is round. (01:37) Step 2: Use the disappearing-ship trick to estimate Earth's size. (02:24) Step 3: Use Eratosthenes' shadow trick to estimate Earth's size again. (03:43) Step 4: Use the similarity of your estimates to surmise that the Sun is really far away. (04:51) Step 5: Use the Sun's distance and apparent size to infer that it's bigger than the Earth. (06:02) Step 6: Use the Earth's shadow on the moon to estimate the Moon's size. (08:15) Step 7: Use the Moon's real size and apparent size to estimate its distance. (08:58) Step 8: Use the Moon's distance to estimate the Sun's distance. (10:24) Step 9: Use the Sun's distance and apparent size to estimate its real size. (10:56) Epilogue The original text contained 3 footnotes which were omitted from this narration. --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/Bc4Ch63cx8KHdQABw/how-big-is-the-sun-how-could-you-figure-it-out --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    12 min

  5. 11 hr ago

    [Linkpost] “AI 2040: Plan A” by Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

    This is a link post. For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it's done! It's called AI 2040: Plan A. It's called Plan A because it's a recommendation, not a prediction. It's what we think should happen, not what will happen, though we think it's plausible enough to aim for. It's called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments. As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia. Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer. What's next for us? Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that [...] --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a Linkpost URL:https://www.ai-2040.com/ --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    2 min

  6. 13 hr ago

    “Announcing our $160M grant from Coefficient Giving” by Geoffrey Irving, Jesse Hoogland, Alex HT, Jacob Pfau, Daniel Murfet, Marco Cozzi, Stan van Wingerden

    We are excited to announce that Resolution (fka Sequent) has a 160 million dollars grant from Coefficient Giving (cG) to put rigorous alignment research on a (closer to) even footing with the frontier labs. We will use it to accelerate progress towards higher-confidence alignment, or to find evidence and obstacles showing why alignment is hard. The grant is structured as a 108 million dollars base plus 52 million dollars conditional on a combination of hiring success and compute needs. The base includes a small regranting budget, which we plan to use both for high-quality non-Resolution alignment research and to give back to shared community infrastructure that we depend on. Coefficient Giving will be our sole funder to start (thank you!); our goal is to raise larger-scale funds from a mixture of sources once we demonstrate success with semiautomated alignment theory and empirics. The time for ambitious alignment funding is now In our launch announcement, we argued that the time to automate alignment is now: frontier systems have finally reached the threshold where they can make nontrivial theoretical progress, and theoretical research has access to more sources of ground truth than empirical metric climbing. There is a second, structural [...] --- Outline: (01:09) The time for ambitious alignment funding is now (02:20) Please join us in speeding up higher-confidence alignment! (03:38) Join us in closing the gap --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/HDKQNqiR2gtfMiWsn/announcing-our-usd160m-grant-from-coefficient-giving --- Narrated by TYPE III AUDIO.

    5 min

  7. 14 hr ago

    “Because 8 ≈ e², Anthropic’s researcher uplift is plausibly >2x” by Thomas Kwa

    Note: the modeling assumptions and conclusion are Thomas Kwa's opinion, and others at METR disagree. [1] Also, the math was checked by Claude but not a second human. Introduction Anthropic's RSI blog post reported that in Q2 2026, Anthropic contributors merged 8× as much code per day as in the 2021-2024 period. What does this imply about the factor by which a researcher's total effective output increased — the (serial) researcher uplift [2] ? Of course, 8× more code doesn't mean 8× more research, as coding is only part of the job. However, if we assume each line of code (LoC) has equivalent quality and verbosity to pre-2025 code and make standard economic modeling assumptions, we can conclude a surprising amount: all models predict researcher uplift at Anthropic from coding agents alone is >2×. (Researcher uplift could be even higher, because these numbers assume no uplift on non-code tasks.) Cobb-Douglas predicts that if pre-AI time spent coding is and code output increases by a factor , then researcher uplift is . CES (constant elasticity of substitution) production functions, due to a fun mathematical coincidence, infer a narrow range of about if code is [...] --- Outline: (00:25) Introduction (03:09) Economic models (03:57) Cobb-Douglas model (04:57) CES model (05:26) Setup (06:31) Because 8 ≈ e², the estimate is robust to σ! (09:15) M = 8 also implies a lower bound on σ (10:47) Code heterogeneity model: What if AI speeds up high-stakes code less? (12:22) Caveats: How could Anthropic's uplift be less than 2x? (12:36) Plausible reasons (12:39) Verbosity (14:42) Barely useful code (16:14) Irrational time allocation, intrinsic desire to use AI, etc. (17:17) Unlikely reasons (17:21) Extreme heterogeneity (18:04) Changing denominator (18:34) Discussion (18:37) Prefer code output over code uplift, for estimating overall uplift (19:40) Why is Anthropic's own estimate much lower? (21:55) There are reasons to expect even higher uplift (22:46) Conclusion (24:03) Appendix: Heterogeneous code model The original text contained 8 footnotes which were omitted from this narration. --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/ix5qEyW9BjGEb4d8k/because-8-e-anthropic-s-researcher-uplift-is-plausibly --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    26 min

  8. 15 hr ago

    “Optimiser Choice Can Amplify or Suppress Emergent Misalignment” by Jason R Brown, Patrick Leask, Lev McKinney

    This is a linkpost for https://arxiv.org/abs/2606.31591. Work done with Patrick Leask and Lev McKinney during the Astra Fellowship. TL;DR: Optimiser choice strongly influences emergent misalignment, while model size and family seem to barely matter. Optimisers that concentrate the LoRA update into fewer directions degrade alignment more, but regularising towards a flatter spectrum can mitigate this and improve alignment. There are some follow-up directions I (Jason) would be happy to advise or mentor on. Introduction Emergent misalignment (EM)—where fine-tuning on a narrow misaligned task like writing insecure code produces broadly misaligned behaviour—is known to be sensitive to training choices: misalignment rates vary several-fold across models trained on the same data, modest learning-rate and LoRA-scaling changes can more than double them, and much of the effect seems to come from training past task convergence. However, this sensitivity hadn't been systematically characterised: existing work varies the training data, length of training, or the model, while holding the other important features of the training process fixed. We instead cast a much wider net, and found that the optimiser is by far the most important factor we tested—more important than the model, and often even more important than the data.[1] What we found Model [...] --- Outline: (00:44) Introduction (01:34) What we found (04:41) Future directions (07:39) Acknowledgements The original text contained 4 footnotes which were omitted from this narration. --- First published: July 9th, 2026 Source: https://www.lesswrong.com/posts/Wq6CaAbiixoCEzbat/optimiser-choice-can-amplify-or-suppress-emergent-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    9 min
See All (250)

About

Audio narrations of LessWrong posts.

Information

  • Creator
    LessWrong
  • Years Active
    2023 - 2026
  • Episodes
    250
  • Rating
    Clean
  • Copyright
    © 2026 All rights reserved
  • Show Website
    LessWrong (30+ Karma)

You Might Also Like