Redwood Research Blog

Redwood Research
Redwood Research Blog

Narrations of Redwood Research blog posts. Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

  1. 1D AGO

    “AI safety techniques leveraging distillation” by Ryan Greenblatt

    Subtitle: Distillation is cheap; how can we use it to improve safety?. It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.1 I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper than the original training.2 In this post, I'll discuss how we could utilize distillation to potentially remove (or possibly detect) misalignment. I'll also discuss a few other applications.3 My overall take is that techniques utilizing distillation are mildly to moderately promising and the low cost of distillation might make them surprisingly viable, but it's quite tricky to reason about how effective these techniques are. Distilling to remove misalignment I'll [...] --- Outline: (01:08) Distilling to remove misalignment (11:52) Detecting problematic actions using (distillation) training (15:46) How does distillation interact with neuralese? (16:45) Other techniques leveraging distillation (16:49) Distillation as a (non-mechanistic) interpretability technique (18:03) Distillation for precise capability and knowledge control --- First published: June 19th, 2025 Source: https://redwoodresearch.substack.com/p/ai-safety-techniques-leveraging-distillation --- Narrated by TYPE III AUDIO.

    22 min

  2. JUN 12

    “When does training a model change its goals?” by Vivek Hebbar, Ryan Greenblatt

    Subtitle: Can a scheming AI's goals really stay unchanged through training?. Here are two opposing pictures of how training interacts with deceptive alignment: “goal-survival hypothesis”:1 When you subject a model to training, it can maintain its original goals regardless of what the training objective is, so long as it follows through on deceptive alignment (playing along with the training objective instrumentally). Even as it learns new skills and context-specific goals for doing well on the training objective, it continues to analyze these as instrumental to its original goals, and its values-upon-reflection aren’t affected by the learning process. “goal-change hypothesis”: When you subject a model to training, its values-upon-reflection will inevitably absorb some aspect of the training setup. It doesn’t necessarily end up terminally valuing a close correlate of the training objective, but there will be some change in values due to the habits incentivized by [...] --- Outline: (03:27) Empirical evidence so far (03:31) Alignment faking (06:27) Sleeper agents (13:50) Synthesizing the sleeper agents and alignment faking results (15:20) Smaller-scale experiments similar to sleeper agents (17:39) Urges vs. reflectively endorsed goals (18:38) Human analogies (18:50) A million years of laying bricks (21:34) Doing a job for 40 years, with/without a heroin drip (22:24) Murder Simulator (24:08) Call of Duty addiction (25:20) Would runners self-modify? (27:01) Terminally valuing money (27:52) Evolution vs. within-lifetime learning (28:54) Author contributions & acknowledgements --- First published: June 12th, 2025 Source: https://redwoodresearch.substack.com/p/when-does-training-a-model-change --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    30 min

  3. MAY 28

    “The case for countermeasures to memetic spread of misaligned values” by Alex Mallen

    Subtitle: Defending against alignment problems that might come with long-term memory. As various people have written about before, AIs that have long-term memory might pose additional risks (most notably, LLM AGI will have memory, and memory changes alignment by Seth Herd). Even if an AI is aligned or only occasionally scheming at the start of a deployment, the AI might become a consistent and coherent behavioral schemer via updates to its long-term memories. In this post, I’ll spell out the version of the threat model that I’m most concerned about, including some novel arguments for its plausibility, and describe some promising strategies for mitigating this risk. While I think some plausible mitigations are reasonably cheap and could be effective at reducing the risk from coherent scheming arising via this mechanism, research here will likely be substantially more productive in the future once models more effectively utilize long-term memory. [...] --- Outline: (01:19) The memetic spread threat model (06:54) Countermeasures (06:57) How much do existing safety agendas help? (09:13) Targeted countermeasures for memetic spread of misaligned values (11:57) Discussion --- First published: May 28th, 2025 Source: https://redwoodresearch.substack.com/p/the-case-for-countermeasures-to-memetic --- Narrated by TYPE III AUDIO.

    14 min

  4. MAY 12

    “AIs at the current capability level may be important for future safety work” by Ryan Greenblatt

    Subtitle: Some reasons why relatively weak AIs might still be important when we have very powerful AIs. Sometimes people say that it's much less valuable to do AI safety research today than it will be in the future, because the current models are very different—in particular, much less capable—than the models that we’re actually worried about. I think that argument is mostly right, but it misses a couple of reasons models as capable as current models might still be important at the point where AI is posing substantial existential risk. In this post, I’ll describe those reasons. Our best trusted models might be at a similar level of capabilities to current models: It might be the case that we are unable to make models which are much more powerful than current models while still being confident that these models aren't scheming against us. That is, the most powerful [...] --- First published: May 12th, 2025 Source: https://redwoodresearch.substack.com/p/ais-at-the-current-capability-level --- Narrated by TYPE III AUDIO.

    7 min

  5. MAY 8

    “Misalignment and Strategic Underperformance: An Analysis of Sandbagging and Exploration Hacking” by Julian Stastny, Buck Shlegeris

    Subtitle: A new analysis of the risk of AIs intentionally performing poorly.. In the future, we will want to use powerful AIs on critical tasks such as doing AI safety and security R&D, dangerous capability evaluations, red-teaming safety protocols, or monitoring other powerful models. Since we care about models performing well on these tasks, we are worried about sandbagging: that if our models are misaligned1, they will intentionally underperform. Sandbagging is crucially different from many other situations with misalignment risk, because it involves models purposefully doing poorly on a task, rather than purposefully doing well. When people talk about risks from overoptimizing reward functions (e.g. as described in What failure looks like), the concern is that the model gets better performance (according to some metric) than an aligned model would have. And when they talk about scheming, the concern is mostly that the model gets performance that is [...] --- Outline: (03:55) Sandbagging can cause a variety of problems (08:13) Training makes sandbagging significantly harder for the AIs (09:58) Training on high-quality data can remove sandbagging (12:51) If off-policy data is low-quality, on-policy data might help (15:12) Models might subvert training via exploration hacking (19:31) Off-policy data could mitigate exploration hacking (24:56) Quick takes on other countermeasures (26:49) Conclusion and prognosis --- First published: May 8th, 2025 Source: https://redwoodresearch.substack.com/p/misalignment-and-strategic-underperformance --- Narrated by TYPE III AUDIO.

    30 min

  6. MAY 6

    “Training-time schemers vs behavioral schemers” by Alex Mallen

    Subtitle: Clarifying ways in which faking alignment during training is neither necessary nor sufficient for the kind of scheming that AI control tries to defend against.. People use the word “schemer” in two main ways: “Scheming” (or similar concepts: “deceptive alignment”, “alignment faking”) is often defined as a property of reasoning at training-time1. For example, Carlsmith defines a schemer as a power-motivated instrumental training-gamer—an AI that, while being trained, games the training process to gain future power. I’ll call these training-time schemers. On the other hand, we ultimately care about the AI's behavior throughout the entire deployment, not its training-time reasoning, because, in order to present risk, the AI must at some point not act aligned. I’ll refer to AIs that perform well in training but eventually take long-term power-seeking misaligned actions as behavioral schemers. When people say that a model is a schemer and [...] --- Outline: (02:23) Training-time schemers that continue to act aligned for the whole deployment (02:42) Empirical evidence (05:19) Theoretical argument (08:01) Related ambiguity about situational awareness (09:12) Behavioral schemers that weren't training-time schemers (12:00) Discussion --- First published: May 6th, 2025 Source: https://redwoodresearch.substack.com/p/training-time-schemers-vs-behavioral --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    13 min

  7. MAY 3

    “What’s going on with AI progress and trends? (As of 5/2025)” by Ryan Greenblatt

    Subtitle: My views on what's driving AI progress and where it's headed.. AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this! This post will assume some familiarity with what is driving AI progress, specifically it will assume you understand the following concepts: pre-training, RL, scaling laws, effective compute. Training compute trends Epoch reports a trend of frontier training compute increasing by 4.5x per year. My best guess is that the future trend will be slower, maybe more like 3.5x per year (or possibly much lower) for a few reasons: [...] --- Outline: (00:54) Training compute trends (08:25) Algorithmic progress (15:04) Data --- First published: May 3rd, 2025 Source: https://redwoodresearch.substack.com/p/whats-going-on-with-ai-progress-and --- Narrated by TYPE III AUDIO.

    17 min

  8. APR 30

    “How can we solve diffuse threats like research sabotage with AI control?” by Vivek Hebbar

    Subtitle: Preventing research sabotage will require techniques very different from the original control paper.. Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the original control paper or Ctrl-Z, because research sabotage differs from those settings in two crucial ways: Firstly, the AI can’t cause a catastrophe by behaving badly on just one occasion. To derail the entire project of solving alignment, the AI probably has to take a large number of malign actions. This contrasts with some other threat models (like rogue internal deployments), where the AI might succeed by acting badly on just a few occasions. Secondly and relatedly, individual bad actions in research sabotage aren’t very incriminating. If a model [...] --- Outline: (03:12) Threat model and basic countermeasures (03:55) Deployer strategies: Training and incrimination (08:34) Sabotage strategies (08:56) Targeting sabotage at the most effective points (09:33) Withholding good content vs. producing bad content (11:52) Code vs. ideas (12:29) What terms should we use for the concentrated/diffuse spectrum? (14:58) Canary string (15:12) Acknowledgements --- First published: April 30th, 2025 Source: https://redwoodresearch.substack.com/p/how-can-we-solve-diffuse-threats --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    16 min
See All (39)

About

Narrations of Redwood Research blog posts. Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

Information

  • Creator
    Redwood Research
  • Years Active
    2K
  • Episodes
    39
  • Rating
    Clean
  • Copyright
    © 2025 All rights reserved
  • Show Website
    Redwood Research Blog

To listen to explicit episodes, sign in.

Stay up to date with this show

Sign in or sign up to follow shows, save episodes, and get the latest updates.
Select a country or region

Africa, Middle East, and India

Asia Pacific

Europe

Latin America and the Caribbean

The United States and Canada