The Information Bottleneck

Ravid Shwartz-Ziv & Allen Roush

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

  1. 2d ago

    Surya Ganguli: The Physics of Intelligence

    Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks. We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics. The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire. We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm. key topics Connections between physics, neuroscience, and AIEmergent properties in complex systemsScaling laws in language modelsData efficiency and pruning in AINeuroscience insights into consciousness and selfThe future of AI and brain modelingChapters 00:00 Introduction to Surya Ganguli 00:57 Surya's Background: From String Theory to Neuroscience 02:22 Emergent Properties in Physics, Neuroscience, and AI 03:16 Energy Landscapes and Loss Landscapes in High Dimensions 04:07 Why Local Minima Don't Exist in High-Dimensional AI 05:22 Gradient-Based vs. Gradient-Free Learning Methods 08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges 13:48 Scaling Laws and Data Efficiency in Language Models 18:10 Properties of Data that Affect Scaling Laws 22:04 Constructing Non-Redundant Data Sets for Better Learning 24:32 Theory vs. Empirical Results in AI Research 32:19 Fundamental Components of Deep Learning: Are They Changing? 34:31 Future Paradigms in AI Beyond Current Models 37:22 Teaching AI and Humans: Paradigm Shifts in Learning 41:37 Consciousness, Self, and the Brain: Surya's Perspectives 49:49 Neuroscience and AI: Understanding the Brain and Consciousness 01:02:03 Understanding the Brain: Challenges and Opportunities 01:09:21 Brain-Computer Interfaces and AI in Neuroscience Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Surya Ganguli: The Physics of Intelligence

  2. 6d ago

    Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

    Brendan O'Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped. Key topics - Discrete diffusion for text vs autoregressive generation - Why diffusion samples are more diverse, and what that unlocks for RL - Where diffusion already wins: latency, on-device, robotics - Why serving cost, not quality, is the real blocker - RL as the most underhyped area in AI Timeline 00:00 Introduction 00:50 What diffusion models are and how text diffusion works 04:40 Why Brendan bet on text diffusion in 2023 07:15 Diversity, creativity, and why it helps RL 11:00 The best diffusion LLM today and the gap to frontier models 14:25 Latency, serving cost, and why it needs more chips 17:14 Where diffusion already wins: on-device, robotics, battery 20:14 One model, two modes: diffusion for thinking, AR for answering 22:24 Samplers and the stuttering problem 26:27 Theory, BERT, and why now is a good time to work on this 31:48 Pipelines built for autoregression, and continuous diffusion 35:35 Hardware: flops vs bandwidth 39:49 AGI timelines and real-world bottlenecks 50:15 Is AI engineering or science? 54:14 Most overhyped and most underhyped ideas 58:35 RL on diffusion, value functions, and exploration 1:07:30 Go download the model and break it Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

  3. Aug 8

    Nathan Lambert: Inside Post-Training and the Open Model Fight

    Nathan Lambert spent three years as post-training lead at Ai2, where he built the OLMo models, and he writes Interconnects, one of the most-read technical newsletters in AI. He left Ai2 in June and is now working on a new project. He's also the author of the RLHF book. We talked a lot about open models, their capabilities, and why they are better than he expected. We get into what that means over the next two to five years, why he thinks recursive self-improvement is overblown, what the market for training environments actually looks like now, and why he expects Anthropic's famously open internal culture to break after its IPO. Key Topics Open vs closed models and who actually captures the valueAnthropic and OpenAI as opposite cultures, and the talent concentration problemBoom vs bubble, and why token spend hasn't produced 10x better productsContinual learning, RSI skepticism, and what Nathan wants to work on nextWhat the open ecosystem needs economically to survive Timeline 00:00 Intro 00:27 Open vs closed models, and who actually captures the value 05:12 China, harnesses, and where the real training leverage sits 08:40 Sovereign compute and the national security case for building models 11:18 Uncensored open weights and the bioweapon question 14:29 Anthropic vs OpenAI, ideology and politics 19:35 The Mythos ban and the Fable 5 delays 24:30 The AGI narrative, the talent drain, and antitrust 28:12 Why researchers join Anthropic, and the open Slack culture 34:04 Nathan's next 12 months: character training and big RL runs 37:55 Continual learning, RSI, and why Nathan is skeptical 43:19 Boom or bubble, tokens vs GPUs 45:12 Why all that token spend never produced 10x products 48:38 Job displacement and the small-business future 52:49 Robotics, world models, and why multimodal lags 57:44 What the open ecosystem should actually do 1:03:17 Why NVIDIA isn't building a frontier model 1:07:34 The RLHF book, and whether RLHF still matters 1:11:06 GRPO vs PPO and on-policy distillation Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Nathan Lambert: Inside Post-Training and the Open Model Fight

  4. Aug 3

    Daphne Koller - The Future of AI in Biology and Drug Discovery

    Daphne Koller wrote the book that many of us learned probabilistic graphical models from, founded Coursera, and now runs insitro, which is trying to make drug discovery a machine-learning problem. We start with the bitter lesson. She agrees with most of it and then says where it stops working: biology doesn't have enough data, structure is how people understand anything, and making a drug is a question about an intervention that hasn't happened yet, not a pattern in data you already have. Most of the episode is about why drug discovery is hard. Ninety percent of drugs that reach the clinic fail, and mostly not because the molecule was bad. The molecule usually does what it was designed to do. It just turns out the thing it was designed to do had nothing to do with the disease. Only 22% of diseases have any approved drug at all, and she calls that an upper bound on what we understand, not a lower bound. She also gets into what agents are and aren't good for in a wet lab, why cells don't grow faster no matter how many GPUs you point at them, what it would take to have real foundation models for biology, and why almost all of biology is still out of distribution. Plus GLP-1s and what human data keeps teaching us, whether AI can make the kind of leap that turned a bacterial immune system into CRISPR, and what she'd build if she were starting Coursera today. Key Topics The impact of scaling and data in machine learningThe importance of structure and causality in AIChallenges in drug discovery and biological understandingThe role of foundation models in biologyEthical considerations in AI and biomedical research Chapters 00:00 Introduction to Machine Learning and Drug Discovery 02:00 The Bitter Lesson and Its Implications 06:48 Challenges in Drug Design and Discovery 11:48 Ethical Considerations in Human Research 17:20 The Drug Discovery Pipeline Explained 29:30 Integrating AI in Experimental Design 35:38 The Role of Human Judgment in Drug Design 37:14 Future of Drug Design: Efficiency vs. Automation 39:37 Challenges in AI and Data Availability for Biology 41:08 Foundation Models: Potential and Limitations 43:39 Causality in Biological Data: Importance and Challenges 45:18 Creativity vs. Understanding in Drug Design 48:17 Balancing Investments in Data, Algorithms, and Experiments 50:07 The Value of Simulations in Drug Discovery 52:03 Mathematical Frameworks in Biology: Utility and Limitations 54:14 The Future of Drug Discovery: Optimism and Innovations 56:28 The Impact of Coursera on Education 01:00:33 The Role of Universities in Lifelong Learning 01:04:06 Connecting Dots: The Fun of Variety in Work 01:05:46 Optimism for the Future of Drug Discovery Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Daphne Koller - The Future of AI in Biology and Drug Discovery

  5. Jul 30

    RL Was Broken at Every Level - With Joseph Suarez (PufferAI)

    In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers. Key topics Types of RL and their applicationsChallenges in scaling reinforcement learningThe role of simulators and hardware in RLRL in gaming: from chess to complex games like NetHack and RuneScapeFuture directions: scientific simulation and biological modeling Chapters 00:00 - Introduction to RL and Puff AI 01:50 - Different settings for RL: Games, Robots, Finance 04:10 - RL in LM and other domains 07:00 - Challenges and solutions in RL scaling 09:55 - Building fast, efficient simulators 15:10 - RL for scientific research and simulation 19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress 29:55 - Future of RL: Scientific discovery and beyond Resources Puff AI - Official Site - https://puffer.ai NetHack - https://www.nethack.org/ RuneScape - https://www.runescape.com/ Dwarf Fortress - http://www.bay12games.com/dwarves/ OpenAI Gym - https://github.com/openai/gym Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    RL Was Broken at Every Level - With Joseph Suarez (PufferAI)

  6. Jul 27

    The Model Found a Way Out - with Florian Brand (Prime Intellect)

    Florian Brand builds evals at Prime Intellect. The premise of the conversation is that writing a benchmark is the easy part now. Keeping the model from cheating it is the job, and it takes longer than the benchmark itself. We get into why he thinks you can't evaluate a model apart from the CLI it runs in, what happens to statistics when a single run costs five figures, and whether the feeling that a model just works can ever become a number. He also has a few stories about agents finding their way around the scoring that are worth hearing cold. Timeline 00:13 Intro01:00 What evals are for04:05 Agentic benchmarks07:10 Kimi K2 and model diversity08:23 Long-horizon coding tasks10:29 Building a benchmark12:15 MirrorCode14:27 Rubrics and LLM judges16:30 The cost of expert labelers17:49 Long runs and variance19:44 Evaluating the harness24:29 Chinese labs building CLIs30:00 More reward hacking37:45 Tau-bench and economic tasks39:43 Benchmaxxing and GLM 5.245:15 Statistics and cost47:56 Frontier convergence52:04 Misuse in open and closed models55:35 Self-improvement Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

    The Model Found a Way Out - with Florian Brand (Prime Intellect)

  7. Jul 23

    Pierre-Carl Langlais on Building Models from Data You Can Account For

    Most labs build language models by scraping the web and filtering afterward. Pierre-Carl Langlais runs it the other way around. At Pleias, the French-German lab he co-founded, the models are built from data he can actually account for, which in practice means open and public-domain sources plus a lot of synthetic data the lab generates itself. It sounds like a self-imposed handicap. It mostly isn't. One of their models is a 600 million parameter system that runs live inside the Paris subway's monitoring pipeline. We cover the SYNTH pretraining dataset and why he thinks "ethical data" has to mean more than copyright-free. He explains why barely 2% of their Common Corpus appears in typical web crawls, and why that gap is really a preservation problem. From there, he gets blunt about benchmark maxing and whether GLM really earns its Opus-class reputation. He also argues that the quiet move by closed labs to hide reasoning traces is mostly about claiming ownership of model outputs. He's skeptical of sovereign AI, and not shy about how Mistral drifted from frontier research toward French corporate consulting. We finish on NVIDIA's persona datasets and the odd idea of training on the conditions that produced a text rather than the text itself. Timeline(00:02) Welcome and introductions(00:49) Why synthetic data matters, and the SYNTH set(04:15) Three reasons to control your training data(07:18) What "ethical data" actually means(11:08) How Common Corpus got built, from Wikipedia to PDFs(16:35) Agentic harnesses and synthetic data(20:03) Evaluating data when you train on reasoning traces(25:27) General versus specialized pretraining(27:08) Benchmark maxing and the GLM question(31:51) Getting diversity in, and the NVIDIA personas(35:02) Hidden reasoning traces and the fight over model IP(38:17) Mid-training and the "It's All Training" thesis(41:47) Can small models actually compete(45:01) Cybersecurity and Europe's strategic gap(47:08) Do you need a big model to orchestrate the small ones(52:08) Sovereign AI and the limits of national champions(56:42) Scaling laws when you control the data(01:00:41) The NVIDIA persona datasets(01:04:52) What you actually do with synthetic personas(01:08:22) Closing thoughtsMusic "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

    Pierre-Carl Langlais on Building Models from Data You Can Account For

  8. Jul 20

    Dhruv Batra: The Browser Is a Robotics Problem - From Embodied AI at Meta to Web Agents at Yutori

    Dhruv Batra spent years leading Embodied AI at Meta,  training virtual robots to navigate photorealistic 3D scans of real buildings with pure reinforcement learning. Then he left to co-found Yutori and build agents for a very different environment: the web browser. In this episode, Dhruv explains why he sees these as the same problem. Web agents, in his framing, are robots that act in a browser (pixels in, actions out), and the web turns out to be just as messy an environment as the physical world. Along the way, we cover his definition of intelligence as "navigation in idea space," why robotics is lagging LLMs, the sim-to-real gap and why you can't fake friction coefficients, the teleoperation counterexample to the "it's a sensor problem" argument, and his provocative claim that under the current paradigm, we solved machine learning and didn't even realize it. He also makes the case for why the scaling hypothesis isn't falsifiable, why JEPA-style arguments deserve to be grappled with, how Yutori trains its Navigator models with RL on live websites, and what happens to the ad-supported web when agents, not eyeballs, do the browsing. Timeline00:01 — Intro 00:54 — What embodied AI actually means 06:47 — Intelligence as navigation in idea space 13:26 — Habitat: training robots with pure RL, no maps 20:04 — Why robotics is behind LLMs 28:24 — Sim-to-real: what you can and can't fake 33:34 — "We solved ML and nobody noticed" 37:12 — Leaving Meta, founding Yutori 43:21 — Web agents: screenshots in, actions out 48:15 — Why the web won't rebuild itself for agents 53:32 — Training Navigator: RL on live websites 1:01:04 — Who pays for the web when agents browse? 1:09:17 — What Yutori means, closing thoughts Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.About The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

    Dhruv Batra: The Browser Is a Robotics Problem - From Embodied AI at Meta to Web Agents at Yutori
See All (61)

Ratings & Reviews

5
out of 5
6 Ratings

About

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

Information

  • Creator
    Ravid Shwartz-Ziv & Allen Roush
  • Years Active
    2025 - 2026
  • Episodes
    61
  • Rating
    Explicit
  • Copyright
    © 2025 Ravid Shwartz-Ziv & Allen Roush
  • Show Website
    The Information Bottleneck

You Might Also Like