The Information Bottleneck

Ravid Shwartz-Ziv & Allen Roush

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

  1. 13 hr ago

    World Models | John Langford (Microsoft AI Labs)

    John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently. We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work. Topics: Implicit vs. explicit world models, and the case against JEPA-style objectivesCompact belief states: why compression beats a growing KV cacheDoes research still matter in the age of scale? The Kimi K3 argumentAgent-driven research: 2,000 pre-training experiments in 90 daysThe invention of CAPTCHAOptimizers from SGD and Vowpal Wabbit to Muon Chapters00:00 Why world models: the sample-complexity gap09:48 The case against JEPA; a transformer-style implicit world model15:52 Compact belief states: epicycles vs. heliocentrism23:41 Does research still matter? The Kimi K3 argument27:35 Open vs. closed models35:57 Recursive self-improvement and agent-driven research42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility54:54 The invention of CAPTCHA1:00:53 Optimizers: from Vowpal Wabbit to MuonMusic "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    World Models | John Langford (Microsoft AI Labs)

  2. 1 day ago

    Which Tabular Model Should You Actually Use? | David Holzmüller (INRIA)

    Description Tabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data. We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box. Links: TabArena: https://tabarena.ai Topics Tabular foundation models and in-context learning on tablesTabArena and Beyond Arena: building a benchmark that stays honestTabFM, TabPFN, TabICL and the tradeoffs between themWhen boosted trees and MLPs still win (large data, CPU, fast inference)Why LLMs are inefficient on tabular data and where they might helpEmbedding text columns with language modelsExplainability, calibration and class imbalanceTime series vs tabular dataOpen problems: invariances, synthetic data, uncertainty, scaling downWhere the field is heading in the next five years Chapters 0:00 Intro 0:31 What changed in tabular ML: TabPFN-style foundation models 2:22 Which model to try first? TabArena and how it was built 5:14 What older benchmarks got wrong, and Beyond Arena 8:45 GPU AutoML vs foundation models 10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs 12:47 Calibration, class imbalance and small vs large data 19:45 Explainability for black-box tabular models 21:34 Why LLMs are bad at tabular data 25:39 Claude Opus 4.6 vs TabICL on tiny datasets 27:51 New classifiers, five-year outlook, real vs synthetic pretraining 33:13 Embedding text columns for tabular foundation models 36:13 Time series vs tabular data 39:59 When gradient boosted trees still win, and feature engineering 45:31 Open research problems and where the field is heading 52:54 Better MLPs and why classical defaults are bad out of the box Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    Which Tabular Model Should You Actually Use? | David Holzmüller (INRIA)

  3. 3 days ago

    Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal)

    Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble. Key topics AI infrastructure challenges and when to train your own modelsGPU resource management and virtualizationInference optimization and speculative decodingThe economics and future of AI hardwareAgents, sandboxing, and open-model security Chapters 00:00 Intro 01:03 Why AI needs special-purpose compute 03:22 Buying vs renting GPUs: the three problems 07:15 Modal's approach, and doing more with less compute 09:46 Do we actually need to spend more? The conflict-of-interest question 13:08 Should companies train their own models? 14:47 Efficient fine-tuning and prompts as fast weights 17:37 Are we in a compute bubble? 20:21 Why inference will dominate compute (the SQLite analogy) 22:42 Speculative decoding 26:44 Why scaling inference is hard, and neuromorphic hardware 28:36 Why NVIDIA's monopoly persists 33:09 Inference chip startups and the hardware lottery 35:24 How Modal stays hardware-agnostic (GPU snapshot restore) 38:45 Will agentic coding erode CUDA's moat? 41:18 Running one agent vs thousands: sandboxing at scale 46:27 The OpenAI/Hugging Face hack and open models as defenders 52:28 Rogue AI, self-replication, and fast takeoff 56:09 What's next: evals, embodiment, edge inference 1:00:27 Modal is hiring (modal.jobs) Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal)

  4. 28 Aug

    Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust

    Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it's part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they've been tested for real. So we spend a good chunk of the episode on what actual containment would look like. Stella's argument is that the tools already exist, the labs just don't use them: run dangerous capability evals on air-gapped networks with no route to the public internet, put the most sensitive testing in SCIF-style secure facilities, and treat model evaluation the way the security world treats classified systems rather than the way startups treat staging environments. And yet Stella remains one of the world's most prominent open-source advocates. From her perspective, the biggest risk isn't the technology; it's unchecked corporate power, and the only durable check on it is an independent scientific research establishment that doesn't depend on the AI industry for its funding or its facts. From there the conversation spans the geopolitics of Chinese open models and whether governments can restrict them, sovereign AI and what it would actually take for other countries to train their own models, why harnesses and UX drive more of AI's perceived progress than raw intelligence, the AI-found counterexample to the Jacobian conjecture, and EleutherAI's "Deep Ignorance" approach to making open-weight models safe by filtering hazardous knowledge out of pretraining. key topics AI governance and regulationCybersecurity incidents involving AI modelsOpen source AI safety and securityThe role of independent research in AI safetyLegal and ethical considerations in AI developmentTimeline 00:13 — Intro: Stella Biderman and EleutherAI, a real non-profit in AI02:05 — News of the week: Kimi K3, and OpenAI's model autonomously hacking Hugging Face05:49 — "Frontier labs can't be trusted": repeated containment failures, air-gapped networks and SCIFs vs. sandboxes22:45 — Can governments ban open or Chinese models? Import restrictions and the six-month open/closed gap27:05 — Why Stella is still pro-open-source: unchecked corporate power as the real danger31:11 — The opioid epidemic analogy: avoiding both regulatory failure and overcorrection34:57 — Offense vs. defense: why open access to AI has empirically favored defenders37:28 — Chinese labs, the CCP, and why safety and fine-tuning are low-prestige work in China42:19 — Sovereign AI: does every country need its own foundation model?49:29 — Sampling, harnesses, and why ChatGPT was really a UX breakthrough54:09 — AI solves the Jacobian conjecture: domain data beats raw intelligence58:02 — Safety is contextual, not a model property — and what HAL 9000 got right1:01:42 — Is Stella optimistic about the future?1:02:50 — Deep Ignorance, the science of AI training dynamics, and how to get involved with EleutherAIMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust

  5. 24 Aug

    Why Deep Learning Finally Works on Tables | Frank Hutter (Prior Labs)

    In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars. We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks behind scaling from 10,000 to a million rows, where LLMs fit into data science (and where they embarrassingly don't), and what happens to XGBoost from here. Beyond the research, Frank talks about the jump from professor to co-CEO, why he refused to merge his 45-person team into SAP's 110,000 employees, the open-weights licensing debate, and the case for building a frontier lab in Freiburg rather than San Francisco. key topics The role of foundation models in tabular dataImpact of SAP acquisition on Pro LabsThe evolution of AutoML and hyperparameter optimizationChallenges and solutions for large context in modelsOpen source models and licensing strategiesThe importance of independence for startup agilityFuture directions in AI for science and medicine00:00 Intro 00:34 The SAP acquisition and staying independent 07:39 Why tabular data is the next big thing in deep learning 14:19 What makes tabular data hard 19:14 AutoML, AutoGluon, and fifteen years of hyperparameter tuning 28:27 Scaling TabPFN: context limits and architectures 34:35 Agentic data science and LLMs 39:30 Online learning, time series, and Bayesian inference in a forward pass 47:05 Open weights and the license debate 54:51 Will LLMs and tabular models merge? 1:00:01 From academia to startup 1:09:42 Why build in Europe 1:12:53 Audience questions and hiring Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Why Deep Learning Finally Works on Tables | Frank Hutter (Prior Labs)

  6. 17 Aug

    Surya Ganguli: The Physics of Intelligence

    Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks. We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics. The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire. We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm. key topics Connections between physics, neuroscience, and AIEmergent properties in complex systemsScaling laws in language modelsData efficiency and pruning in AINeuroscience insights into consciousness and selfThe future of AI and brain modelingChapters 00:00 Introduction to Surya Ganguli 00:57 Surya's Background: From String Theory to Neuroscience 02:22 Emergent Properties in Physics, Neuroscience, and AI 03:16 Energy Landscapes and Loss Landscapes in High Dimensions 04:07 Why Local Minima Don't Exist in High-Dimensional AI 05:22 Gradient-Based vs. Gradient-Free Learning Methods 08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges 13:48 Scaling Laws and Data Efficiency in Language Models 18:10 Properties of Data that Affect Scaling Laws 22:04 Constructing Non-Redundant Data Sets for Better Learning 24:32 Theory vs. Empirical Results in AI Research 32:19 Fundamental Components of Deep Learning: Are They Changing? 34:31 Future Paradigms in AI Beyond Current Models 37:22 Teaching AI and Humans: Paradigm Shifts in Learning 41:37 Consciousness, Self, and the Brain: Surya's Perspectives 49:49 Neuroscience and AI: Understanding the Brain and Consciousness 01:02:03 Understanding the Brain: Challenges and Opportunities 01:09:21 Brain-Computer Interfaces and AI in Neuroscience Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Surya Ganguli: The Physics of Intelligence

  7. 14 Aug

    Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

    Brendan O'Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped. Key topics - Discrete diffusion for text vs autoregressive generation - Why diffusion samples are more diverse, and what that unlocks for RL - Where diffusion already wins: latency, on-device, robotics - Why serving cost, not quality, is the real blocker - RL as the most underhyped area in AI Timeline 00:00 Introduction 00:50 What diffusion models are and how text diffusion works 04:40 Why Brendan bet on text diffusion in 2023 07:15 Diversity, creativity, and why it helps RL 11:00 The best diffusion LLM today and the gap to frontier models 14:25 Latency, serving cost, and why it needs more chips 17:14 Where diffusion already wins: on-device, robotics, battery 20:14 One model, two modes: diffusion for thinking, AR for answering 22:24 Samplers and the stuttering problem 26:27 Theory, BERT, and why now is a good time to work on this 31:48 Pipelines built for autoregression, and continuous diffusion 35:35 Hardware: flops vs bandwidth 39:49 AGI timelines and real-world bottlenecks 50:15 Is AI engineering or science? 54:14 Most overhyped and most underhyped ideas 58:35 RL on diffusion, value functions, and exploration 1:07:30 Go download the model and break it Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Text Diffusion Models with Brendan O'Donoghue (Google DeepMind)

  8. 8 Aug

    Nathan Lambert: Inside Post-Training and the Open Model Fight

    Nathan Lambert spent three years as post-training lead at Ai2, where he built the OLMo models, and he writes Interconnects, one of the most-read technical newsletters in AI. He left Ai2 in June and is now working on a new project. He's also the author of the RLHF book. We talked a lot about open models, their capabilities, and why they are better than he expected. We get into what that means over the next two to five years, why he thinks recursive self-improvement is overblown, what the market for training environments actually looks like now, and why he expects Anthropic's famously open internal culture to break after its IPO. Key Topics Open vs closed models and who actually captures the valueAnthropic and OpenAI as opposite cultures, and the talent concentration problemBoom vs bubble, and why token spend hasn't produced 10x better productsContinual learning, RSI skepticism, and what Nathan wants to work on nextWhat the open ecosystem needs economically to survive Timeline 00:00 Intro 00:27 Open vs closed models, and who actually captures the value 05:12 China, harnesses, and where the real training leverage sits 08:40 Sovereign compute and the national security case for building models 11:18 Uncensored open weights and the bioweapon question 14:29 Anthropic vs OpenAI, ideology and politics 19:35 The Mythos ban and the Fable 5 delays 24:30 The AGI narrative, the talent drain, and antitrust 28:12 Why researchers join Anthropic, and the open Slack culture 34:04 Nathan's next 12 months: character training and big RL runs 37:55 Continual learning, RSI, and why Nathan is skeptical 43:19 Boom or bubble, tokens vs GPUs 45:12 Why all that token spend never produced 10x products 48:38 Job displacement and the small-business future 52:49 Robotics, world models, and why multimodal lags 57:44 What the open ecosystem should actually do 1:03:17 Why NVIDIA isn't building a frontier model 1:07:34 The RLHF book, and whether RLHF still matters 1:11:06 GRPO vs PPO and on-policy distillation Music "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    Nathan Lambert: Inside Post-Training and the Open Model Fight
See All (66)

About

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

Information

  • Creator
    Ravid Shwartz-Ziv & Allen Roush
  • Years Active
    2025 - 2026
  • Episodes
    66
  • Rating
    Explicit
  • Copyright
    © 2025 Ravid Shwartz-Ziv & Allen Roush
  • Show Website
    The Information Bottleneck

You Might Also Like