The Information Bottleneck

Ravid Shwartz-Ziv & Allen Roush

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

  1. 18h ago

    What a VC Actually Looks For in AI Founders | Sarah Catanzaro

    In this episode, we talked with Sarah Catanzaro, a general partner at Amplify Partners, about how AI research looks from the investor's side. Sarah has backed Runway and Modal since their seed rounds, and she usually gets to know researchers long before they start a company. This one is a bit different from our usual episodes because we talk about business more than research. We started with what makes a research idea a good business. Sarah's view is that startups shouldn't compete with the frontier labs directly and should go after things the labs put lower on their list. Voice is her example. Early experiments suggested that adding audio data hurt reasoning, so the labs didn't prioritize it and companies like ElevenLabs got ahead. She thinks the new auto research startups are making a similar bet. We also talked about AI slop. Sarah invested in Taste Labs, which is trying to end slop first and leave the question of good taste for later, because people agree much more easily on what is bad. From there we got into benchmarks. She likes public benchmarks, but when we asked if a benchmark win impresses her in a pitch, she said "I don't care." What she looks at is whether people adopt the product and whether deals close. We spent a lot of time on founders. Sarah wants a founder to have a big vision and a concrete first problem to solve, and she stays away from what she calls "research" for a problem. She told us you can't evaluate someone from a resume, so she reads their papers and talks to them. On funding, Sarah thinks the growth in AI is real, but early-stage neolabs look bubbly to her. By her count, close to a dozen auto research companies have raised more than $100 million each, and none of them has revenue. She explains why founders have about two years to show progress and why she tells her companies to pay top dollar for people. She is also critical of tranche rounds announced at the second-tranche valuation. Near the end, we talk about spending on tokens vs people, whether fame helps a founder, investor FOMO, and the SF echo chamber. Topics When great research is also a great businessWhere startups can beat the frontier labsAI slop and benchmarking tastePublic benchmarks vs closing dealsWhat Sarah looks for in foundersIs AI a bubble? Neolabs and auto researchHow much to raise and how to spend itDoes fame help a founder, and investor FOMOTimestamp 0:00 Intro 0:44 Great research vs a great business 6:04 Where startups can beat the frontier labs 8:54 Regulation and "capabilities maxing" 12:10 AI slop and benchmarking taste 21:44 Do benchmarks impress investors? 25:46 How to get an investor's attention 30:29 What Sarah looks for in founders 37:23 Is AI a bubble? 40:55 How much to raise and how to spend it 46:29 Tokens vs people 50:57 Does fame matter? 57:08 FOMO, the SF echo chamber, enterprise vs consumer 1:03:32 Best advice for founders Music * "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    What a VC Actually Looks For in AI Founders | Sarah Catanzaro
  2. 4d ago

    Post-Training Is Where Models Are Made Now | Maxime Labonne

    In this episode, we talked with Maxime Labonne, Head of Post-Training at Liquid AI and author of the LLM Course on GitHub, about what happens to a model after pre-training and why that stage now matters so much. Maxime walks us through the short history of post-training, from supervised fine-tuning and preference optimization to reinforcement learning with verifiable rewards and now multi-domain on-policy distillation (MOPD), where separate teachers for code, math and other domains get merged back into one student. He explains why you still can’t skip SFT, why the order of the stages matters, and why RL works even though a long rollout gives you about one bit of signal. His intuition is that SFT and distillation raise pass@k, and RL then sharpens it into pass@1. We also get into training small models for agentic tasks directly inside the harness, which open-source RL stacks are good enough for serious work, looping transformers, and how much of the work big models do today could move to small ones running locally. We also talk about why models write worse than they used and safety issues. We talk about regulation, open source and Chinese models,  whether a small startup can beat the big labs in a niche, and why he thinks RL environment companies are overrated. We finish with France vs San Francisco, Mistral and the regulation moat, and whether Europe can produce frontier models. Topics Why post-training matters more every yearThe modern pipeline: SFT, RL and distillationTraining agents and small modelsWhy models write worse than they used toSafety, incentives and regulationOpen source and Chinese modelsStartups vs the big labsEurope, Mistral and sovereign AIChapters 0:00 Intro 0:57 Why post-training matters 3:44 A short history of post-training 7:20 The modern pipeline: SFT, RL and distillation 19:41 Agents, architectures and small models 27:15 Why models write worse now 31:44 Safety, incentives and regulation 38:26 Open source and Chinese models 41:39 Recursive self-improvement, startups vs big labs 46:10 World models vs LLMs 49:16 What's overrated and what's next 55:36 Europe, Mistral and sovereign AI 1:02:23 Wrap-up Music * "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    Post-Training Is Where Models Are Made Now | Maxime Labonne
  3. Nov 13

    Physical AI for Science: Why It’s Harder Than Coding | Rishabh Agarwal (Periodic Labs)

    Rishabh Agarwal has worked on reinforcement learning for years, at Mila, Google DeepMind and Meta. He is now a founding member at Periodic Labs, a company building AI scientists together with the physical labs where they run experiments, and an adjunct professor at McGill. By his own count he understands about ten percent of the science they do there. In this episode he tells us why he made that move. His argument is that science is a different problem from coding. A coding agent never has to leave the computer, so progress is mostly about compute and good environments. To make a new material, someone has to mix powders, heat them and measure what came out, and you can't simulate that well enough to skip it. Rishabh walks us through the loop his lab runs and shows where the machines are the limit and where intelligence is. He brings up Bernd Matthias, who found more than a hundred superconductors over a career of, by Rishabh's estimate, maybe 20,000 experiments. What would a scientist like that do with a million? When he was deciding whether to join, people told him to wait: build superintelligence first and let it do the science. He didn't buy it. "Doing AI for science needs doing AI for science." With 1,300 older GPUs and data from their own lab, his team beat a frontier model on a task they care about. We also talk about Anthropic's recent biology announcement and the role of PR in industry science, who oversees experiments when an AI proposes them, and whether skills learned on games like NetHack carry over to real science. Near the end Rishabh explains why, after all these years, he thinks RL is "super dumb as a paradigm", and why that inefficiency helps the big labs. His closing message is that more people in AI should work on harder problems. Topics * Physical AI vs coding agents * Why you can't simulate your way to a new material * "Geniuses in a data center" vs geniuses with a lab * Anthropic's biology announcement and PR in industry science * Oversight for AI-run experiments * The materials science loop: hypothesis, synthesis, measurement, analysis * Bernd Matthias and the case for a million experiments * How a model learns from lab data * NetHack, games and what generalizes * Predicting experiment outcomes as an RL task * Scientific taste and what humans still do better * Why RL is inefficient, and why that helps the incumbents * Science is falsifiable, not verifiable Timeline 0:00 Intro 0:47 How physical AI differs from coding agents 6:27 Anthropic's biology announcement and PR in industry science 10:12 Why science starts in hard mode 13:22 Oversight when AI runs a lab 18:11 The experimental loop in a materials lab 25:23 What a scientist could do with a million experiments 27:47 Why scaling up experiments is not enough 31:29 How a model learns from lab data 34:45 NetHack and what transfers from games 42:04 Predicting an experiment before running it 44:00 Humans vs models and scientific taste 49:20 Why RL is a "dumb paradigm" that works anyway 55:20 What to do with 200k GPUs 57:36 Science is falsifiable, not verifiable 59:21 Closing: work on harder things Music * "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    Physical AI for Science: Why It’s Harder Than Coding | Rishabh Agarwal (Periodic Labs)
  4. Nov 7

    Dan Card: AI Hasn’t Changed Cybersecurity As Much As You Think

    Dan Card has spent more than 26 years in cybersecurity as a penetration tester, defender and architect. He helped organizations respond to some of the biggest incidents of the last two decades, from MS Blaster to WannaCry and Log4j. In this episode we ask him what has actually changed now that attackers and defenders both have strong AI models. Dan runs a lab where Claude attacks and ChatGPT defends, and he explains why a swarm of ten thousand agents hitting a defended server mostly gets you blocked faster. AI does help attackers with reconnaissance and research, and it helps defenders work through hundreds of alerts a day. Still, most breaches come from bad passwords and misconfigurations, not from exotic zero days. We also talk about air gaps versus sandboxes, why finding more vulnerabilities doesn’t always mean more risk, prompt injection, robots and autonomous weapons, and the story of how Dan tracked down someone who attacked his website. His main message is that security in the real world should be boring, like a good flight. Topics Social engineering, fake job candidates and mail bombing attacksWho the attackers really are, from teenagers to nation statesWhether an AI could escape and spread across the internetAir gaps, sandboxes and the Stuxnet storyHow AI attackers compare to human onesWhy agent swarms are easy to blockVulnerabilities that exist versus vulnerabilities that can be exploitedUsing LLMs to triage security alertsPrompt injection and how much we can trust modelsRobots, drones and physical riskWhy you defend against the brick before the aliensChapters 00:00 Intro and Dan’s background 01:13 What cybersecurity is and who the attackers are 09:36 Can an AI escape and spread? 19:31 Air gaps, sandboxes and Stuxnet 25:04 How AI is changing attack and defense 34:04 Why agent swarms don’t scare defenders 42:05 AI for security alerts and trusting models 54:57 Crime, deterrence and spending on security 1:02:38 When someone attacked Dan’s website 1:07:29 Robots and physical risk 1:16:28 Start with the brick, not the aliens 1:20:08 Wrap up Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

    Dan Card: AI Hasn’t Changed Cybersecurity As Much As You Think
  5. Oct 31

    Why Video Games Still Don’t Use Modern AI | Julian Togelius (NYU)

    Julian Togelius, an NYU professor and co-founder of modl.ai, is living the dream of every kid who grew up as a computer geek and gamer. He’s been doing AI research on video games since 2004, back when people told him it wasn’t real research. So why do most video games still run on behavior trees and dialogue trees when we have LLMs? Julian’s answer is that games are designed not to need AI. Most genres were locked in decades ago, and nobody has figured out how to design a good game around a model that can say anything. We talk about the state of the game industry and why so many developers there hate AI. Julian also explains why learning a brand new game as fast as a person would is a better test of intelligence than most benchmarks, and why he thinks we built the wrong kind of AI by training it to copy us. Near the end he gets into making games with Claude and where video world models like Genie might fit. Timeline 0:14 Why games matter for AI 5:21 What counts as AI? 10:02 Why games barely use modern ML 17:36 Why LLM characters break games 20:59 The state of the game industry 27:09 The AI backlash in games 30:41 AI in casual games 36:07 AI-made mods and fake players 40:42 Games as a test of intelligence 47:33 Can GPT-6 beat any game? 51:10 Playing alongside AI agents 53:36 We invented the wrong AI 58:33 Personal assistants and Jeb 1:03:22 Making games with AI 1:07:50 Video world models 1:11:16 Joining Julian’s lab 1:13:48 Favorite games Topics Why games are a good testbed for AI Classic game AI vs machine learning Why shipped games rarely use neural networks LLM NPCs and game design The game industry slump and the AI backlash AI in mobile games like Candy Crush AI-generated mods and simulated players Games as a test of general intelligence Automated playtesting and reward hacking Designing games around AI agents “We invented the wrong AI” Generating games with code, asset models and world models Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

    Why Video Games Still Don’t Use Modern AI | Julian Togelius (NYU)
  6. Oct 20

    What’s Left for Mathematicians? | Daniel Litt

    In this episode, we talked with Daniel Litt, a mathematician at the University of Toronto. In March 2025, Daniel bet that AI would not be able to autonomously produce an Annals of Mathematics-level paper by 2030. He has since conceded. He told us what changed and why. The models became superhuman at a small set of skills, like knowing the entire literature and grinding through technical arguments, and that turned out to be enough to do real math. They are still weak at the fuzzy part of the job, like figuring out what is actually true about an object or which questions are worth asking. We also got into how anyone can check a proof that no human has read yet, and what a Lean formalization does and doesn't guarantee. Daniel thinks prestige and credit in math are dead, and that we will have to judge people through thesis defenses and seminars rather than through PDFs. He explains why he thinks math got Goodharted and why the labs put so much effort into it in the first place. We finished with listener questions, including how to teach kids math now and whether P vs NP might be just one idea away. Topics - Why Daniel conceded his AI math bet - What models are superhuman at in math, and where they fall short - Why AI won't "solve math" - Checking AI-written proofs and the limits of Lean - Why different models keep proving the same results - How Daniel uses AI in his own research - Evaluating mathematicians when prestige and credit are dead - Why math got "Goodharted" - AI labs, race dynamics and publication norms - Teaching kids math, and P vs NP 00:00 Intro 00:15 The bet Daniel conceded 03:23 What AI is good and bad at in math 08:13 Will AI solve all of math? 11:37 Why we still need human mathematicians 13:59 Math is about understanding, not only proof 21:26 Axioms, foundations and Lean 27:42 How do you check an AI-written proof? 31:22 When models cheat 36:18 Are all the models proving the same things? 39:51 How Daniel uses AI in his own research 45:20 "Prestige and credit are dead" 49:58 AI risk and social upheaval 51:38 Is math special, or did it get Goodharted? 57:45 Why the labs care about math 59:54 PR, race dynamics and the pause 1:06:49 Would Daniel join a lab? 1:07:53 Listener questions: can understanding be formalized? 1:12:05 Teaching kids math with AI around 1:15:38 P vs NP and the Riemann hypothesis 1:17:37 Wrap-up Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

    What’s Left for Mathematicians? | Daniel Litt
  7. Oct 16

    The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic)

    In this episode of the the information bottleneck podcast, we talked with Andy Hall, a Stanford professor and MTS at Anthropic. The conversation explores the political economy of AGI, analyzing how superintelligence could reshape job markets, wealth redistribution, and public trust in government. Hall presents an optimistic view that hyperintelligent agents could enhance democracy by helping citizens make better political decisions, likening AI's potential impact to the invention of the printing press. The discussion also covers the challenges of regulating open-source models, the origins and mitigation of ideological biases in AI, and the geopolitical implications of the AI race between the US and adversaries like China. List of TopicsThe definition of the political economy of superintelligence and its potential to improve civic decision-making.The future of work, the likelihood of white-collar job loss, and the role of the existing tax system in wealth redistribution.Public skepticism toward AI, the decline of trust in the US government, and the contrast between AI power users and the general public.The tension between personal freedom and collective safety regarding open-source and open-weight AI models.Using AI tools to assist legislators in drafting policy and auditing frontier AI models.The geopolitical AI race, the threat of state-run hacking groups, and the challenges for middle-power countries.Center-left ideological biases in current AI models, their roots in training data and reinforcement learning, and experiments to fine-tune models with alternative viewpoints.The concept of AI constitutions, market competition over model values, and the risks of government intervention.AI-driven astroturfing on social media, ungovernable AI swarms, and the debate over regulatory "kill switches".Timeline00:14 - Introduction of Andy Hall and an overview of the political economy of superintelligence.05:38 - Discussion on the future of work, potential mass unemployment, and the political backlash of job loss during periods of economic growth.11:41 - Analysis of public hostility toward AI, drawing analogies to the regulation of social media.19:26 - The role of existing tax systems in wealth redistribution and the political dynamics of state versus federal taxation.27:31 - The debate over open-source models, balancing the desire for personal AI agents with national cyber risk concerns.33:10 - How AI can aid government bureaucracy, specifically in writing complex legislation and overseeing tech labs.46:25 - The geopolitical implications of AI, the race against China, and the threat of state-run hacking organizations like the Lazarus Group.51:38 - Examination of political biases in AI models, including Hall's lab experiments in fine-tuning models to hold conservative views.1:02:38 - Discussion on model constitutions, the distinction between a "soul document" and a true political constitution, and the regulation of AI values.1:13:19 - The rise of AI astroturfing in politics, botnets, and whether "kill switch" legislation is a practical solution.1:19:40 - Theoretical discussions on the future of automated law enforcement, robot armies, and AI's impact on class mobility.Music - "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic)
  8. Oct 10

    John Platt (Google): AI for Climate and Science

    John Platt is a Google Fellow and leads the Applied Science team at Google Research, the group working on climate and science. He's also the person behind SMO and Platt scaling. In this episode, we talk with him about how Google decides which scientific problems are worth its time, and what happens when AI meets real physical systems. A big part of the conversation is contrails. A small share of flights creates most of the warming from contrails, which together add up to roughly 1% of human-caused warming. John explains how his team detects contrails from weather satellites and predicts where they will form. In trials with American Airlines, rerouting flights cut contrail formation by 62% with almost no extra fuel. We also get into ERA, Google's empirical research assistant, and where AI agents actually help scientists. John sees them taking over the drudgery of building models and munging data, while humans keep the taste and the rigor. He shares why he's skeptical of gradient-free optimization, why the bitter lesson runs into energy and data center limits, and why physical AI that makes manufacturing cheaper could matter a lot for climate. We finish with FireSat, a satellite constellation (about 50 satellites at full scale) designed to catch wildfires while they're still small enough to put out. Timeline 00:00 Intro 01:09 How Google picks science problems: comparative advantage 04:15 Projects that didn't work: RNA aptamers and tool/problem fit 05:46 DNA, RNA and XNA 07:41 Contrails and why they warm the planet 13:01 Three AI systems for avoiding contrails 15:23 Cloud seeding and why cloud physics is hard 18:17 Airline trials: 62% fewer contrails 19:46 Navier-Stokes, wind tunnels and where simulation still fails 25:07 Will agents replace scientists? ERA and scorable tasks 31:36 Overfitting, Kaggle and the half-pixel bug 33:41 Agents that exploit bugs: Lean and Goodhart's law 36:45 Gradient-free optimization 38:53 SMO, SVMs and hinge loss in neural networks 42:12 The bitter lesson and the limits of scaling 43:36 Where agents will help science in the next 12 months 45:13 Robotics and making climate tech cheaper to build 48:22 AI risk and misuse 50:51 Climate outlook: past 2°C 54:39 Why AI won't just solve climate change 57:04 Room-temperature superconductors, fusion and HVDC 1:02:25 FireSat: detecting wildfires from orbit 1:05:16 Why existing satellites aren't enough 1:07:53 The economics of 50 satellites 1:09:34 False positives 1:11:16 Closing Topics Comparative advantage as the test for which science problems Google takes onContrails: the physics, the AI pipeline, and the American Airlines trialsERA and AI agents for scientific softwareWhat scientists still do that agents can’t: taste, creativity, rigorOverfitting, Goodhart’s law, and agents exploiting bugsGradient-free methods, SMO, and hinge lossThe bitter lesson and the physical limits of scalingClimate outlook, fusion, and physical AIRoom-temperature superconductors and long-distance power transmissionFireSat and early wildfire detectionMusic - "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

    John Platt (Google): AI for Climate and Science

Ratings & Reviews

5
out of 5
6 Ratings

About

Two AI Researchers - Ravid Shwartz Ziv, and Allen Roush, discuss the latest trends, news, and research within Generative AI, LLMs, GPUs, and Cloud Systems.

You Might Also Like