Benchtalks

Snorkel AI

Benchtalks is Snorkel AI's podcast series at the intersection of AI evaluation, data quality, and real-world impact. Hosted by the Snorkel team, each episode brings together researchers, practitioners, and leaders to dig into the questions that matter most as AI benchmarks grow more sophisticated, dynamic, and reflective of the complexity found in real-world deployments. We explore the full stack of what it takes to build AI that actually works — from the design of rigorous, open benchmarks that close the gap between what we measure and what we encounter in production, to the expert-in-the-loop data creation and curation pipelines that make reliable evaluation possible. Along the way, we get into reinforcement learning, reward modeling, and the evolving science of data quality that underpins it all. Whether you're building agents that operate over long horizons, crafting rubrics that go beyond pass/fail, or trying to understand what "good" looks like for a multi-artifact deliverable — this is the conversation for you. New episodes drop regularly on YouTube and wherever you get your podcasts. Follow us @SnorkelAI (LinkedIn, X, YouTube) to stay current as the field moves fast. 

Episodes

  1. 6d ago ·  Video

    Benchtalks #3: Parth Asawa (Continual Learning Bench) - We Taught AI Everything Except How to Learn

    For our third Benchtalks, the series dedicated to the researchers building the measurement toolkits that frontier labs hill-climb on, Snorkel AI co-founder Vincent Sunn Chen sat down with Parth Asawa, a UC Berkeley PhD student advised by Matei Zaharia and Joey Gonzalez, and the creator of Continual Learning Bench, the first standardized benchmark for measuring whether AI systems actually learn from experience over time. This interview covers: Why your coding agent re-reads your entire codebase from scratch every single session, and why a human developer stops doing that after task oneHow the field got so good at scaling that it quietly set aside the original question: can a model actually learn on its own after training ends?The poker task: if your opponent calls every hand, a human exploits it in minutes, and models run in separate sessions don't figure it outThe gain metric: why cumulative reward alone can't tell you if a system is learning or just smarter to begin with, and how stateful minus stateless performance isolates actual learningWhat Anthropic's Fable release revealed, including how they used Continual Learning Bench to show Fable outperformed systems using Opus or Sonnet as a backboneWhy in-context learning still leads the leaderboard, and why Parth's long-run bet is on parametric systemsThe false dichotomy in AI safety: unsafe open models vs. consolidated power, and what Parth and Joey Gonzalez propose as the third pathFull interview/transcript: snorkel.ai/blog/benchtalks-parth-asawa-continual-learning-bench/

    51 min

  2. Jun 3 ·  Video

    Benchtalks #2: John Yang (SWE-bench, ProgramBench) - The future of coding benchmarks

    For our second Benchtalks, the series dedicated to the researchers building the measurement toolkits that frontier labs hill-climb on, Snorkel AI co-founder Vincent Sunn Chen sat down with John Yang, a Stanford PhD student and creator of the SWE-bench franchise, SWE-smith, CodeClash, and most recently ProgramBench. This interview covers:  Why every frontier model scored 0% at launch — until GPT-5.5 cracked the first task (cmatrix)Why ProgramBench grades the runnable artifact, not the implementation — and lets models build in any languageThe post-training tell hiding in plain sight: how much models (especially GPT) love Python, even when it looks like a handicapThe reward-hacking problem — models with internet access cheated up to 36% of the time, and why nine LLM judges still couldn't agree on what counts as cheatingThe lineage from SWE-bench to SWE-smith to CodeClash, and what ProgramBench needs from expert contributors to growFull interview/transcript: https://snorkel.ai/blog/benchtalks-john-yang-programbench/

    54 min

  3. Mar 31 ·  Video

    Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) - Building the benchmark factory

    In this inaugural episode of Benchtalks, Snorkel AI co-founder Vincent Chen sits down with Alex Shaw, MTS at Laude Institute and co-creator of Terminal-Bench, to unpack what the rapid hill-climbing on TB2 reveals about the state of AI agent evaluation — and where the field needs to go. This interview covers:  Why TB2 went from 20–30% during development to 75–80% at the frontier todayThe bet on the terminal as the right abstraction for general computer useHow Harbor became a benchmark factory — and why that matters for RL post-trainingThe "benchmaxxing" problem and how the community is keeping TB2 honestWhat Terminal-Bench 3 needs from expert contributors to shape model development for the next yearFull interview/transcript: https://snorkel.ai/blog/benchtalks-alex-shaw-terminal-bench-harbor-building-the-benchmark-factory/

    22 min
  • 3 Episodes

About

Benchtalks is Snorkel AI's podcast series at the intersection of AI evaluation, data quality, and real-world impact. Hosted by the Snorkel team, each episode brings together researchers, practitioners, and leaders to dig into the questions that matter most as AI benchmarks grow more sophisticated, dynamic, and reflective of the complexity found in real-world deployments. We explore the full stack of what it takes to build AI that actually works — from the design of rigorous, open benchmarks that close the gap between what we measure and what we encounter in production, to the expert-in-the-loop data creation and curation pipelines that make reliable evaluation possible. Along the way, we get into reinforcement learning, reward modeling, and the evolving science of data quality that underpins it all. Whether you're building agents that operate over long horizons, crafting rubrics that go beyond pass/fail, or trying to understand what "good" looks like for a multi-artifact deliverable — this is the conversation for you. New episodes drop regularly on YouTube and wherever you get your podcasts. Follow us @SnorkelAI (LinkedIn, X, YouTube) to stay current as the field moves fast. 

Information

  • Creator
    Snorkel AI
  • Years Active
    2K
  • Episodes
    3
  • Rating
    Clean
  • Copyright
    © 2026 Benchtalks
  • Show Website
    Benchtalks

You Might Also Like