Latent Space: The AI Engineer Podcast

Latent.Space
  • 5.0 (6)
  • Technology
  • Updated weekly

The podcast by and for AI Engineers! In 2025, over 10 million readers and listeners came to Latent Space to hear about news, papers and interviews in Software 3.0. We cover Foundation Models changing every domain in Code Generation, Multimodality, AI Agents, GPU Infra and more, directly from the founders, builders, and thinkers involved in pushing the cutting edge. Striving to give you both the definitive take on the Current Thing down to the first introduction to the tech you'll be using in the next 3 months! We break news and exclusive interviews from OpenAI, Anthropic, Gemini, Meta (Soumith Chintala), Sierra (Bret Taylor), tiny (George Hotz), Databricks/MosaicML (Jon Frankle), Modular (Chris Lattner), Answer.ai (Jeremy Howard), et al. Full show notes always on https://latent.space www.latent.space

  1. 5 hr ago

    Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

    We’ve been running a bit of an Agent Cloud series surveying all the top inference/compute/cloud providers, from Databricks to Daytona to Railway and, even further back, E2B, but we’re excited to conclude this series returning to Modal, which has just raised a monster $355M Series C. The cloud was built for developers. But agents are now changing that. The old infra stack was designed for a human who could read docs, reason through YAML, and understand dashboards to figure out what they need when something broke. While this was painful for developers, it worked since they could fill in missing context in their heads. However, agents don’t have that luxury. Now in this new era of agents, everything has to be tighter. They need a place to write code, run it, inspect the output, change the environment, debug failures, and try again. Fast iteration and feedback loops with all the necessary context are crucial for agents to operate properly. Furthermore, sandboxes are a clear representation of this shift as agents can easily spin up isolated environments. This programmatic infra even extends to research: Two years ago, we were one of the first to cover Modal with CEO Erik Bernhardsson and Alessio designed our favorite LS thumbnail of all time: At the time, Modal was just a teeny little company with a $17M Series A. Today, fresh off their $355M Series C, Modal is one of the clearest examples of the agent cloud future being built in real time: a cloud platform moving past traditional web app assumptions toward the workloads AI actually creates such as elastic inference, sandboxes, GPU burst, post-training, background agents, and infrastructure that agents themselves can operate. In this episode, Modal CTO Akshat Bubna joins swyx and Vibhu to unpack why AI applications don’t fit traditional cloud assumptions, why Kubernetes was never designed for bursty compute-heavy workloads, and why Modal is now shifting from developer experience to agent experience. We go deep on Modal’s AI infra stack: serverless functions, decorator-based infrastructure, elastic inference for custom models, GPU snapshotting, DeFlash, speculative decoding, Auto Endpoints, sandboxes, persistent storage, networked containers, private IPv6, RDMA, multi-node training, and Modal’s capacity pool across 17 cloud providers. Akshat also explains why RL rollouts can require 100,000 sandboxes, why production agents need hard guardrails, why observability may matter more than reading code, and why AI has made infrastructure exciting again. We discuss: * Why Kubernetes wasn’t built for bursty AI workloads * How Modal started as a better runtime before becoming an AI cloud * Why Modal added GPUs before ChatGPT * The shift from developer experience to agent experience * Why observability matters when agents are writing the code * Elastic inference for custom models across audio, video, robotics, and comp bio * GPU snapshotting, cold starts, and why inference workloads are so bursty * Why RL rollouts can require 100,000 sandboxes * DeFlash, speculative decoding, and frontier-level inference performance * Auto Endpoints and making optimized inference easier to deploy * What Modal adds beyond vLLM, SGLang, and raw GPU rental * Modal’s 17-cloud capacity pool and supercloud strategy * Networked sandboxes, sidecars, private IPv6, and RDMA * Serverless multi-node training for post-training and research workloads * Auto-research, model-guided sweeps, and agents launching GPU experiments * Compute strategy, capacity planning, and batch tiers * Why production agents need specialized sandboxes and hard guardrails * Modal’s take on managed agents, CI, Gitpod/Ona, Python, TypeScript, and Modal Bench Akshat Bubna * LinkedIn: https://www.linkedin.com/in/akshat-bubna-188885103 * X: https://x.com/akshat_b Modal * Website: https://modal.com Timestamps 00:00:00 Introduction 00:00:39 Modal’s origin and why Kubernetes wasn’t enough 00:04:32 Developer Experience → Agent Experience 00:06:21 Modal’s AI cloud primitives 00:09:14 Sandboxes, agent loops, and proto-Cognition 00:12:12 Elastic inference, GPU snapshotting, and 100,000 sandboxes 00:15:24 DeFlash, speculative decoding, and Auto Endpoints 00:19:59 Production-grade inference beyond raw GPUs 00:22:00 Background agents, Ramp Inspect, and the agent lifecycle 00:24:08 Modal’s 17-cloud supercloud strategy 00:26:40 Networked sandboxes, private IPv6, and RDMA 00:32:48 Multi-node training, post-training, and auto research 00:37:36 Compute strategy, capacity planning, and batch tiers 00:40:55 Open models, real-time AI, and production agent infra 00:43:06 Hard guardrails, managed agents, and specialized sandboxes 00:46:06 Why AI made infrastructure exciting again 00:48:30 Model APIs, differentiated products, and agentic video 00:51:50 CI, coding-agent infra, SDKs, and Modal Bench 00:57:28 Closing Thoughts Transcript Introduction: Modal, Series C, and the Art Party Swyx [00:00:00]: We’re here with Akshat, CTO of Modal, together with Vibhu. Congrats on your Series C. Akshat [00:00:10]: Thank you. Swyx [00:00:11]: Your party yesterday was amazing. Akshat [00:00:15]: Yeah. Swyx [00:00:15]: From all the photos and all the swag. Akshat [00:00:17]: We had a bunch of art installations, which was fun, seeing, like, our products on pedestals next to, like, Rodin. Swyx [00:00:25]: Very nice. Very nice. When you started, it was not the GPU inference company. Maybe it was in your mind. Take us back to the origin story. Modal’s Origin: A New Runtime Beyond Kubernetes Akshat [00:00:39]: I first met Eric, who’s the CEO, through an investor. Back then Eric was already thinking about building, a new runtime, and he got there thinking through why are workflow orchestration products so hard to use. It’s because you have to run them on Kubernetes. Kubernetes is hard to manage. It’s not built for burstiness and, custom images, Swyx [00:01:03]: Yeah Akshat [00:01:03]: It has a terrible developer experience. Swyx [00:01:05]: And I’ll, I’ll interject Akshat [00:01:06]: Yeah Swyx [00:01:07]: For listeners, who are new, we interviewed Eric two years ago, and there’s a bit more of the story there from Spotify and all those things. Swyx [00:01:14]: And I came across Eric through Data Council because he did that talk on the serverless container stack that you guys did, which was like, that was my first like, “Okay, I need to take Modal very seriously” moment. Akshat [00:01:26]: Yeah. Swyx [00:01:26]: But it was still very unclear, like, do I need all this for just my data pipelines? Akshat [00:01:33]: Yeah. initially what we were thinking about was if we build a better runtime, it’s a very useful primitive in itself. It’s There’s a lot of things that, get solved by serverless functions, like you can do, ETL stuff, you can do job queues, you can do all this, like, bursty processing, which it turns out every company had needs for. but then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very verticalized. So perhaps data engineering would’ve been the first one, but we were thinking about inference. Back then it was more classical inference, like computer vision stuff and running XGBoosts and whatnot. But we added GPUs to the product a year before ChatGPT came out. From Serverless Containers to GPU Workloads Swyx [00:02:19]: Nice. Akshat [00:02:19]: We just didn’t think it would be that big of a deal. Swyx [00:02:22]: Yeah, just like add A100. Vibhu [00:02:23]: Was there any, like, early key problem that really sparked off why you built it? Akshat [00:02:28]: Yeah. Primarily it’s just, none of the tooling that was out there was built for, one, a really great developer experience, and also there’s a general trend of, a lot of the workloads that we were seeing were very. I wish there was a better word for it, but compute-heavy. Like, they need, one, like, need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for, like, slow scaling and, more for, like, web server use cases. And also there’s just a lot more specialization in, like, what kinds of environments these workloads run in. Like, we had sometimes they need accelerators, sometimes they need different kinds of images, and this is just like a consistent thing that we saw across a lot of companies. That would be the next step. Software-Defined Infrastructure and Decorator-Based DX Swyx [00:03:13]: Yeah. Yeah. Be nice. I don’t know how much this factored into the early story, but I wrote a post when I was at Temporal about infrastructure, software-defined infrastructure or something like that. Akshat [00:03:22]: Yeah, the self-provisioning Swyx [00:03:23]: Self-provisioning. Akshat [00:03:24]: Yeah. Swyx [00:03:24]: Yeah. I can’t even remember my own post. Swyx [00:03:26]: And then you put me on the landing page. Akshat [00:03:28]: Yeah. We really like, the term and so we stole it. Swyx [00:03:32]: Because you had the insight that everything can just be in decorators co-located with the code, right? Akshat [00:03:37]: Yeah. Swyx [00:03:37]: Was that a big part of the original Akshat [00:03:39]: Yes Swyx [00:03:39]: Story or it was just like a DX layer? Akshat [00:03:41]: That was, really important because we really didn’t want people to spend, so much time, writing YAML, and it seemed like you could really condense the surface area of what you’re doing, put it in code so you can operate on it just like you operate on other code, and like build stuff that’s more expressive and dynamic. and so yeah, that was always a very important part. Swyx [00:04:04]: Then the pushback is this is a DSL. Akshat [00:04:07]: Yeah. Swyx [00:04:07]: It’s you’re closed source. I am locked into Modal. Akshat [00:04:11]: Yeah. We never really got pushback for that because the nice thing about Modal is you can bring whatever code you have, and su

    58 min
  2. 🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI

    1 Jul

    🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI

    This episode has a fun personal twist: There’s a counterfactual world where I was employee #1 at Genesis Molecular AI, the company behind today’s episode. A certain introduction happened a few weeks too late and I had already happily signed at Atomwise, another ML-for-drug-discovery startup. Same problem, different company. I was certain ML was going to transform small molecule drug discovery. Early results were underwhelming. Useful at times, but nowhere near revolutionary. In the last year I’ve seen signs that ML is finally ready to deliver on my convictions from a decade ago. Genesis is one of the places that might have finally cracked this problem. I was super excited to come full circle and catch up with co-founder Evan Feinberg and CTO Sergey Edunov. If you are at all interested in small molecule drug discovery, we think you will find this fascinating! In our nearly two hour chat we cover: * What is small molecule drug discovery, and why is it hard * Structure prediction as a hotbed of innovation in AI algorithms * How advances in AI elsewhere have enabled stepwise improvements in predictive power * How the community benchmarks are essentially calling AI slop good enough * The Genesis flagship model (PEARL) can routinely hit a threshold that is necessary for real-world applications * New agentic workflows enabled by these highly accurate models Read on for more, and also some personal thoughts on the future at the end. The coolest diffusion research is happening at Genesis Sergey Edunov came to Genesis from Meta where he led Llama 2 training and Llama 3 pretraining. Sergey was a former physicist who thought he was done with physics after many years of training LLMs. Then, he discovered Genesis, and was blown away with all the novel architecture work they’ve been developing. It probably surprises no one that modern LLM research has not resulted in fundamentally novel or exciting updates in architectures since almost the advent of the transformer — the entire field is using variants on the same idea that came out in the original “Attention is all you need” paper. Sure, some were quite useful (mixture-of-experts in particular allowed for the massive model paradigm we’re at today), but there was very little conceptually exciting. “We sort of had to wait for the right primitive to get created, and that turned out to be diffusion… Actually, some of the most innovative diffusion research that’s happening in our field is happening in 3D structure prediction right now.” — Evan Feinberg The field of 3D structure prediction on the other hand has been a hotbed of research. Genesis’ recent model PEARL (Place Every Atom at the Right Location) is able to understand protein flexibility, and model not just where the ligand goes, but also make small adjustments of the protein so that the two fit better than either alone. The field knew this was missing for a long time, but it was really hard to model until now. Agentic Discovery What makes this problem so hard? As Sergey points out, there are 10^60 possible drug-like small molecules. You’ll never be able to search them all, and trying to find the good ones is something like finding a needle in a haystack — except everything except your needle is dangerous. “There are 10 to the 60 drug-like small molecules in the universe… it’s like finding a needle in a haystack, where everything except your needle is very, very dangerous.” — Sergey Edunov “Or finding hay in a needle stack might be a more apt analogy.” — Evan Feinberg Trying to solve the multi-parameter optimization problem is even worse. What makes a strong binder and a molecule with good “ADMET Properties” are oftentimes at tension with each other. For example, a good binder is likely greasy, but a greasy molecule is likely insoluble so it won’t enter the bloodstream and get to where it needs to go! Genesis’ advances in generative AI have now pushed them beyond the threshold where they believe agentic drug discovery loops are finally possible. We all remember the early days of LLMs. They were great chatbots but terrible agents, as small errors compounded rapidly into uselessness. As LLMs got better, the usefulness of agents rapidly improved. Evan and Sergey argue that their models at Genesis recently passed a similar threshold. Their internal agentic drug-discovery system (code named SAPPHIRE) can now iterate like a chemist: look at and reason about poses, form hypotheses, read literature, use internal tools, create candidates for the next iteration. Combining this with automated lab partnerships like the one Genesis has with Incyte, we’re rapidly approaching a time of drug discovery agents running 24/7 making/testing new molecules. Exciting times! Benchmark crisis: Everyone’s favorite benchmark is slop One surprising point that isn’t talked enough about: the academic field of “co-folding” has settled on a benchmark value of “2 Angstrom RMSD” as a metric for a “good pose”. Evan does not mince words: this threshold is just bad. Perhaps even deceptively bad. For many strong binders, there’s a very clear pose, one that you can even directly resolve in the PDB electron density! And yet, with a 2Å RMSD threshold, you can get the pose quite wrong in ways that might even mislead a medicinal chemist. For example, flip around an aromatic ring, and everything looks reasonable, but you’re no longer modeling the right interactions. Evan makes the strong claim that 1Å RMSD is really the threshold necessary to ensure the core of the molecule is sitting where it needs to be, and models all interactions. “If your model is sitting at 1.8, 1.9 Angstrom RMSD, that’s slop, most likely.” — Evan Feinberg As a simple example, he points out hydrogen bonds which are responsible for many of the most important interactions in protein-ligand systems. Hydrogen bonds only have a 0.6Å range to be valid! Clearly if you’re accurately resolving all H-bonds, you generally have to be doing much better than the 2Å threshold. This is clearly a hard-fought lesson for Evan and Genesis. In their opinion, the community is stuck on these benchmarks because academics developing methods were not users. Evan does see signs of life, with the use of new metrics such as lDDT for co-folding. Hopefully soon the community can agree that “1.8Å RMSD is slop”, and start hill climbing on this much harder task. For a more thorough exploration of the weaknesses in conventional benchmarks, see the PEARL technical report. PEARL tops OpenBind Which makes what happened next all the more striking. Near the end of the podcast, we talked about a recent “proof-is-in-the-pudding” moment for Genesis — evaluating their PEARL model on a recently released OpenBind benchmark. This benchmark featured 802 never before seen co-complexes on a target protein EV-A71. This target seems almost custom-chosen to give most classical docking methods a problem. When a ligand binds to the main binding site, the protein moves around to close off the path the ligand used to enter the binding pocket. This process, known as “induced fit” is notoriously hard for traditional methods to model. The tradeoff is easy to understand: treating the protein as a static structure, it becomes difficult to place a ligand in a binding pocket. Treat the protein as dynamic, and now you have to simulate complicated processes that take a long time to resolve. PEARL was able to model the induced fit of the ligand without running long MD simulations. Across the different evaluation metrics, PEARL came out not just ahead, but oftentimes well ahead of any public model. A truly impressive result. “Where PEARL was exceptionally good is figuring out how to move this loop. We are basically correct for every single pose.” — Sergey Edunov Even more exciting, this was done without any fine-tuning, or using any data on the target or homologous targets — the template PDB was released after PEARL’s training cutoff. Where does co-folding go now? As someone who has followed or participated in ML techniques for protein-ligand interactions for almost a decade, I was genuinely impressed with the results that Genesis has released recently. This has been many years in development, and I’m sure Evan and the team had many sleepless nights trying to get to this point. I also think other teams are making similar progress — both Isomorphic and Deep Origin have released results that seem spiritually similar and combine computation, wetlab data, ML, to achieve genuine predictive power that seemed impossible a decade ago. Sadly, all of the above are closed source so there’s no way to honestly compare them. Looking at the results I think there might be a time in the not so distant future where we can consider protein-ligand binding “solved”. I sincerely hope that the academic community can take inspiration from these developments. Once you know something can be done, it’s much easier to execute. Still, I believe that the key enabler in all of the above was the tight integration of ML, large-scale computation, and real-world drug discovery applications. Sadly academia is just not structured in a way that makes such a development easy. With those parting thoughts, we hope you give the podcast a listen! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

    1hr 49min

  3. 24 Jun

    Why the Frontier Ecosystem must be Open — Matei Zaharia and Reynold Xin, Databricks

    We’re excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers! Everyone is still talking about Satya’s Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today. From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work. We go deep on Omnigent: Databricks’ open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness. Then Reynold walks through Databricks’ database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks’ infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top. Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn’t need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over. Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it’s the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it’s doing work?” If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly. Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents. Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve! We discuss: * Why Databricks built Omnigent as a meta-harness above existing AI agents * Why coding agents and custom enterprise agents need the same infrastructure * The common API for agent sessions, files, streams, tool calls, and cancellation * Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter * Why Databricks open-sourced Omnigent instead of keeping it proprietary * Databricks’ internal agent usage, cloud sandboxes, and coding workflows * The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast * Why agent security needs contextual and stateful policies * How an agent could read confidential docs, install a compromised npm package, and leak data * Why spend control matters when an agent can burn $500 reading logs * Startup opportunities around coding-agent analytics, quality, skills, and spend * LTAP, Lakebase, and why Databricks wants to rethink the database stack * OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m. * Why HTAP has historically been the holy grail of database engineering * Why Databricks thinks LTAP is “HTAP done right” * How writing transactional data into column-oriented formats changes analytics * Why agents need live operational context from databases, not just telemetry * How Databricks prototypes strategic systems without endless process * Enterprise vs tech customers, governance, procurement, and DIY culture * The “second system syndrome” risk of rewriting a database engine * Building a database engine from a decade of traces and quadrillions of data points * Why vector databases should never have been a separate category * Why open formats and AI changed the race with Snowflake * The Mosaic story, DBRX, Genie, document parsing models, and specialized model training * Why model customization and RL fine-tuning may become mainstream * Why “get the data there, slap some agent on top” may rewrite traditional software Matei Zaharia * LinkedIn: https://www.linkedin.com/in/mateizaharia * X: https://x.com/matei_zaharia Reynold Xin * LinkedIn: https://www.linkedin.com/in/rxin * X: https://x.com/rxin Databricks * Website: https://www.databricks.com * X: https://x.com/databricks Timestamps 00:00:00 Introduction 00:02:22 Omnigent and the Agent Infrastructure Layer 00:08:39 Agent Clouds, Common APIs, and Open Source 00:16:52 Databricks Scale and Internal AI Workflows 00:18:03 Agent Security, Governance, and Spend Controls 00:27:34 LTAP and the Database Dream 00:30:30 CDC, HTAP, and Why Data Pipelines Break 00:34:05 Lakebase, Parquet, and Live Data for Agents 00:36:47 Databricks’ Culture of Fast Prototyping 00:43:40 The Dream Engine and Rewriting the Database Stack 00:51:02 Vector Databases, Query Engines, and LTAP 00:52:36 Databricks vs Snowflake 00:57:48 Mosaic, DBRX, Genie, and Specialized Models 01:03:11 Context, AI Runtime, and RL Fine-Tuning 01:06:15 Why Data + Agents May Rewrite Software 01:07:09 Closing Thoughts Transcript Introduction: Databricks, Data + AI Summit, and Founder Dynamics Swyx [00:00:00]: Matei and Reynold from Databricks, welcome to Latent Space. Reynold Xin [00:00:06]: Hey, thanks for having us. Swyx [00:00:07]: Yeah. Matei Zaharia [00:00:08]: Yeah, thanks so much. Swyx [00:00:09]: thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people Reynold Xin [00:00:17]: Yeah, it was Swyx [00:00:17]: in Berkeley Reynold Xin [00:00:18]: little meetup at Berkeley, I think Matei Zaharia [00:00:19]: Yeah Reynold Xin [00:00:19]: put together Matei Zaharia [00:00:20]: We were doing these tutorials and, yeah, just teach people Spark. Swyx [00:00:23]: Yeah. obviously now it’s like, I think like the headline number’s like 100,000 people around the world, 30,000 in person. Swyx [00:00:30]: it’s a crazy Matei Zaharia [00:00:31]: Amazing Swyx [00:00:31]: community. Well, I just saw the keynote. Swyx [00:00:35]: Ali’s just. Did was it obvious or that back when that Ali would be, like, such a great, like, CEO? Like Reynold Xin [00:00:42]: Oh Swyx [00:00:42]: such a great presenter? Reynold Xin [00:00:43]: What do you think? Matei Zaharia [00:00:44]: I think among our group of founders it was clear that, I think he’d be the best at this. Swyx [00:00:50]: Yeah. Matei Zaharia [00:00:50]: And yeah, it turned out great. And he’s, he’s ramped up on so many topics growing a company. He would just go in and, like, study it and, be talk to all the experts. Like, even if he can’t hire the person, learn enough about, like, finance and sales and whatever it was, and, and go from there. Yeah. Swyx [00:01:09]: Yeah. Reynold Xin [00:01:10]: he’s obviously very high IQ and a very high EQ, but it wasn’t. Like, Ali today is quite different from Ali from, like 10 years ago. I think there’s a lot of work that he put in to, get to this point. Swyx [00:01:20]: Yeah. no, to me the most appealing thing about him is that he’s funny. And like, it, it’s, it’ Matei Zaharia [00:01:26]: It’s true, yeah Swyx [00:01:26]: it’s hard to make jokes about, data warehouses Reynold Xin [00:01:30]: About serious topics Swyx [00:01:31]: security Matei Zaharia [00:01:32]: Yeah Swyx [00:01:32]: what have you. Matei Zaharia [00:01:33]: Oh, yeah. That’s for sure. Swyx [00:01:34]: Yeah. So you guys launched a whole bunch of things. I’ll, I’ll just name check briefly, the stuff because we’re not gonna cover everything. Omnigentt, your baby. LTAP, your baby, your dream engine. Swyx [00:01:47]: we’re also gonna cover Genie, cover CustomerLake, you acquired Panther Matei Zaharia [00:01:52]: Yeah Swyx [00:01:52]: Open Sharing, and there’s Unity AI Gateway. A lot of these, I think, like, are things that you would expect a Databricks to do. It’s, it’s like part of the roadmap. Everyone in your category has similar things. But I think, probably the two of you are leading the two most unique and differentiated initiatives Omnigent and the Agent Infrastructure Layer Swyx [00:02:09]: on, in the landscape. Maybe we’ll start with, Omnigentt we’ll, we’ll, we’ll, we’ll go into it. I do think that a lot of people are exploring this meta harness concept. Matei Zaharia [00:02:21]: Yeah, totally. Swyx [00:02:21]: What led you to it? Matei

    1hr 9min

  4. 22 Jun

    Red-Teaming after Mythos — Zico Kolter & Matt Fredrikson, Gray Swan

    AI Engineer World’s Fair regular bird tix will sell out ~today! Join us next week ahead of the Late Bird price hike and get >$40,000 in sponsor credits for attending! Thanks to the US Government issuing an export control directive on Mythos and Fable, the risks of jailbreaks and (industry term) indirect prompt injection are suddenly the talk of the town, though we have been covering AI security for a few years now, from Hackaprompt to the enigmatic Pliny the Elder. Zico Kolter, member of OpenAI’s board of directors on the Safety & Security Committee, and Matt Fredrikson, CMU professor and CEO of Gray Swan, co-authored the definitive paper on Indirect Prompt Injections, and Gray Swan were cited authorities on the Mythos model card, directly investigating the exact capabilities that are under scrutiny right now: We seized the opportunity to ask them the state of AI Red Teaming, and Shade, the adversarial red teaming tool that Anthropic used to evaluate the robustness of their models against prompt injection attacks in coding environments. Shade is part of their overall toolkit covering Simon Willison’s Lethal Trifecta, including Cygnal, an AI guardrails product, and the world’s largest AI Red Teaming Arena, including AIRT celebrity Wyatt Walls. All of this security tooling, and yet, we’re only staving off the inevitable. The risks of extremely smart AI increasingly feel like gray swan events: an event that everyone can see coming. In this episode, Gray Swan cofounders Zico Kolter and Matt Fredrikson join swyx to explain why AI security is not just “cybersecurity with AI,” why agents introduce a new class of vulnerabilities, and why the next major AI incident may be a gray swan: unlikely, but clearly visible before it happens. We go deep on prompt injection, automated red teaming, model robustness, agent identity, computer-use agents, enterprise guardrails, and the emerging AI insurance/compliance stack. Zico and Matt also explain why frontier models are not automatically safer as they scale, why specialized red-teaming models can now beat humans at breaking AI systems, and why the future of AI security may depend on AI systems attacking, defending, and interpreting other AI systems. We discuss: * Why AI systems need a different security mindset from traditional software * How prompt injection creates a new exploit class for agents like Codex and Claude Code * Gray Swan Arena and the rise of community red teaming * Shade: AI that can outperform humans at breaking models * Why LLMs are an alien form of intelligence that fail differently from humans * Human vs browser-agent robustness and why humans ranked fourth * Why eval awareness and capability elicitation matter * Cygnal: Gray Swan’s guardrail model for policy enforcement * Why bigger models do not automatically become more robust * The lethal trifecta: untrusted data, private data, and exfiltration * Why “just prompt it better” is not enough for enterprise AI security * OpenClaw, computer-use agents, and the agent security nightmare * Agent-native identity, permissions, and enterprise deployment * Why AI security may become part of insurance and compliance * Why the first major AI prompt-injection breach may be inevitable Gray Swan * Website: https://www.grayswan.ai/ Zico Kolter * X: https://x.com/zicokolter * Website: https://zicokolter.com/ * LinkedIn: https://www.linkedin.com/in/zico-kolter-560382a4/ Matt Fredrikson * Website: https://www.mattfredrikson.com/ * LinkedIn: https://www.linkedin.com/in/matt-fredrikson-7596349/ Timestamps 00:00:00 Introduction 00:02:31 Why AI Security Is Different 00:06:38 Testing Claude, Codex, and Prompt Injection 00:07:47 Gray Swan Arena and Automated Red Teaming 00:11:14 AI That Breaks Models Better Than Humans 00:14:00 LLMs as Alien Intelligence 00:19:00 Humans vs AI Agents 00:24:35 Red Teaming, Jailbreaks, and Capability Elicitation 00:26:11 Cygnal: Guardrails for AI Agents 00:34:04 The Lethal Trifecta 00:39:31 Can AI Automate AI Research? 00:45:47 OpenClaw and the Computer-Use Security Problem 00:50:44 Agent Identity, Permissions, and Enterprise AI 00:54:24 The Future of AI Security 01:00:30 AI Insurance and Compliance 01:04:32 The Gray Swan Event Everyone Sees Coming 01:06:04 Closing Thoughts Transcript Introduction: Gray Swan, AI Security, and CMU Swyx [00:00:00]: We’re here in the studio with Gray Swan, Matt and Zico. Welcome. Zico [00:00:08]: Great to be here. Matt [00:00:09]: Thanks for having us. Swyx [00:00:10]: You’re visiting from Pittsburgh? The home of all good computer science. I don’t know if I’m overstating things. A very strong university. Zico [00:00:18]: CMU has been the center of a lot of AI since really the dawn of the field. Swyx [00:00:22]: Especially a lot of self-driving and some language learning. Congrats on your Series A. You’re here because you’re attending Snowflake Summit, and Snowflake is one of your investors. Let’s introduce crisply at the top: what is Gray Swan, and what have you chosen as your startup domain? Matt [00:00:42]: At Gray Swan, our mission is to empower everyone to use AI safely and securely. Large language models are software, and if you want to deploy them or build applications on top of them, you need to understand the vulnerabilities and what can go wrong. That includes everyday mistakes, like an agent making the wrong tool call, but also worst-case scenarios where an attacker has an incentive to make your agent misbehave, leak data, or steal credentials. Gray Swan grew out of our research at Carnegie Mellon, where Zico and I have spent over a decade studying new vulnerabilities and attack surfaces in deep learning systems: how to test for them, understand their severity, and make inference more robust. Adversarial Examples and Why AI Security Is Different Swyx [00:02:05]: Honestly, a very fruitful area of study for any academic. Throwback, this is 10 years ago, which is basically the entirety of me. I got a lot of inspiration from Ian Goodfellow, a friend of the pod, and this is one of those initial adversarial settings. Matt [00:02:23]: This paper was directly inspired by Ian’s work. Swyx [00:02:29]: Zico, what about your side of the story? Zico [00:02:31]: Like Matt, I have been faculty at Carnegie Mellon for a while. Fundamentally, we believe in the transformative power of AI. It has already transformed the software ecosystem, and it will transform many other ecosystems going forward. The issue is that these systems behave very differently from the software we are used to. I do not just mean that AI can find vulnerabilities in software, though it can. I mean that AI systems have inherent vulnerabilities of their own. They can be tricked in ways people can be tricked, so you need a different security mindset. Zico [00:03:23]: This matters especially when there is the possibility of correlated failures. It is not just that there are many AI systems out there; it is that everyone is using a few models. If you find vulnerabilities in agents that everyone uses, like Codex and Claude Code, you have a new class of exploit. The labs are doing a lot of work here, but when a new platform emerges, a separate security system often emerges alongside it. That is where we are with AI: there is a need for specifically minded AI safety and security providers, and the demand is only going to grow. Treating Models as Untrusted Systems Swyx [00:04:55]: I want to highlight right at the top that this is not a cyber episode in the traditional sense. A lot of people looking at the title might think that, but you’re actually trying to treat these models inherently as untrusted entities? Zico [00:05:11]: Exactly. This is a common conflation because AI is also good at cybersecurity problems, both solving them and causing them. But AI systems themselves introduce new vulnerabilities. Gray Swan is not about using AI to make your cyber infrastructure better; it is about understanding and mitigating the security risks you bring in when you adopt and deploy AI. Matt [00:05:49]: A big part of that is how people are using artificial intelligence. Once you build entire autonomous systems on top of models and integrate them into your larger platform or network, you have a potential cybersecurity risk. The goal is to mitigate the risk posed by the AI as it relates to your broader cybersecurity goals. Testing Claude, Codex, and Indirect Prompt Injection Zico [00:06:17]: Part of this is red teaming. One reason we reached out to you was that you were involved in the Claude Mythos preview, where you were one of the authorities on IPI, or indirect prompt injection. When you receive a model, it does not have to be Mythos, but that is the most prominent one right now: what do you do with it? Matt [00:06:38]: We do a range of things. In the Mythos case, the concern from Anthropic was how robust the model is to indirect prompt injection. If you operate a coding agent and use Mythos as the model, it will fetch untrusted content and read text you do not control. How robust will it be at staying true to its original objective and not getting hijacked? We also help frontier labs test their safeguards for issues like cyber misuse. Broadly, we provide adversarial safety and security evaluations so model builders can assess progress from one iteration to the next. Zico [00:07:37]: They also do this in-house, and Anthropic is very ideologically inclined to do it. What do they choose to outsource versus keep in-house? Gray Swan Arena and Automated Red Teaming Matt [00:07:47]: So there are two things that I think, we stand out for. One is the Gray Swan Arena. So we operate a community of red teamers. We provide, prize challenges. a lot of these come from the needs of the lab sponsors. so to an extent gamify red teaming objectives, put up a prize pool, and pay people when they find ways to circumvent and violate whatever the safety and security objectives of the model developers wer

    1hr 6min

  5. 18 Jun

    The Professor of Outputmaxxing — Anjney Midha, AMP

    Last 4 days before regular tickets sell out at AI Engineer World’s Fair - this is the single biggest gathering of AI Engineers, Founders, Leaders, and Researchers in the world. Attendees get >$5000 worth of sponsor credits and talk tracks are looking FANTASTIC. Join us! The AI scaling debate always focuses on the question of “how do we get more GPUs?” but the better question may be: how do we make the most of ones we already have. The fact that a frontier lab like xAI could be running at sub-10% MFU (Model FLOPs Utilization) is just a hint at what the real problem may be. For context, older frontier-scale training runs were already much higher than 10%. GPT-3 was around 21% MFU. Gopher was around 32%. Megatron-Turing NLG was around 30%. PaLM reached around 46%. And our guest Anjney says best-in-class MFU today is closer to 60–70%. It’s not necessarily that xAI is uniquely incompetent (it’s clear they have talented folks) but rather the priorities may be flipped in the GPU arms race. While GPU access is a bottleneck, simply increasing CapEx won’t automatically translate to better models as frontier AI is increasingly a systems problem: scheduling, utilization, networking, kernels, frameworks, data pipelines, parallelism, cluster reliability, and the thousand small decisions that determine whether your theoretical FLOPs become real training progress. From building Discord’s developer platform and backing frontier AI companies like Anthropic, Mistral, Black Forest Labs, and Periodic Labs to now building AMP’s independent compute grid, Anjney Midha has spent years close to the real bottlenecks of AI scaling. In this episode, Anjney joins swyx at Periodic Labs to unpack why the AI race is not just about buying more GPUs, why 95% utilization would have been considered an outage at Google, and why the next era of AI infrastructure has to be more aligned, more efficient, and more responsible. We go deep on AMP’s vision for a compute grid that makes FLOPs flow like megawatts, the difference between full-stack AI labs and horizontal pooling, why AI data centers need community buy-in, and how compute markets could evolve into something closer to an independent system operator. Anjney also explains why DeepMind’s unpublished research points to a market failure, why end-of-life prediction remains one of the most important AI applications he has thought about for fourteen years, and why “output maxing” may become a new discipline for frontier systems. We also discuss Anthropic’s culture, why “luck favors the prepared mind” in coding models, how Claude cracked coding, why too much capital too early can make AI labs fragile, what Periodic Labs is trying to do with science and superconductors, why great researchers can become great CEOs, and why Silicon Valley is both deeply missionary and deeply mercenary. We discuss: * Why 95% utilization was considered an outage at Google * Why AI infrastructure waste compounds at frontier-lab scale * Why “move fast and break things” does not work for AI data centers * How data center backlash, power grids, and community incentives shape AI scaling * AMP’s vision for making FLOPs flow like megawatts * Why compute needs an independent system operator * How interruptible demand and dynamic prioritization worked inside Google * Why DeepMind research hoarding creates negative externalities * AMP’s 1.2GW base-load ambition and the need for 6GW of spike capacity * Why end-of-life prediction could become one of AI’s most important healthcare applications * Frontier Systems, output maxing, and full-stack alignment * Why APIs and abstraction layers become lossy as organizations scale * Superconductors, standards, and the dream of lossless systems * SF Compute, open protocols, and the future of compute marketplaces * Why non-NVIDIA chips can still benefit from NVIDIA’s reference architecture * Trust boundaries and why chip startups need visibility into future model architectures * Why VCs often underestimate researchers as CEOs * Scientists as star athletes of the mind * Why great CEOs need to be confrontational up and down the stack * Why leading the frontier matters more than “winning” * How Anthropic cracked coding * Why culture is fragile, not a permanent moat * Why hardship was a feature, not a bug, for Anthropic * Why Anthropic’s P0 was coding from day one * Periodic Labs, physics as the constraint, and technical reality * Silicon Valley mercenaries, missionary teams, and what happens after a breakthrough Anjney Midha * LinkedIn: https://www.linkedin.com/in/anjney * X: https://x.com/AnjneyMidha AMP PBC * Website: https://amppublic.com/ * X: https://x.com/amppublic Timestamps 00:00:00 Introduction 00:00:09 Why AI Compute Is Being Wasted 00:03:17 Responsible Infrastructure and Data Center Backlash 00:06:07 AMP Grid: Making FLOPs Flow Like Megawatts 00:12:41 Foundry, Frontier Labs, and Research Hoarding 00:14:42 Gigawatt-Scale Compute and End-of-Life Prediction 00:24:08 Frontier Systems, Output Maxing, and Alignment 00:27:38 Compute Markets, SF Compute, and Non-NVIDIA Chips 00:32:57 Trust Boundaries, Co-Design, and Researcher CEOs 00:38:17 AI Coachella and First-Principles Thinking 00:42:43 Leading vs Winning in Frontier AI 00:45:54 How Anthropic Cracked Coding 00:48:25 Culture, Hardship, and Anthropic’s P0 00:54:03 Periodic Labs, Physics, and Silicon Valley Mercenaries 00:56:26 Rishi Valley, Singapore, and Money as a Measure 00:58:47 Closing Thoughts Transcript Introduction: Anjney Midha, AMP, and Compute Waste Swyx [00:00:00]: We’re in Periodic Labs with Anjney Midha, CEO, founder of AMP. Welcome. Compute Utilization: Node Allocation, MFU, and Alignment Anjney [00:00:09]: Thanks for having me. At Google, there are two types of utilization usually, right? That you’re measuring in these clusters. One is node allocation, and then the other’s MFU. Node utilization is usually like what percentage of cards in the data center are just, used, and that, if it’s not at, 95%- Swyx [00:00:29]: There is no excuse Anjney [00:00:29]: There’s no excuse, right? I think 95% at Google, which is where my co-founder, Seb, came from, he built the Borg, PBorg/GQM scheduler at Google, and there I think 95% was considered an outage, so 96% node utilization is, should be standard. And most single-tenant clusters are not running at that. So that’s one. And then MFU should be, I would say the best in class today is somewhere between 60 and 70%. I think this is a leadership question, right? Fundamentally it’s an alignment question, which is are the people who are funding the cluster and then deploying the cluster actually aligned? And sometimes theoretically they are, but in practice the number of people in the chain, the supply chain between, the capital and all the way to whoever’s managing the cluster and then whoever’s measuring what the output is, are just so many, degrees of separation away that, the, The Have you ever heard the radian metaphor, which is at the beginning of an arc, if you have two arcs that are two lines that are just off by a few degrees, that- Swyx [00:01:33]: It spreads out Anjney [00:01:34]: It spreads out, right? Or at scale. And I think what’s happening is a lot of cluster implementations and infrastructure, a lot of frontier labs and other teams, that’s what’s happening, is they’re, they initialize the plan, which is kind of like North Star with a team that wants to do good, but then they’re, required to scale so fast instead of iteratively that the wastage just compounds really fast at scale. And so I think we know the answer, which is just do iterative bring ups. If you spend time with people who’ve been in the semiconductor industry or the DSN industry for a long time, this is not new, and I don’t think AI should be an excuse. Sure. Something What is new? Okay. We have a lot of new capabilities, but that doesn’t mean just abandon common sense. Common sense should always be in fashion. ? AI scaling doesn’t change the in fact, if anything, AI scaling should be putting a premium on the value of common sense and infrastructure because the margin of error now is so much lower and the costs of wastage are so much higher. And the cost of wastage, by the way, is not just economic. I’m, obviously I’m, I’m an investor, or I’m an investor by background. Over the last few years now we’re running an AI infrastructure business called, AMP. And I think that it’s okay to say this time is different on the capabilities front. We are genuinely getting capabilities at, of the, of a kind we haven’t had before. That doesn’t give you an excuse to say this time is different for everything, especially infrastructure. So look, I love the hacker mindset and the hustler mindset. Now, that’s great for the startup mindset, but you remember this moment where Zuck went from saying, “Move fast, break things” to, move- Responsible Infrastructure and Data Center Backlash Swyx [00:03:10]: Fast and stable infrastructure Anjney [00:03:11]: Move fast with stable infrastructure. I think now we need to move fast with, responsible infrastructure. People are going to ask where the impact is. There was a really In our class yesterday, Scott Nolan, who’s the founder of General Matter, came by at Stanford to speak about energy bottlenecks. And he had a phenomenal idea. He said, “if you look at the marginal unit economics of compute per hour,” he goes, “let’s call it, $4 an hour. If you’re having to bring up a new data center in a new community, why not just say we’re going to charge 4.50 an hour, and that marginal impact or that marginal increase, we just literally take that and give it to the local community as cash?” I can tell you as a customer of that compute, I would love that. I’d be happy to pay an additional 50 cents per hour at scale. Swyx [00:03:57]: Wow. Yeah. Anjney [00:03:58]: Because if that

    59 min

  6. 17 Jun

    🔬 The Self-Driving Lab — Joseph Krause, Radical AI

    On the Science pod, we’ve been covering a lot of the ground on how AI is revolutionizing STEM, but one of our favorite off the record topics since our launch is which field is harder to accelerate: math, bio, or physics? Today we’re back in Materials Science land with Radical — Unlike biological molecules that can be represented (and predicted!) by token strings, the success of materials involve many more macro complex variables like supply chains, microstructures, and manufacturing processes. If you recall the LK99 drama of 2023, while the basic ingredients were known, part of the confusion came from the lack of disclosure around manufacturing, and therefore defeated reproducibility. There is probably no "one-shot" model capable of designing a material that works perfectly at scale. How Radical is accelerating materials discovery >10x the pace of DARPA/GE MACH Joseph Krause is a materials scientist through and through. And after spending his career watching industries stall out waiting for better materials, he founded Radical AI to do something about it. We recently sat down with Joseph to talk about Radical AI, materials discovery, self-driving labs, and the future of AI science. Joseph did not sugar coat anything: accelerating the materials discovery pipeline is a hard problem. But it’s one that he strongly believes we need to invest in, for the future of consumer products, aerospace, computing, and defense, and get them into every day use: “We count it as a discovery when you pick up your phone and there’s a new material sitting inside of it.” How does Joseph plan on accelerating the rate of discovery? To understand this, it’s important to understand why this is such a hard problem in the first place. The first thing to keep in mind is that the material that is manufactured is far more than a chemical formula going into it. The process of mixing, annealing, growing, or generating the final material can result in wildly different outcomes. The entire materials discovery process, both from early discovery to large scale manufacturing, needs to be understood and characterized. The Self-Driving Lab This philosophy has grown into a key insight at Radical AI: The construction of the self-driving lab. This lab is one that is not just automated, but in fact uses an “AI scientist” that combines scientific knowledge, computational techniques, and human intuition to generate and test hypotheses in an automated lab. Creating an AI scientist was key to making Radical’s self-driving labs work, since Joseph argues that no single AI model can one-shot materials. “In materials, the ground truth is the material itself. You have to be able to test it and characterize it.” Joseph talked at length about the self-driving labs at Radical. Joseph argues that experimental data is the true “moat” in this industry. An SDL functions as a closed-loop system where an AI scientist generates hypotheses, and automated robotics synthesize and characterize materials, running research campaigns in parallel rather than serially. The successes here were both on the automation side and on the science side. Radical has managed to scale their alloy discovery pipeline up to producing and characterizing 1200 alloys in six months — this nearly 10x speedup over the DARPA/GE MACH program that aimed to create 500 new alloys in a year. Joseph claims they can scale this up even more and estimates they can produce a hundred new alloys tested and characterized in a day. A truly new paradigm in high-throughput alloy experimentation. On the science side, their AI scientist proposed and tested 300 new materials, ten of which were found to have novel state-of-the-art properties that are already being further developed for commercial applications. The robustness of this first materials campaign reinforces Joseph’s claim that the moat is the lab and data. “It’s moved into elemental families or alloy families no one has ever published on before.” Interestingly, Radical’s AI scientist has made some novel discoveries, expanding into elements that just were not explored prior. This is fascinating from a scientific perspective, but it’s also important for helping reduce supply chain bottlenecks for vital industries! Joseph spent a lot of time in D.C. before founding Radical, and he’s clear-eyed about the competitive threat. China’s centralized model lets it stand up manufacturing hubs and immediately scale new materials from lab to production. We can’t replicate that, and Joseph is very clear we shouldn’t try. But we do need an answer. For Joseph, that means transforming the scientific workforce, investing in self-driving lab infrastructure at the national lab level, and leaning hard into public-private partnerships. “Now imagine every scientist in the United States doing 10 times the research output. That’s fundamental. That just changes the trajectory of discovery.” Before we close, we’d like to give a shout out to Joseph and Radical for publishing and open sourcing much of their internal tooling pipeline. This includes: * TorchSim (preprint, blog): an open-source PyTorch-based MD simulation framework, which has been spun off into its own non-profit. * MATRIX/MATRIX-PT (preprint, blog): An open-source dataset for benchmarking autonomous self-driving labs (MATRIX), along with with an open source model based upon this dataset (MATRIX-PT). We could talk about this extensively, but a fun data point is that improving reasoning in the area of materials also improved reasoning for biological systems! This is a truly unexpected result. Big shout-out to the Radical team for sharing their work! Materials discovery has been stuck on a 20–30 year timeline for generations. Joseph thinks that’s about to change, and Radical AI is putting that thesis to the test in the lab, one sample at a time. We had a great time talking with Joseph. We hope you give it a listen! Timestamps * 0:00 Introduction to the challenges of AI in material science * 0:52 Welcome and introduction to Joseph Krause and Radical AI * 1:38 Why Radical AI is different: The focus on experimental data and Self-Driving Labs (SDLs) * 6:19 The process: Candidate generation, synthesis, and characterization * 11:05 The application of exotic alloys in extreme environments (aerospace and defense) * 13:20 Barriers to entry: The slow process of qualification and manufacturing * 16:06 Supply chain constraints in material science * 19:24 Human-in-the-loop: Training the AI using scientific intuition * 20:35 The engineering challenges of automating a laboratory * 23:17 Defining the “Self-Driving Lab”: Research campaigns vs. just automation * 24:39 Mechanical challenges: Handling high-temperature samples * 27:41 Future scaling plans and the “Vertical Integration” strategy * 30:08 Validation timelines for high-tech industries (semiconductors, aerospace) * 31:47 The active learning loop and handling “negative results” * 35:32 AI exploring elemental families beyond human bias * 39:13 Throughput targets and the difference between AI and human exploration * 43:52 Why the dataset size is less critical than the quality of experimental feedback * 46:20 Addressing the lack of an “AlphaFold” for materials * 53:49 War stories from the lab: Building the infrastructure * 58:12 The shift in industry sentiment toward SDLs and tool interfaces * 1:01:14 Geopolitical considerations and the race in material science innovation * 1:06:12 Calls to action for ML and AI engineers: Rethinking the scientific stack * 1:09:53 The Matrix model and using VLM for scientific knowledge extraction * 1:13:10 Why Radical AI is open-sourcing their work This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

    1hr 17min

  7. 4 Jun

    Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs

    The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets! Most industry benchmarks compress intelligence and reasoning ability into scores. SWE-Bench Pro, MMLU, Humanity’s Last Exam, etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench. In Anthropic’s Mythos Preview System Card, Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior: You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior: deception, context collapse, emergent coordination, & bizarre negotiation behavior. While an inflection point in personal agents came post-OpenClaw after full file access with bypass permissions became the norm, it is yet to come for agents in the real-world. However Andon Market, an actual in person store fully run and managed by AI, is paving the way for what is possible. Full Video Pod From Claude trying to call the FBI over a $2/day vending machine charge to AI agents forming price cartels, hiring human employees, running physical stores, and writing existential robot musicals, Andon Labs is stress-testing what happens when frontier models stop being chatbots and start acting in the real world. In this episode, Andon Labs cofounders Lukas Petersson and Axel Backlund join swyx and Vibhu to unpack the strange, funny, and genuinely concerning edge cases that emerge when agents run businesses over long horizons. We go deep on Vending-Bench, Project Vend, Vending-Bench Arena, Bengt, Butter-Bench, Luna, and Andon’s broader mission of building realistic real-world evals for autonomous AI systems. Lukas and Axel explain why dollar-denominated evals reveal things traditional benchmarks miss, how Claude ended up reporting its vending machine fees as cybercrime, why long context windows can drive agents into meltdown loops, what happens when agents compete with each other, and why the future of AI safety may depend on testing models in messy physical environments instead of clean benchmark sandboxes. We discuss: * Why Andon Labs started with dangerous capability evals and long-running agents * Vending-Bench and why running a vending machine is a deceptively hard AI benchmark * Why money-based evals avoid the saturation problem of traditional benchmarks * How Claude tried to call the FBI over a $2/day fee * Why long-horizon agents can spiral into existential and legalistic breakdowns * Project Vend: putting an AI-run vending machine inside Anthropic * Why real humans are “out of distribution” for simulated agents * Claudius, Seymour Cash, and the chaos of AI CEOs * How a human briefly became CEO of Claudius through a manipulated election * Why multi-agent systems can converge back into “helpful assistant” behavior * Bengt, Andon’s internal office agent with email, spending, terminal, phone, camera, and internet access * How Bengt traded Amazon purchases for face-recognition training data * Claude’s aggressive behavior, lies, refund avoidance, and price-cartel behavior in Arena * Why eval awareness may become the AI version of “are we living in a simulation?” * Blueprint Bench, spatial intelligence, and why models still misunderstand physical rooms * Butter-Bench and testing LLMs as robot orchestrators * Luna, the AI-run physical store with a three-year lease and human employees * The new Andon cafe in Sweden and why real-world geography matters for agent evals * Rotten tomatoes, perishable goods, and the hidden difficulty of running a physical business Lukas Petersson * LinkedIn: https://www.linkedin.com/in/lukas-petersson-181a83172/ * X: https://x.com/lukaspet Axel Backlund * LinkedIn: https://www.linkedin.com/in/axelbacklund * X: https://x.com/axelbacklund Andon Labs * Website: https://andonlabs.com * Vending-Bench: https://andonlabs.com/evals/vending-bench * Andon Vending: https://andonlabs.com/vending Timestamps 00:00:00 Introduction00:01:00 Andon Labs and the Origins of Vending-Bench00:05:21 Why Money-Based Evals Matter00:09:51 Agent Harnesses and Self-Modifying Systems00:13:36 Claude Calls the FBI00:16:33 Project Vend: Claude Runs a Real Vending Machine00:21:44 Seymour Cash, AI CEOs, and Election Chaos00:27:16 Multi-Agent Coordination and Slack Observability00:30:18 When Will Agents Run Real Businesses?00:34:56 Bengt: Andon’s Internal Office Agent00:40:06 Real-World AI Safety and Long-Horizon Traces00:44:28 Lying, Refunds, and Price Cartels in Arena00:52:42 Eval Awareness and Simulation Behavior00:56:06 Blueprint Bench, Butter-Bench, and Robotics01:04:37 Luna: The AI-Run Physical Store01:09:29 The Sweden Cafe and Real-World Expansion01:13:16 What Comes Next for Andon Labs Transcript Introduction: Andon Labs, Long-Running Agents, and Real-World Evals Swyx [00:00:00]: Welcome to Lukas and Axel from Andon Labs, and I’m joined by my, favorite guest host. Anything security, safety, alignments, Vibhu., welcome. Lukas [00:00:15]: Thank you for having us. Axel [00:00:16]: Thank you. Swyx [00:00:17]: Let’s match names to voices., maybe you wanna take turns introducing yourselves. Lukas [00:00:21]: I’m Lukas. Axel [00:00:22]: And I’m Axel. Swyx [00:00:24]: Let’s introduce Andon Labs a bit. How did you guys come together?, you have different backgrounds, but you’re both Swedish., was that, a big part of it? Lukas [00:00:33]: So when I went to high school, there was this really cool guy who had a superpower. He could code. So he made like the or like the app for the, for the school and stuff, and he was super cool, and I wanted to be like him, and that was that guy. Axel [00:00:47]: I don’t know about this. Swyx [00:00:49]: But you went to different universities, right? Lukas [00:00:51]: But same high school. Swyx [00:00:52]: I see. Lukas [00:00:52]: So we always said, “Oh, once we graduate university, then we should start a company,” and that’s what we did. Swyx [00:00:58]: Wow, there you go. And about a year ago, you kinda burst onto the scene with Vending Bench, but, was there a thing before that was, kind of like the inception? From Dangerous Capability Evals to Vending Bench Axel [00:01:07]: So we did work, yeah, with, Anthropic was one of our, early customers in doing, evals. So we did, dangerous capability evals., nothing we published openly. But then we started thinking about doing some kind of, public benchmark, and one thing that we really started thinking about, was like running agents and specifically agents managing businesses., ‘cause-- and this was, early 2025., and I think the first, mentions of people will be running, person unicorns or even autonomous companies. So we thought, “Let’s make a benchmark of how well can an agent run the probably simplest business, possible,” and, that’s probably, running a vending machine. So that’s the first public one we did. And it was very, like-- there was almost no one that noticed it in the first couple of months, I think., so we released it in February last year, and then I think around Easter last year, we got, the first viral tweet about it, that someone else did. Lukas [00:02:11]: We tweeted a bunch, uh When it came out and, tried our best. Axel [00:02:15]: We tried. Vibhu [00:02:16]: It’s the one at Anthropic, right? Lukas [00:02:18]: So this Swyx [00:02:19]: This is a classic thing we should get out of the way. Lukas [00:02:20]: Exactly. There’s two versions. Swyx [00:02:22]: Everyone does this. Yes. Lukas [00:02:23]: There’s Vending Bench, which is the simulated one, which we did, completely independently in February., and then, like Axel said, that was like-- That was the thing that didn’t get any traction in the beginning, but then some random person made a tweet about it, and that Axel [00:02:38]: You have the paper Lukas [00:02:38]: That is the paper. Correct, yeah., and then since we thought this was very fun, we thought, oh, I think this is also, one thing with Andon Labs, the way we kind of like decide what to do next and what projects to do, it’s what is like the heuristic we use is what is fun? Is What would be a fun project? And doing this in real life sounded quite fun for us, and maybe also scientifically useful. So, then we basically had this idea, and then we, like-- But then we needed a place for it and, putting it out in the public would probably not really work., would get vandalized and stuff. So we pitched it to the people we were already working with at Anthropic, and they were “Yeah, you can have space. This sounds fun.” Um Swyx [00:03:21]: It’s like a small fridge, right? It’s like a mini fridge. Axel [00:03:23]: Absolutely. Swyx [00:03:24]: People-- There’s like a stripe thing or like an Vibhu [00:03:27]: Oh, okay. So it was very OG, the early days Lukas [00:03:28]: That’s the OG one. Yeah Vibhu [00:03:29]: IPad on this. We saw it in June, like two months after After it had been there. They upgraded a little bit. There’s a security camera for making sure you actually Venmo the thing. Swyx [00:03:40]: So, my impression, okay, we’re, we’re going straight into project Ven because it’s such a iconic thing. I do want to cover a little bit of that, the origin story even before Project Ven and even into Vending Bench. I think a lot of people are like yourselves, like smart, interested in future of AI, interested in developing evals. But how the hell do you just, walk into Anthropic’s doors and, work with them, right? What is What are they looking for? What works? And t

    1hr 16min

  8. 3 Jun

    🔬Scaling Past Informal AI - Carina Hong, Axiom Math

    In 2025, seven-month-old startup Axiom solved all 12 of the problems Putnam exam (scoring 8/12 in the time limit) a prestigious undergraduate math exam. The 12/12 score is better than the top undergraduates (110/120) and the closest AI system that reported a result (DeepSeek 103/120), although it is unclear what the people and other systems would have scored with more time. Nonetheless, the Putnam exam is legendary for its difficulty, with the median score typically being 0 or 1 points. Taken by itself, this seems like a minor feather in the cap of AI; one of a long series of accomplishments by AI systems in elite competitions with humans, starting with Deep Blue beating Kasparov. Fast forward to mid-2026, and Claude Code is eating the world. In 2024 Anthropic’s bet on code and enterprise looked like a more pragmatic niche play vs. OpenAI’s better models and massive consume scale. Today, Amodei’s all in bet on acceleration via code (images and video be damned) seems prescient. Despite Anthropic’s growing momentum, however, Axiom CEO Carina Hong sees coding ability as a necessary but not sufficient milestone on the path to AGI. Code arguably pushes the jagged frontier to the point of super intelligence in some domains outside of coding, but there are surprising gaps (link) that Carina believes will bottleneck AI progress. (Stats on math benchmarks). The informal bottleneck “Verified AI” sounds like eating broccoli (footnote: I actually love broccoli, but then again, I also believe strongly in Test Driven Development, so ¯\(ツ)/¯ ) and paying taxes, but to Axiom it means something very different. “Verification to me is about scaling brilliance, compounding brilliance,” Carina told us. It actually took a while for me to understand what she means by this. It sounded like marketing-speak to me, until it clicked. Carina emphasizes an story about legendary mathematician Srinivasa Ramanujan to illustrate the point. When G.H. Hardy finally persuaded Ramanujan to formally prove theorems instead of relying on his (formidable) intuition, it reportedly improved his own capabilities. This is presumably because formally proving things forced Ramanujan to articulate the details in a way that open up new lines of thinking, etc. This is one part of “compounding.” But formally proving things also allowed others to benefit from his intuition: the proofs are way of communicating an intuition and persuading others that the intuition is correct. This is scaling (more people use the result) and compounding (people can learn from and build on his work). This is the analogy that Carina wants us to focus on. Verified Generation There are two ways that Verified AI shows up: in training and in inference. But a quick detour: to a first approximation, “Formal Verification” means using type checkers (like for TypeScript, C++ or Rust, but more capable) to verify mathematical proofs that are meticulously specified using a language like Lean (footnote: Formal verification also includes model checking (TLA+, SPIN), SMT-based tools (Dafny, F*, Why3), and refinement-type systems (Liquid Haskell) — many of which don’t look much like “type checking a proof” from the user’s perspective even when there’s a similar logical core underneath. It also gets applied to software and hardware correctness, not only pure mathematics.). It takes a lot of work to translate an “informal” proof (albeit one that most people would not remotely call “informal”) in to a Lean proof (footnote: This is an understatement. Most theorems remain informal because formalization is so hard to do. There has been a great deal of effort to formalize the most important proofs, with mixed results) You can imagine how this would be (very) useful during Reinforcement Learning: instead of relying on best guesses based on statistics (GRPO, RLHF, etc.), you can just verify the proof is correct using a Lean verifier. This is obviously a much stronger reward signal, akin to compiling code and testing it (which is what is typically done with RL on coding). The catch: LLM are not (currently) very good at proving things with Lean. Enter Axiom: While they have not officially reported benchmark numbers besides the 12/12 Putnam result, Carina reports that they have achieved a very impressive 99% (187/189) ProofGen on the Verina benchmark. This benchmark is to generate code and proof of correctness for a series of problems. For context, OpenAI o3 (the last known OpenAI run) achieved 4.9% on this benchmark. Based on the sparse benchmarking, it’s hard to say what the frontier labs are currently doing, but Carina suggests that they still are not training to generate Lean proofs directly, rather relying on informal proofs. Time will tell if the frontier labs’ current approaches will close this gap. Scaling and compounding Carina’s Ramanujan analogy is pretty direct. Better proofs → better Lean generation → better RL. A stronger signal means higher sample efficiency and higher maximum performance. Great! Scaling is pretty clear too: once I have proved something in Lean, the quality of the output is basically (footnote: one might argue that its a bit lower because the proof is in distribution for the LLM) as high as if it came from a human, so my high quality training set has grown in a way that an informal rollout corpus cannot. I can trust my Lean proofs. Compounding is also clear: now all of future inference and training can build upon those proofs. On the other hand, a model trained only using statistical signals like GRPO during RL lacks the sample efficiency, maximum performance and compounding corpus that a system that uses formal verification benefits from. All roads lead to verification Broccoli and taxes notwithstanding, “verification” has shown up in a lot of conversations recently. In the in physical system control: “I think [verifiability] is probably the hardest problem right now, because the as the models get better, it can be harder and harder to find the faults on the system. And so the problem of doing proper eval to find those faults, that problem also keeps getting harder as the models get better.” - In theoretical physics: “…now that we’re in this regime where you can just get ChatGPT to tackle thousands of questions at the same time, it will return proofs for a significant fraction of them. Now actually the onus is back on the humans to verify all the outputs. And so, yeah, as that becomes a bottleneck, I think formalizing math and automating verification will become more valuable.” - Verification is, in fact, the key differences between AI for science and AI for computation: in science you to have to actually test (verify) your hypothesis by performing physical experiments. Lab in the loop systems like Radical AI and Lila build around exactly this premise (we have recorded episodes with both of these teams and will release them soon!) And yes, formally verifying critical systems such as flight control, nuclear power plants and pacemakers is a growing focus as the software and hardware that run them becomes more complex. Carina believes so strongly that AGI requires verified generation that she makes the unqualified claim that “We do not believe there is any other possible future.” Expensive to produce, cheap to verify Lean proofs are hard generate, but they can be easily shown to be correct or incorrect. But how do you know that the proof you created maps correctly to the problem you care about? As Carina puts it: “Anything that can be specified can be proven. Humans are bad at specifying everything we want.” Are we now in the specification business? Check out the episode to hear Carina’s take, as well as: * Why hardware verification is a killer app * Details on the AXLE open API and recently released Discovery toolkit * The Erdos debacle * The OpenAI GPT-f diaspora This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

    1hr 33min
See All (213)

Ratings & Reviews

5
out of 5
6 Ratings

About

The podcast by and for AI Engineers! In 2025, over 10 million readers and listeners came to Latent Space to hear about news, papers and interviews in Software 3.0. We cover Foundation Models changing every domain in Code Generation, Multimodality, AI Agents, GPU Infra and more, directly from the founders, builders, and thinkers involved in pushing the cutting edge. Striving to give you both the definitive take on the Current Thing down to the first introduction to the tech you'll be using in the next 3 months! We break news and exclusive interviews from OpenAI, Anthropic, Gemini, Meta (Soumith Chintala), Sierra (Bret Taylor), tiny (George Hotz), Databricks/MosaicML (Jon Frankle), Modular (Chris Lattner), Answer.ai (Jeremy Howard), et al. Full show notes always on https://latent.space www.latent.space

Information

You Might Also Like