“It would be sad if local models were not an option and there were only proprietary models. It’s good to have alternatives. Competition is good for business.” — Sebastian Raschka, on open-weight AI Kimi K3’s weights landed about an hour before Hugo Bowne-Anderson and Sebastian Raschka went live. Sebastian had already updated his architecture diagram. That speed captures his approach to the current model wave: wait until the weights exist, run the model in the harness where it will actually work, then inspect the architecture closely enough to understand what changed. The conversation arrived during a larger fight over who supplies the models underneath global software. Three days earlier, twenty-five companies including NVIDIA, Meta, Microsoft, Hugging Face, and IBM published Open Weights and American AI Leadership. Their argument closely matches Sebastian’s practical case for local models: open weights create competition, reduce dependence on a single provider, and let organizations choose a model at the right capability and cost. Update: Four days after we recorded, DeepSeek released V4 Flash 0731, a re-post-trained API model for agentic coding. Developers are already reporting that it can debug multi-project codebases and stay on task across very long contexts. You can find the full episode on Spotify, Apple Podcasts, and YouTube. 👉 The next cohort of Master Agentic Data Science starts August 4. Hugo is teaching it with Thomas Wiecki and Luca Fiaschi from PyMC Labs. You will build agents that explore data, run predictive and causal workflows, challenge one another’s conclusions, and produce analysis humans can inspect and reproduce. Podcast listeners get 20% off with code MADSAGC20. Join the cohort. 👈 In This Episode * Kimi K3’s weights landed about an hour before we went live. Sebastian had already updated his architecture diagram, then walked us through how Kimi Delta Attention and its mixture-of-experts design turn Kimi Linear’s research architecture into a production model. * When a local model is already good enough. Sebastian separates everyday coding and private workloads from problems where the strongest available model, extra reasoning time, and higher cost are justified. * Why the model and harness must be evaluated together. The same open-weight model behaves differently across Qwen Code, Claude Code, Codex, Pi, and other harnesses because context handling, compaction, tool conventions, and post-training all affect the result. * The harness should choose the model and its reasoning effort. Sebastian describes a router that uses the task, conversation history, context, and previous loops to make that decision automatically. * How long agent contexts are changing model architecture. Sebastian moves through grouped-query attention, multi-head latent attention, sparse attention, mixture-of-experts, DeltaNet, and Mamba-style hybrid layers. * The implementation does not lie. A from-scratch model can be checked against open reference weights tensor by tensor, and the first divergence exposes the missing RoPE scaling detail, RMSNorm variant, or architectural assumption that prose left out. * When fine-tuning earns its cost. A specialized task performed thousands of times a day creates different economics from a task performed eight times, so the decision depends on quality, volume, latency, and token cost. * Sebastian keeps the creative work and gives agents the tedious checks. He uses Codex and Claude to check links, enforce editorial conventions, and compare notebook code with the corresponding book manuscript. About Sebastian Raschka Sebastian Raschka is an independent AI researcher, author, and educator focused on large language models, reasoning models, deep learning, and practical machine-learning systems. He created Ahead of AI and is the author of Build a Reasoning Model (From Scratch) and Build a Large Language Model (From Scratch). Resources * Build a Reasoning Model (From Scratch): podcast listeners get 45% off the ebook or print edition with code 45VGraschka2. * Sebastian Raschka’s LLM architecture gallery * Kimi K3 in the architecture gallery * Build a Large Language Model (From Scratch) * Ahead of AI * Using Local Coding Agents * Open Weights and American AI Leadership * Anthropic’s position on open-weight models Listen or Watch You can find the full episode on Spotify, Apple Podcasts, and YouTube. 👉 The next cohort of Master Agentic Data Science starts August 4. Hugo is teaching it with Thomas Wiecki and Luca Fiaschi from PyMC Labs. You will build agents that explore data, run predictive and causal workflows, challenge one another’s conclusions, and produce analysis humans can inspect and reproduce. Podcast listeners get 20% off with code MADSAGC20. Join the cohort. 👈 How You Can Support Vanishing Gradients Vanishing Gradients is a podcast, workshop series, blog, and newsletter focused on what you can build with AI right now. More than 70 episodes with expert practitioners from Google DeepMind, Netflix, Stanford, and elsewhere. Hundreds of hours of free, hands-on workshops. All independent, all free. If you want to help keep it going: * Become a paid subscriber, from $8 per month * Share this episode with a builder who would find it useful * Subscribe to the Vanishing Gradients YouTube channel * Join another Vanishing Gradients workshop Get full access to Vanishing Gradients at hugobowne.substack.com/subscribe
