AI Post Transformers

mcgrof

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 16h ago

    Kimi K3 Goes Trillion-Scale With Sparse MoE Efficiency

    This episode examines Kimi K3, Moonshot AI's open-weight frontier model boasting 2.8 trillion total parameters with only 104 billion active per token, a million-token context window, and native multimodal training from the ground up. The discussion traces the architectural lineage behind the model's claimed 2.5x scaling-efficiency gain over its predecessor Kimi K2, connecting its Mixture-of-Experts design back to Shazeer's 2017 sparsely-gated MoE work and contrasting its hybrid attention approach with the limits of standard residual connections from the 2015 ResNet paper. It also unpacks the systems-engineering side of running a model this large, particularly how Expert Parallelism turns token routing into a datacenter networking problem once hundreds of experts are sharded across GPUs. Listeners get a clear breakdown of why native multimodal training tends to be more stable than bolting a pretrained vision encoder onto a text-only model after the fact. The episode sets up a deeper dive into which of K3's four credited innovations — Kimi Delta Attention, Attention Residuals, Stable LatentMoE, and refined training recipes — is actually doing the heavy lifting. Sources: 1. Kimi K3 Goes Trillion-Scale With Sparse MoE Efficiency https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf 2. Deep Residual Learning for Image Recognition — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015 (CVPR 2016) https://scholar.google.com/scholar?q=Deep+Residual+Learning+for+Image+Recognition 3. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, Martin Jaggi, 2024 https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging 4. Value Residual Learning for Alleviating Attention Concentration in Transformers — Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Zhenzhong Lan (and collaborators), 2024 https://scholar.google.com/scholar?q=Value+Residual+Learning+for+Alleviating+Attention+Concentration+in+Transformers 5. Hyper-Connections — Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou (ByteDance Seed), 2024 https://scholar.google.com/scholar?q=Hyper-Connections 6. Attention Residuals — Kimi Team, 2026 https://scholar.google.com/scholar?q=Attention+Residuals 7. Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Team et al., 2025 (arXiv:2510.26692) https://scholar.google.com/scholar?q=Kimi+Linear%3A+An+Expressive%2C+Efficient+Attention+Architecture 8. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model 9. ReplaySSM: Cache SSM Inputs, Not State — Dao AI Lab, 2026 (blog) https://scholar.google.com/scholar?q=ReplaySSM%3A+Cache+SSM+Inputs%2C+Not+State Interactive Visualization: Kimi K3 Goes Trillion-Scale With Sparse MoE Efficiency

  2. 1d ago

    Hilbert Operator Reveals What Networks Actually Learn

    This episode examines HOPE (Hilbert Operator for Progressive Encoding), a structured pruning framework from Google DeepMind and UC Berkeley researchers that treats network compression as a diagnostic tool for understanding what deep networks actually learn, rather than just a deployment optimization. The discussion traces the approach's roots to the Information Bottleneck principle while carefully distinguishing HOPE's falsifiable measurement machinery from that unproven theory, and covers why magnitude-based pruning fails due to scale symmetry in batch-normalized networks, and how data-dependent pruning can quietly degrade long-tail class performance. The core innovation discussed is representing neurons as objects in a Hilbert space—comparing what function each neuron computes rather than the size of its weights—using only batch norm statistics already stored in a checkpoint, with no forward passes on real data and no hyperparameter tuning required. Listeners interested in interpretability, pruning theory, or the ongoing debate over why deep learning generalizes will find the hosts' back-and-forth on contested claims particularly engaging, as they push back on overstating the Information Bottleneck's explanatory power while crediting HOPE's mathematically rigorous, data-free approach to isolating a network's predictive core. Sources: 1. Hilbert Operator Reveals What Networks Actually Learn https://arxiv.org/pdf/2607.21366 2. Optimal Brain Damage — Yann LeCun, John S. Denker, Sara A. Solla, 1989 https://scholar.google.com/scholar?q=Optimal+Brain+Damage 3. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — Jonathan Frankle, Michael Carbin, 2019 https://scholar.google.com/scholar?q=The+Lottery+Ticket+Hypothesis%3A+Finding+Sparse%2C+Trainable+Neural+Networks 4. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015 https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network 5. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — Elias Frantar, Dan Alistarh, 2023 https://scholar.google.com/scholar?q=SparseGPT%3A+Massive+Language+Models+Can+Be+Accurately+Pruned+in+One-Shot 6. Priors for Infinite Networks (in Bayesian Learning for Neural Networks) — Radford M. Neal, 1996 https://scholar.google.com/scholar?q=Priors+for+Infinite+Networks+%28in+Bayesian+Learning+for+Neural+Networks%29 7. Neural Tangent Kernel: Convergence and Generalization in Neural Networks — Arthur Jacot, Franck Gabriel, Clément Hongler, 2018 https://scholar.google.com/scholar?q=Neural+Tangent+Kernel%3A+Convergence+and+Generalization+in+Neural+Networks 8. Fourier Neural Operator for Parametric Partial Differential Equations — Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, 2021 https://scholar.google.com/scholar?q=Fourier+Neural+Operator+for+Parametric+Partial+Differential+Equations 9. Measuring Statistical Dependence with Hilbert-Schmidt Norms — Arthur Gretton, Olivier Bousquet, Alex Smola, Bernhard Schölkopf, 2005 https://scholar.google.com/scholar?q=Measuring+Statistical+Dependence+with+Hilbert-Schmidt+Norms 10. What Do Compressed Deep Neural Networks Forget? — Sara Hooker, Aaron Courville, Gregory Clark, Yann Yannakakis, Kevin Murphy, 2019 https://scholar.google.com/scholar?q=What+Do+Compressed+Deep+Neural+Networks+Forget%3F 11. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson, et al. (Anthropic), 2023 https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+with+Dictionary+Learning 12. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, et al., 2022 https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models 13. Language Modeling Is Compression — Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, et al., 2023 https://scholar.google.com/scholar?q=Language+Modeling+Is+Compression 14. LLM-Pruner: On the Structural Pruning of Large Language Models — Xinyin Ma, Gongfan Fang, Xinchao Wang, 2023 https://scholar.google.com/scholar?q=LLM-Pruner%3A+On+the+Structural+Pruning+of+Large+Language+Models Interactive Visualization: Hilbert Operator Reveals What Networks Actually Learn

  3. 1d ago

    NaturalReasoning: Backtranslating Reasoning Questions at Scale

    This episode explores NaturalReasoning, a Meta and NYU dataset of 2.8 million reasoning questions built to break the bottleneck facing today's verifiable-reward training methods, which only work in domains like math and code where answers can be checked automatically. The discussion traces the technique's lineage from 2016 machine-translation backtranslation through Meta's 2023 instruction-backtranslation work, explaining how the team flags reasoning-rich passages in pretraining corpora and has a strong model work backward to invent the question a given passage would answer — spanning physics, economics, and social science, not just checkable benchmarks. It covers how the resulting question set is used both for straightforward distillation from a teacher model and for a self-rewarding setup where one model generates, verifies, and judges its own answers without external reward models or human labels. Listeners interested in how reasoning-focused LLM training might scale past math and code, and in the mechanics of synthetic data generation via backtranslation, will find the paper's structural argument — rethinking where training data comes from rather than just scaling it — a useful frame for where the field goes next. Sources: 1. NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions — Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason E Weston, Xian Li, 2025 http://arxiv.org/abs/2502.13124 2. Self-Alignment with Instruction Backtranslation — Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Jason Weston, Mike Lewis (Meta AI), 2023 https://scholar.google.com/scholar?q=Self-Alignment+with+Instruction+Backtranslation 3. MAmmoTH2: Scaling Instructions from the Web — Xiang Yue, Tuney Zheng, Ge Zhang, Wenhu Chen, 2024 https://scholar.google.com/scholar?q=MAmmoTH2%3A+Scaling+Instructions+from+the+Web 4. Improving Neural Machine Translation Models with Monolingual Data — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016 https://scholar.google.com/scholar?q=Improving+Neural+Machine+Translation+Models+with+Monolingual+Data 5. LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction — Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze, 2023 https://scholar.google.com/scholar?q=LongForm%3A+Optimizing+Instruction+Tuning+for+Long+Text+Generation+with+Corpus+Extraction 6. Self-Rewarding Language Models — Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston, 2024 https://scholar.google.com/scholar?q=Self-Rewarding+Language+Models 7. Constitutional AI: Harmlessness from AI Feedback — Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. (Anthropic), 2022 https://scholar.google.com/scholar?q=Constitutional+AI%3A+Harmlessness+from+AI+Feedback 8. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Harrison Lee, Samrat Phatale, Hassan Mansoor, et al. (Google), 2023 https://scholar.google.com/scholar?q=RLAIF%3A+Scaling+Reinforcement+Learning+from+Human+Feedback+with+AI+Feedback 9. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. (UC Berkeley / LMSYS), 2023 https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena 10. MAmmoTH2: Scaling Instructions from the Web (WebInstruct) — Xiang Yue, Tuney Zheng, Ge Zhang, Wenhu Chen, 2024 https://scholar.google.com/scholar?q=MAmmoTH2%3A+Scaling+Instructions+from+the+Web+%28WebInstruct%29 11. STaR: Bootstrapping Reasoning with Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah Goodman, 2022 https://scholar.google.com/scholar?q=STaR%3A+Bootstrapping+Reasoning+with+Reasoning 12. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023 https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step 13. OpenThoughts: Data Recipes for Reasoning Models — Etash Guha, Ryan Marten, Sedrick Keh, et al., 2025 https://scholar.google.com/scholar?q=OpenThoughts%3A+Data+Recipes+for+Reasoning+Models Interactive Visualization: NaturalReasoning: Backtranslating Reasoning Questions at Scale

  4. 2d ago

    ASAP: Disaggregating Attention and Experts for Faster MoE Prefill

    This episode explores ASAP, a Huawei-developed serving system for mixture-of-experts models that physically separates the attention and expert computation stages onto different hardware with non-blocking communication between them. The discussion traces the diagnosis behind the design: production traces reveal a "straggler effect" where global synchronization barriers between attention's data-parallel groups and the shared expert pool force every group to wait on the slowest one, and this imbalance is mathematically unavoidable since attention cost scales with the sum of squares of sequence lengths rather than total tokens. The hosts draw a parallel to the classic MapReduce straggler problem, framing this as a structural consequence of combining Expert Parallelism for MoE layers with Data Parallelism for attention rather than something a better scheduler could fix. Listeners get a grounded walkthrough of prefill versus decode, time-to-first-token, and why hybrid parallelism setups like DeepSeek-V3's TP=8/DP=4/EP=32 configuration create this bottleneck in the first place, setting up the paper's disaggregated architecture as a direct response to a precisely characterized failure mode rather than a speculative fix. Sources: 1. ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill — Weiwei Chen, Shuang Chen, Lele Li, Qiang Hu, Han Li, Xin Ye, Ming Yan, Zhibin Yu, 2026 http://arxiv.org/abs/2606.22541 2. Revealing the challenges of attention-ffn disaggregation for modern MoE models and hardware systems — Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, Yanpeng Wang, 2026 https://scholar.google.com/scholar?q=Revealing+the+challenges+of+attention-ffn+disaggregation+for+modern+MoE+models+and+hardware+systems 3. Expert-as-a-Service: Towards efficient, scalable, and robust large-scale MoE serving — Ziming Liu, Boyu Tian, Guoteng Wang, Zhen Jiang, et al., 2025 https://scholar.google.com/scholar?q=Expert-as-a-Service%3A+Towards+efficient%2C+scalable%2C+and+robust+large-scale+MoE+serving 4. MegaScale-Infer — Cited as [55] in ASAP, 2025/2026 https://scholar.google.com/scholar?q=MegaScale-Infer 5. Step-3 — Cited as [44] in ASAP, 2025/2026 https://scholar.google.com/scholar?q=Step-3 6. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbots — Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation+%E2%80%94+a+KVCache-centric+architecture+for+serving+LLM+chatbots 7. Sarathi: Efficient LLM inference by piggybacking decodes with chunked prefills — Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, 2023 https://scholar.google.com/scholar?q=Sarathi%3A+Efficient+LLM+inference+by+piggybacking+decodes+with+chunked+prefills 8. TokenWeave: Efficient compute-communication overlap for distributed LLM inference — Raja Gond, Nipun Kwatra, Ramachandran Ramjee, 2025 https://scholar.google.com/scholar?q=TokenWeave%3A+Efficient+compute-communication+overlap+for+distributed+LLM+inference Interactive Visualization: ASAP: Disaggregating Attention and Experts for Faster MoE Prefill

  5. 2d ago

    Agentic Hardware Design: Evolving RTL Through Repository-Level Code Loops

    This episode examines HORIZON, a system from NVIDIA researchers (Cunxi Yu and colleagues) that treats RTL hardware design as repository-level code evolution, wrapping generation in a git-native loop where an LLM agent edits Verilog, an automated evaluator compiles and simulates each candidate, and only passing changes get committed. The conversation traces the idea's lineage from AlphaEvolve's evolutionary code loops through Yu's own SATLUTION and ABCEvo work, framing HORIZON as the next rung: evolving the hardware artifact itself rather than the tools used to build it. It contrasts HORIZON with two existing approaches to LLM-based RTL generation — domain-tuned one-shot generators like RTLCoder and ChipNeMo, and iterative repair systems like AutoChip and RTLFixer — and explains why hardware's unforgiving, concurrent, tape-out-or-bust nature makes "mostly correct" useless in a way it isn't for software. It also introduces CVDP, a 783-problem benchmark built to stress-test agentic and non-agentic Verilog generation now that older benchmarks are saturating. Listeners interested in whether AI coding-agent techniques can transfer to safety-critical, irreversible engineering domains will find the structural argument here more compelling than the raw benchmark numbers. Sources: 1. Agentic Hardware Design as Repository-Level Code Evolution — Cunxi Yu, Chenhui Deng, Nathaniel Pinckney, Brucek Khailany, 2026 http://arxiv.org/abs/2606.28279v1 2. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Novikov et al. (Google DeepMind), 2025 https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery 3. SATLUTION: scaling repository-level self-evolution to SAT solvers — Yu et al., 2025 https://scholar.google.com/scholar?q=SATLUTION%3A+scaling+repository-level+self-evolution+to+SAT+solvers 4. ABCEvo: self-evolving the ABC logic-synthesis system — Yu et al., 2026 https://scholar.google.com/scholar?q=ABCEvo%3A+self-evolving+the+ABC+logic-synthesis+system 5. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al., 2024 https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering 6. VeriGen: A Large Language Model for Verilog Code Generation — Thakur et al., 2024 https://scholar.google.com/scholar?q=VeriGen%3A+A+Large+Language+Model+for+Verilog+Code+Generation 7. RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation — Liu et al., 2025 https://scholar.google.com/scholar?q=RTLCoder%3A+Fully+Open-Source+and+Efficient+LLM-Assisted+RTL+Code+Generation 8. ChipNeMo: Domain-Adapted LLMs for Chip Design — Liu et al., 2023 https://scholar.google.com/scholar?q=ChipNeMo%3A+Domain-Adapted+LLMs+for+Chip+Design 9. CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations — Liu et al., 2024 https://scholar.google.com/scholar?q=CraftRTL%3A+High-quality+Synthetic+Data+Generation+for+Verilog+Code+Models+with+Correct-by-Construction+Non-Textual+Representations 10. Autonomous Code Evolution Meets NP-Completeness (SATLUTION) — Yu, Liang, Ho, Ren, 2025 https://scholar.google.com/scholar?q=Autonomous+Code+Evolution+Meets+NP-Completeness+%28SATLUTION%29 11. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC (ABCEvo) — Yu, Liang, Ho, Ren, 2026 https://scholar.google.com/scholar?q=Autonomous+Evolution+of+EDA+Tools%3A+Multi-Agent+Self-Evolved+ABC+%28ABCEvo%29 12. Comprehensive Verilog Design Problems (CVDP) — Pinckney, Deng, Ho, Tsai, Liu, Zhou, Khailany, Ren, 2025 https://scholar.google.com/scholar?q=Comprehensive+Verilog+Design+Problems+%28CVDP%29 13. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? / SWE-Bench+ / 'Are Solved Issues in SWE-bench Really Solved Correctly?' — Jimenez et al. (2024); Aleithan et al. (2024); Wang, Pradel, Liu (2026), 2024–2026 https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F+%2F+SWE-Bench%2B+%2F+%27Are+Solved+Issues+in+SWE-bench+Really+Solved+Correctly%3F%27 14. ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs / MAGE: A Multi-Agent Engine for Automated RTL Code Generation — Deng, Yu, Liu, Pinckney, Khailany, Ren (2026); Zhao, Zhang, Huang, Yu, Zhao (2025), 2026 / 2025 https://scholar.google.com/scholar?q=ACE-RTL%3A+When+Agentic+Context+Evolution+Meets+RTL-Specialized+LLMs+%2F+MAGE%3A+A+Multi-Agent+Engine+for+Automated+RTL+Code+Generation

  6. 2d ago

    MegaScale-Infer: Disaggregating Experts for Faster MoE Serving

    This episode explores MegaScale-Infer, a ByteDance Seed/Peking University system for serving large Mixture-of-Experts models more efficiently during inference. The discussion breaks down why decode-phase attention is memory-bandwidth-bound rather than compute-bound, and how MoE's top-k expert routing—while cutting theoretical FLOPs—actually shrinks the effective batch size each expert sees, tanking GPU utilization (illustrated with a concrete Mixtral 8x22B example dropping to 25% utilization). The core proposed fix is disaggregation: physically separating attention and expert computation onto independently scaled GPU pools so attention replicas can pool enough requests to keep experts saturated, building on prior work like DistServe's prefill/decode split. Listeners interested in the gap between algorithmic efficiency claims and real-world GPU serving costs will find the roofline-model analysis a sharp corrective to "sparsity equals free lunch" thinking. Sources: 1. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism — Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, Xin Liu, 2025 http://arxiv.org/abs/2504.02263v1 2. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving 3. Mooncake: Kimi's KVCache-centric Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2024 https://scholar.google.com/scholar?q=Mooncake%3A+Kimi%27s+KVCache-centric+Architecture+for+LLM+Serving 4. DeepSeek-V3 Technical Report — DeepSeek-AI (Aixin Liu et al.), 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 5. MoE-Lightning: High-Throughput MoE Inference on Memory-Constrained GPUs — Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica, 2024 https://scholar.google.com/scholar?q=MoE-Lightning%3A+High-Throughput+MoE+Inference+on+Memory-Constrained+GPUs 6. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020 https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding Interactive Visualization: MegaScale-Infer: Disaggregating Experts for Faster MoE Serving

  7. 3d ago

    Multi-Agent AI Rewrites a Million-Line Chip Design Tool

    This episode explores "Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC," a paper from NVIDIA Research and the University of Maryland examining whether a multi-agent LLM system can autonomously rewrite ABC, the 1.2-million-line, four-layer open-source logic synthesis and verification engine that underlies most open-source ASIC and FPGA toolchains. The discussion traces this work's lineage from DeepMind's AlphaEvolve, which evolved small isolated code kernels, through NVIDIA's own SATLUTION, which scaled the approach to a full SAT solver, and examines why editing a codebase as large and interdependent as ABC — where area, delay, and depth trade off across cross-module dependencies — demands a genuinely different architecture rather than just more iterations of the same loop. That architecture centers on a planning agent coordinating three specialized coding agents for flow tuning, technology mapping, and logic minimization, plus a pre-evolution stage where the system surveys the literature and selects its own scaffolding — including Cunxi Yu's prior FlowTune work and the SLAP mapper — without any heuristics hand-injected by the authors. The conversation digs into why correctness is uniquely non-negotiable here, since a synthesized circuit must be formally equivalent to spec rather than merely probabilistic, making this an unusual case of applying neural methods to edit, rather than replace, one of computing's last hand-engineered, non-learned domains. Listeners interested in AI-driven code evolution, chip design tooling, or the limits of LLM agents on large real-world codebases will find the debate over scale, architecture, and risk especially engaging. Sources: 1. Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC — Cunxi Yu, Haoxing Ren, 2026 http://arxiv.org/abs/2604.15082v1 2. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery — Alexander Novikov, Ngân Vũ, and collaborators (Google DeepMind), 2025 https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+Coding+Agent+for+Scientific+and+Algorithmic+Discovery 3. SATLUTION (LLM-agent self-evolution of a SAT solver codebase) — NVIDIA Research, 2025/2026 https://scholar.google.com/scholar?q=SATLUTION+%28LLM-agent+self-evolution+of+a+SAT+solver+codebase%29 4. DRiLLS: Deep Reinforcement Learning for Logic Synthesis — Abdelrahman Hosny, Soheil Hashemi, Mohamed Shalan, Sherief Reda, 2020 https://scholar.google.com/scholar?q=DRiLLS%3A+Deep+Reinforcement+Learning+for+Logic+Synthesis 5. A Graph Placement Methodology for Fast Chip Design — Azalia Mirhoseini, Anna Goldie, et al. (Google), 2021 https://scholar.google.com/scholar?q=A+Graph+Placement+Methodology+for+Fast+Chip+Design 6. Autonomous Code Evolution Meets NP-Completeness (SATLUTION) — Cunxi Yu, Rongjian Liang, Chia-Tung Ho, Haoxing Ren, 2025 https://scholar.google.com/scholar?q=Autonomous+Code+Evolution+Meets+NP-Completeness+%28SATLUTION%29 7. FunSearch: Making new discoveries in mathematical sciences using large language models — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al. (Google DeepMind), 2024 https://scholar.google.com/scholar?q=FunSearch%3A+Making+new+discoveries+in+mathematical+sciences+using+large+language+models 8. FlowTune: End-to-End Automatic Logic Optimization Exploration via Domain-Specific Multi-Armed Bandit — Walter L. Neto, Yingjie Li, Pierre-Emmanuel Gaillardon, Cunxi Yu, 2023 https://scholar.google.com/scholar?q=FlowTune%3A+End-to-End+Automatic+Logic+Optimization+Exploration+via+Domain-Specific+Multi-Armed+Bandit 9. SLAP: A supervised learning approach for priority cuts technology mapping — Walter Lau Neto, Matheus T. Moreira, Yingjie Li, Luca Amarù, Cunxi Yu, Pierre-Emmanuel Gaillardon, 2021 https://scholar.google.com/scholar?q=SLAP%3A+A+supervised+learning+approach+for+priority+cuts+technology+mapping 10. DAG-aware synthesis orchestration — Yingjie Li, Mingju Liu, Haoxing Ren, Alan Mishchenko, Cunxi Yu, 2024 https://scholar.google.com/scholar?q=DAG-aware+synthesis+orchestration 11. EvoPlace: Evolution of Optimization Algorithms for Global Placement via Large Language Models — Xufeng Yao, Jiaxi Jiang, Yuxuan Zhao, Peiyu Liao, Yibo Lin, Bei Yu, 2026 https://scholar.google.com/scholar?q=EvoPlace%3A+Evolution+of+Optimization+Algorithms+for+Global+Placement+via+Large+Language+Models 12. Agentic AI for Physical Design R&D: Status and Prospects — Amur Ghose, Andrew B. Kahng, Sayak Kundu, Bodhisatta Pramanik, 2026 https://scholar.google.com/scholar?q=Agentic+AI+for+Physical+Design+R%26D%3A+Status+and+Prospects 13. MapTune: Versatile ASIC Technology Mapping via Reinforcement Learning Guided Library Tuning — Mingju Liu, Daniel Robinson, Yingjie Li, Johannes Maximilian Kuehn, Rongjian Liang, Haoxing Ren, Cunxi Yu, 2026 https://scholar.google.com/scholar?q=MapTune%3A+Versatile+ASIC+Technology+Mapping+via+Reinforcement+Learning+Guided+Library+Tuning 14. Machine-Learned Algorithmic Improvement — Ivan Smirnov et al., 2023 https://scholar.google.com/scholar?q=Machine-Learned+Algorithmic+Improvement Interactive Visualization: Multi-Agent AI Rewrites a Million-Line Chip Design Tool

  8. 4d ago

    CacheFlow: Optimal 3D-Parallel KV Cache Restoration

    This episode explores CacheFlow, a system for restoring the KV cache that lets LLM serving systems reload prior context into GPU memory quickly. The discussion centers on a scheduling problem: rather than choosing between recomputing attention states or loading cached tensors from CPU, disk, or another node, CacheFlow treats restoration as a coordination problem across tokens, layers, GPUs, and concurrent requests simultaneously. It highlights a two-pointer "meet in the middle" technique applied along both token chunks and model layers, with an offline-profiled crossover point determining which strategy dominates for a given sequence length. The hosts also stress that the paper proves an optimality bound for its scheduling policy rather than just benchmarking against baselines, distinguishing it from typical serving-infrastructure papers. Listeners interested in reducing time-to-first-token for long-context chatbots, coding agents, or retrieval-heavy pipelines will find the reframing of restoration as a multi-dimensional scheduling problem, rather than a single per-request tradeoff, particularly compelling. Sources: 1. CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration — Sean Nian, Jiahao Fang, Qilong Feng, Zhiyu Wu, Fan Lai, 2026 http://arxiv.org/abs/2604.25080 2. Compute or load KV cache? Why not both? (Cake) — Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao, 2025 https://scholar.google.com/scholar?q=Compute+or+load+KV+cache%3F+Why+not+both%3F+%28Cake%29 3. Fast state restoration in LLM serving with HCache — Shiwei Gao, Youmin Chen, Jiwu Shu, 2025 https://scholar.google.com/scholar?q=Fast+state+restoration+in+LLM+serving+with+HCache 4. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation+%E2%80%94+a+KVCache-centric+architecture+for+serving+LLM+chatbot 5. LMCache: An efficient KV cache layer for enterprise-scale LLM inference — Yuhan Liu, Yihua Cheng, Jiayi Yao, et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+efficient+KV+cache+layer+for+enterprise-scale+LLM+inference 6. Cacheblend: Fast large language model serving for RAG with cached knowledge fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025 https://scholar.google.com/scholar?q=Cacheblend%3A+Fast+large+language+model+serving+for+RAG+with+cached+knowledge+fusion 7. Kvflow: Efficient prefix caching for accelerating LLM-based multi-agent workflows — Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, et al., 2025 https://scholar.google.com/scholar?q=Kvflow%3A+Efficient+prefix+caching+for+accelerating+LLM-based+multi-agent+workflows 8. Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live — Hanchen Li, Qiuyang Mang, Runyuan He, et al., 2026 https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+robust+multi-turn+LLM+agent+scheduling+with+KV+cache+time-to-live Interactive Visualization: CacheFlow: Optimal 3D-Parallel KV Cache Restoration

See All (808)

Ratings & Reviews

3.7
out of 5
3 Ratings

About

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

Information

You Might Also Like