AI Post Transformers

mcgrof

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 1 day ago

    DualDecoder: Fixing GPU Memory Bloat in Long-Context KV Cache Offloading

    This episode covers "DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch," which tackles a hidden cost in dynamic sparse KV-cache systems: the GPU-resident bookkeeping state (landmarks, reconstructed keys) used to make host-memory offloading fast can itself consume up to 64% of GPU memory — 8.5 times larger than the actual sparse KV entries it's meant to retrieve. Drawing on the lineage from H2O's heavy-hitter observation to ShadowKV's landmark-based retrieval, the discussion explains how this auxiliary overhead quietly erodes the memory savings these systems promise, with ShadowKV reaching only 6.7% of its idealized batch-size capacity on a 32-billion-parameter model. DualDecoder's proposed fix is predictive prefetching: rather than permanently parking retrieval-support state on the GPU, it predicts the next decoding step's needs one step ahead and pulls entries from host memory just in time. Listeners interested in LLM inference efficiency will find a concrete, measured account of how a fix for one memory wall can quietly build a smaller one right next to it — and a proposed way out. Sources: 1. DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch — Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, Yuan Cheng, 2026 http://arxiv.org/abs/2607.26475 2. SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs — Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai, 2026 https://scholar.google.com/scholar?q=SpeContext%3A+Enabling+Efficient+Long-context+Reasoning+with+Speculative+Context+Sparsity+in+LLMs 3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023 https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models 4. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024 https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval 5. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu, Heyang Huang, Junhao Hu, et al., 2024 https://scholar.google.com/scholar?q=MemServe%3A+Context+Caching+for+Disaggregated+LLM+Serving+with+Elastic+Memory+Pool 6. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 Interactive Visualization: DualDecoder: Fixing GPU Memory Bloat in Long-Context KV Cache Offloading

  2. 1 day ago

    FreeAct: Rethinking One-to-One Transforms for LLM Quantization

    This episode explores FreeAct, a new approach to quantizing large language models down to 4-bit weights and activations (W4A4), presented by researchers from the National University of Singapore, Huawei Technology, and Central South University. The discussion traces how prior methods like QuaRot and FlatQuant rely on a rigid one-to-one pairing between a rotation matrix applied to activations and its exact inverse applied to weights — an assumption that breaks down for diffusion language models, where masked and unmasked tokens have different statistical profiles, and for multimodal models mixing vision and text tokens through the same layers. The hosts unpack the outlier-channel problem that makes activation quantization so much harder than weight quantization, tracing it back to Dettmers' LLM.int8 findings, and explain how FreeAct exploits a linear-algebra insight — dubbed Proposition 1 — showing that rank-deficient activation matrices allow a whole family of transformations rather than a single exact inverse, enabling different token types to use different activation-side matrices while keeping one shared weight-side transform. It's a compelling listen for anyone tracking how quantization techniques are adapting to increasingly heterogeneous token streams in modern AI systems. Sources: 1. FreeAct: Freeing Activations for LLM Quantization — Xiaohao Liu, Xiaobo Xia, Manyi Zhang, Ji-Fu Li, Xianzhi Yu, Fei Shen, Xiu Su, See-Kiong Ng, Tat-Seng Chua, 2026 http://arxiv.org/abs/2603.01776 2. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale 3. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, Song Han, 2023 https://scholar.google.com/scholar?q=SmoothQuant%3A+Accurate+and+Efficient+Post-Training+Quantization+for+Large+Language+Models 4. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving — Yilong Zhao, Chien-Yu Lin, Kan Zhu, et al., 2024 https://scholar.google.com/scholar?q=Atom%3A+Low-bit+Quantization+for+Efficient+and+Accurate+LLM+Serving 5. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs — Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, et al., 2024 https://scholar.google.com/scholar?q=QuaRot%3A+Outlier-Free+4-Bit+Inference+in+Rotated+LLMs 6. FlatQuant: Flatness Matters for LLM Quantization — Yuxuan Sun, et al., 2025 https://scholar.google.com/scholar?q=FlatQuant%3A+Flatness+Matters+for+LLM+Quantization 7. SpinQuant: LLM Quantization with Learned Rotations — Zechun Liu, Changsheng Zhao, et al. (Meta AI), 2024 https://scholar.google.com/scholar?q=SpinQuant%3A+LLM+Quantization+with+Learned+Rotations 8. QuIP: 2-Bit Quantization of Large Language Models With Guarantees — Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa, 2023 https://scholar.google.com/scholar?q=QuIP%3A+2-Bit+Quantization+of+Large+Language+Models+With+Guarantees 9. MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization — Jiangyong Yu, Sifan Zhou, Dawei Yang, et al., 2025 https://scholar.google.com/scholar?q=MQuant%3A+Unleashing+the+Inference+Potential+of+Multimodal+Large+Language+Models+via+Static+Quantization 10. DLLMQuant: Quantizing Diffusion-based Large Language Models — Chen Xu, Dan Yang, 2025 https://scholar.google.com/scholar?q=DLLMQuant%3A+Quantizing+Diffusion-based+Large+Language+Models 11. Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models — Tianao Zhang, Zhiteng Li, Xianglong Yan, Haotong Qin, Yong Guo, Yulun Zhang, 2025 https://scholar.google.com/scholar?q=Quant-dLLM%3A+Post-Training+Extreme+Low-Bit+Quantization+for+Diffusion+Large+Language+Models 12. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs — Haokun Lin, Haobo Xu, Yichen Wu, et al., 2024 https://scholar.google.com/scholar?q=DuQuant%3A+Distributing+Outliers+via+Dual+Transformation+Makes+Stronger+Quantized+LLMs Interactive Visualization: FreeAct: Rethinking One-to-One Transforms for LLM Quantization

  3. 1 day ago

    Global Memory Bloat in Long-Context LLM Serving

    This episode surveys how large language model serving systems manage the key-value cache — the memory storing every token's key and value vectors — as it has grown from a disposable per-request tensor into a resource actively managed, moved, and contended for across GPUs, nodes, and storage tiers. Drawing on a Texas Tech University paper classifying over thirty existing systems, the hosts unpack the arithmetic behind why KV cache footprint balloons with long context windows (reaching roughly 40 gigabytes for a single 128K-token request on a 70-billion-parameter model) and why bandwidth, not just capacity, becomes the real bottleneck during decode. They trace the field's foundational shift back to PagedAttention, the vLLM technique that introduced OS-style paging for KV memory, and explain how nearly every later system builds on its block-table abstraction. The conversation then turns to a four-dimensional taxonomy — locality, lifetime, ownership, and transport — used to organize the design space, highlighting a striking gap where two of five lifetime categories contain zero real-world systems. Listeners interested in LLM infrastructure, memory hierarchies, or the practical limits of long-context and agentic serving will find a clear framework for reasoning about a problem that's easy to underestimate with a single "the cache grows" intuition. Sources: 1. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving — Jie Li, Tongyang Wang, Yong Chen, 2026 http://arxiv.org/abs/2607.02574 2. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model 3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 5. Medusa / EAGLE speculative decoding work (Cai et al. 2024; Li et al. 2024) — Tianle Cai et al.; Yuhui Li et al., 2024 https://scholar.google.com/scholar?q=Medusa+%2F+EAGLE+speculative+decoding+work+%28Cai+et+al.+2024%3B+Li+et+al.+2024%29 6. S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Ying Sheng et al., 2023/2024 https://scholar.google.com/scholar?q=S-LoRA%3A+Serving+Thousands+of+Concurrent+LoRA+Adapters

  4. 3 days ago

    Adapting Without Forgetting: A Lifelong Learning Roadmap for LLM Agents

    This episode explores "Lifelong Learning of Large Language Model based Agents: A Roadmap," a survey examining how AI agents can continuously adapt to changing environments without losing prior knowledge. The discussion centers on the stability-plasticity dilemma—the tension between preserving learned capabilities and remaining flexible enough to absorb new information—and how this classical problem from connectionist neuroscience resurfaces in a new form for modern agents that rarely fine-tune their underlying weights. Key arguments include the concept of "functional forgetting," where information technically persists in vector stores but becomes practically inaccessible if retrieval or context limits fail to surface it, and a four-part memory taxonomy spanning working, episodic, semantic, and parametric memory. The hosts also trace how this survey synthesizes and extends two separate research lineages—internal-knowledge-focused LLM surveys and agent-architecture surveys—into a unified framework modeled as a goal-conditioned POMDP. Listeners interested in why coding assistants, web-browsing agents, and other AI tools degrade over time as their environments shift will find concrete framing for that problem here. Sources: 1. Lifelong Learning of Large Language Model based Agents: A Roadmap — Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, Qianli Ma, 2025 http://arxiv.org/abs/2501.07278 2. Overcoming Catastrophic Forgetting in Neural Networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, et al. (DeepMind), 2017 https://scholar.google.com/scholar?q=Overcoming+Catastrophic+Forgetting+in+Neural+Networks 3. Continual Lifelong Learning with Neural Networks: A Review — German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, 2019 https://scholar.google.com/scholar?q=Continual+Lifelong+Learning+with+Neural+Networks%3A+A+Review 4. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein (Stanford / Google), 2023 https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior 5. Voyager: An Open-Ended Embodied Agent with Large Language Models — Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (NVIDIA, Caltech, UT Austin), 2023 https://scholar.google.com/scholar?q=Voyager%3A+An+Open-Ended+Embodied+Agent+with+Large+Language+Models 6. Towards Lifelong Learning of Large Language Models: A Survey — J. Zheng, S. Qiu, C. Shi, Q. Ma, 2024 https://scholar.google.com/scholar?q=Towards+Lifelong+Learning+of+Large+Language+Models%3A+A+Survey 7. Loss of Plasticity in Deep Continual Learning — S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, R. S. Sutton, 2024 https://scholar.google.com/scholar?q=Loss+of+Plasticity+in+Deep+Continual+Learning 8. A Survey on Large Language Model Based Autonomous Agents — L. Wang, C. Ma, X. Feng, et al., 2024 https://scholar.google.com/scholar?q=A+Survey+on+Large+Language+Model+Based+Autonomous+Agents 9. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models — P. Wang, Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, F. Huang, H. Chen, 2024 https://scholar.google.com/scholar?q=WISE%3A+Rethinking+the+Knowledge+Memory+for+Lifelong+Model+Editing+of+Large+Language+Models 10. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem — M. McCloskey, N. J. Cohen, 1989 https://scholar.google.com/scholar?q=Catastrophic+Interference+in+Connectionist+Networks%3A+The+Sequential+Learning+Problem Interactive Visualization: Adapting Without Forgetting: A Lifelong Learning Roadmap for LLM Agents

  5. 3 days ago

    Data Temporality's Hidden Impact on LLM Pretraining

    This episode explores why open-weight LLMs like Llama 3.1, Gemma3, Qwen3, and Olmo3 systematically lose 11–39% relative accuracy on facts from 2023–2024 compared to facts from 2020–2021, even though the more recent data falls within their training window. Drawing on Kyutai's paper "Understanding Data Temporality Impact on Large Language Models Pre-training," the discussion traces this "knowledge horizon gap" to a design choice baked into standard pretraining: corpora from many years are pooled and globally shuffled before training, erasing any timestamp signal and letting older, more frequently re-crawled data dominate. The hosts connect this to learning-rate decay schedules, arguing that data seen late in training — when updates are small and durable — gets imprinted far more strongly than data seen early, so chronological ordering (feeding snapshots 2018 through 2025 in sequence) could exploit that same mechanism to anchor recent facts instead of losing them. They situate the work against Zhao et al.'s "Set the Clock" research and Bengio's foundational curriculum-learning ideas, framing chronological training as a strikingly cheap intervention — same tokens, same compute, same architecture — for a problem the field has largely ignored. It's a compelling listen for anyone puzzling over why "knowledge cutoff" claims don't match what models actually seem to know. Sources: 1. Understanding Data Temporality Impact on Large Language Models Pre-training — Hippolyte Pilchen, Romain Fabre, Franck Signe Talla, Patrick Perez, Edouard Grave, 2026 http://arxiv.org/abs/2605.22769 2. Set the Clock: Temporal Alignment of Pretrained Language Models — Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, 2024 https://scholar.google.com/scholar?q=Set+the+Clock%3A+Temporal+Alignment+of+Pretrained+Language+Models 3. Time-Aware Language Models as Temporal Knowledge Bases — Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen, 2022 https://scholar.google.com/scholar?q=Time-Aware+Language+Models+as+Temporal+Knowledge+Bases 4. TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models — Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo, 2022 https://scholar.google.com/scholar?q=TemporalWiki%3A+A+Lifelong+Benchmark+for+Training+and+Evaluating+Ever-Evolving+Language+Models 5. RealTime QA: What's the Answer Right Now? — Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Yutaro Yamada, Deqing Fu, Tushar Khot, Ashish Sabharwal, Rik Koncel-Kedziorski, Yejin Choi, Noah A. Smith, Kentaro Inui, 2022 https://scholar.google.com/scholar?q=RealTime+QA%3A+What%27s+the+Answer+Right+Now%3F 6. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009 https://scholar.google.com/scholar?q=Curriculum+Learning 7. TimeLMs: Diachronic Language Models from Twitter — Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, Jose Camacho-Collados, 2022 https://scholar.google.com/scholar?q=TimeLMs%3A+Diachronic+Language+Models+from+Twitter 8. In-Context Pretraining: Language Modeling Beyond Document Boundaries — Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen-tau Yih, Mike Lewis, 2023 https://scholar.google.com/scholar?q=In-Context+Pretraining%3A+Language+Modeling+Beyond+Document+Boundaries 9. Towards Continual Knowledge Learning of Language Models — Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, Minjoon Seo, 2022 https://scholar.google.com/scholar?q=Towards+Continual+Knowledge+Learning+of+Language+Models 10. TiC-LM: A web-scale benchmark for time-continual LLM pretraining — Li, J., Armandpour, M., Mirzadeh, I., Mehta, S., Shankar, V., Vemulapalli, R., Bengio, S., Tuzel, O., Farajtabar, M., Pouransari, H., Faghri, F., 2025 https://scholar.google.com/scholar?q=TiC-LM%3A+A+web-scale+benchmark+for+time-continual+LLM+pretraining 11. How do language models learn facts? Dynamics, curricula and hallucinations — Zucchet, N., Bornschein, J., Chan, S. C., Lampinen, A. K., Pascanu, R., De, S., 2025 https://scholar.google.com/scholar?q=How+do+language+models+learn+facts%3F+Dynamics%2C+curricula+and+hallucinations 12. Data mixing can induce phase transitions in knowledge acquisition — Gu, X., Lyu, K., Li, J., Zhang, J., 2026 https://scholar.google.com/scholar?q=Data+mixing+can+induce+phase+transitions+in+knowledge+acquisition 13. TiMoE: Time-aware mixture of language experts — Faro, R., Fan, D., Alphaidze, T., Jaggi, M., 2025 https://scholar.google.com/scholar?q=TiMoE%3A+Time-aware+mixture+of+language+experts 14. Does your data spark joy? Performance gains from domain upsampling at the end of training — Blakeney, C., Paul, M., Larsen, B. W., Owen, S., Frankle, J., 2024 https://scholar.google.com/scholar?q=Does+your+data+spark+joy%3F+Performance+gains+from+domain+upsampling+at+the+end+of+training Interactive Visualization: Data Temporality's Hidden Impact on LLM Pretraining

  6. 3 days ago

    Distributed Weight Data Parallelism Cuts LLM Inference Stalls

    This episode explores DWDP (Distributed Weight Data Parallelism), a new NVIDIA-authored approach to LLM inference on NVL72 systems that targets a subtle but costly inefficiency: GPUs sitting idle while they wait to synchronize with slower peers. The hosts unpack how existing model-parallelism strategies—expert, tensor, and pipeline parallelism—all share a hidden flaw, forcing every GPU to hit a synchronization barrier at each layer boundary, which the paper's own baseline shows can waste around twelve percent of total inference time even under ordinary workload imbalance. They explain why smarter scheduling alone (cache-aware or load-aware routing) can't fix this, since it only shrinks the imbalance feeding into the wait rather than eliminating the wait itself. The discussion then turns to DWDP's core idea: keeping GPUs fully data-parallel while having each one asynchronously prefetch missing expert weights from peers on demand, timed to hide the fetch behind ongoing compute. Listeners interested in the mechanics of large-scale MoE inference, GPU synchronization bottlenecks, and practical systems-level solutions to straggler problems will find the technical walkthrough especially rewarding. Sources: 1. DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 — Wanqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang, Kefeng Duan, June Yang, 2026 http://arxiv.org/abs/2604.01621 2. DeepSeek-V3 Technical Report — DeepSeek-AI, Aixin Liu, Bei Feng, et al., 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 3. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot 4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 5. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, et al., 2024 https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting 6. Tutel: Adaptive Mixture-of-Experts at Scale — Changho Hwang, Wei Cui, Yifan Xiong, et al., 2023 https://scholar.google.com/scholar?q=Tutel%3A+Adaptive+Mixture-of-Experts+at+Scale 7. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale — Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al., 2022 https://scholar.google.com/scholar?q=DeepSpeed-MoE%3A+Advancing+Mixture-of-Experts+Inference+and+Training+to+Power+Next-Generation+AI+Scale Interactive Visualization: Distributed Weight Data Parallelism Cuts LLM Inference Stalls

  7. 4 days ago

    AdaJEPA: Self-Adapting Latent World Models via Test-Time MPC

    This episode explores AdaJEPA, an adaptive latent world model that challenges the standard "train once, freeze forever" assumption behind robot planning systems. The hosts trace the technical lineage from Yann LeCun's Joint-Embedding Predictive Architecture concept through model predictive control's decades-old roots in process engineering and rocket landing, showing how these pieces combine to let a deployed robot keep updating its internal model using only the consequences of its own actions — no new labels, demonstrations, or retraining pipeline required. Central to the discussion is how distribution shift causes small prediction errors to compound across multi-step planning horizons, and how test-time adaptation, borrowed from image classification and paralleled to cerebellar motor learning, closes that loop by treating each observed transition as a live training example. The conversation grounds abstract control theory in concrete deployment scenarios, from unfamiliar object shapes to shifting friction and lighting. Listeners interested in robotics, control theory, or self-supervised learning will find a clear walkthrough of why frozen world models fail in the wild and what it means for a model to keep learning after "training" officially ends. Sources: 1. AdaJEPA: An Adaptive Latent World Model — Ying Wang, Oumayma Bounou, Yann LeCun, Mengye Ren, 2026 http://arxiv.org/abs/2606.32026 2. Model Predictive Control: Theory and Practice — A Survey — Carlos E. García, David M. Prett, Manfred Morari, 1989 https://scholar.google.com/scholar?q=Model+Predictive+Control%3A+Theory+and+Practice+%E2%80%94+A+Survey 3. Model Predictive Control: Classical, Robust and Stochastic — Basil Kouvaritakis, Mark Cannon, 2016 https://scholar.google.com/scholar?q=Model+Predictive+Control%3A+Classical%2C+Robust+and+Stochastic 4. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (PETS) — Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine, 2018 https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+in+a+Handful+of+Trials+using+Probabilistic+Dynamics+Models+%28PETS%29 5. Learning Latent Dynamics for Planning from Pixels (PlaNet) — Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson, 2019 https://scholar.google.com/scholar?q=Learning+Latent+Dynamics+for+Planning+from+Pixels+%28PlaNet%29 6. Dino-wm: World models on pre-trained visual features enable zero-shot planning — Zhou, G., Pan, H., LeCun, Y., and Pinto, L., 2025 https://scholar.google.com/scholar?q=Dino-wm%3A+World+models+on+pre-trained+visual+features+enable+zero-shot+planning 7. Temporal straightening for latent planning — Wang, Y., Bounou, O., Zhou, G., Balestriero, R., Rudner, T. G., LeCun, Y., and Ren, M., 2026 https://scholar.google.com/scholar?q=Temporal+straightening+for+latent+planning 8. Closing the train-test gap in world models for gradient-based planning — Parthasarathy, A., Kalra, N., Agrawal, R., LeCun, Y., Bounou, O., Izmailov, P., and Goldblum, M., 2025 https://scholar.google.com/scholar?q=Closing+the+train-test+gap+in+world+models+for+gradient-based+planning 9. Td-mpc2: Scalable, robust world models for continuous control — Hansen, N., Su, H., and Wang, X., 2024 https://scholar.google.com/scholar?q=Td-mpc2%3A+Scalable%2C+robust+world+models+for+continuous+control 10. Adawm: Adaptive world model based planning for autonomous driving — Wang, H., Ye, X., Tao, F., Pan, C., Mallik, A., Yaman, B., Ren, L., and Zhang, J., 2025 https://scholar.google.com/scholar?q=Adawm%3A+Adaptive+world+model+based+planning+for+autonomous+driving 11. Test-time training with self-supervision for generalization under distribution shifts — Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M., 2020 https://scholar.google.com/scholar?q=Test-time+training+with+self-supervision+for+generalization+under+distribution+shifts Interactive Visualization: AdaJEPA: Self-Adapting Latent World Models via Test-Time MPC

  8. 4 days ago

    Cross-Family Speculative Prefill Cuts Long-Context Latency

    This episode explores cross-family speculative prefill, a technique for cutting long-context inference latency by using a small "draft" model to identify which parts of a lengthy prompt matter before a much larger target model processes it. The hosts unpack why this is a hard problem in principle — draft and target models often use completely different tokenizers and architectures, meaning attention-based importance signals shouldn't obviously transfer between them — and trace the lineage from speculative decoding through the original same-family Speculative Prefill work to this paper's cross-family generalization. They highlight the practical motivation: models like DeepSeek and Kimi-K2 have no smaller sibling in their own family, so a technique that only works with matched draft/target pairs is a dead end for real deployments. Key results discussed include an 18x reduction in time-to-first-token, and the episode weighs supporting evidence from prior work on attention sinks against the stronger, less obvious claim that a full salience ranking over a 100,000-token document can transfer across unrelated architectures. Listeners interested in practical LLM efficiency techniques and the mechanics of long-context inference will find the back-and-forth skepticism over whether the method should even work, given the tokenizer mismatch, particularly engaging. Sources: 1. Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models — Shubhangi Upasani, Ravi Shanker Raju, Bo Li, Mengmeng Ji, John Long, Chen Wu, Urmish Thakker, Guangtao Wang, 2026 http://arxiv.org/abs/2603.02631 2. Speculative Prefill — Liu et al., 2025 https://scholar.google.com/scholar?q=Speculative+Prefill 3. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 4. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024) https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks 5. SnapKV: LLM Knows What You Are Looking For Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024 https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation 6. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2023 (EMNLP 2023) https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models 7. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, 2024 (ACL 2024) https://scholar.google.com/scholar?q=LongLLMLingua%3A+Accelerating+and+Enhancing+LLMs+in+Long+Context+Scenarios+via+Prompt+Compression 8. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang, 2024 (ACL Findings 2024) https://scholar.google.com/scholar?q=LLMLingua-2%3A+Data+Distillation+for+Efficient+and+Faithful+Task-Agnostic+Prompt+Compression 9. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023 (NeurIPS 2023) https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens 10. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation — Jingyu Liu, Beidi Chen, Ce Zhang, 2025 https://scholar.google.com/scholar?q=Speculative+Prefill%3A+Turbocharging+TTFT+with+Lightweight+and+Training-Free+Token+Importance+Estimation 11. SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators — Jonathan Li, Nasim Farahini, et al. (SambaNova), 2025 https://scholar.google.com/scholar?q=SnapStream%3A+Efficient+Long+Sequence+Decoding+on+Dataflow+Accelerators 12. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference — Guangtao Wang, Shubhangi Upasani, Chen Wu, et al. (SambaNova), 2025 https://scholar.google.com/scholar?q=LLMs+Know+What+to+Drop%3A+Self-Attention+Guided+KV+Cache+Eviction+for+Efficient+Long-Context+Inference 13. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention — Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, et al., 2024 https://scholar.google.com/scholar?q=MInference+1.0%3A+Accelerating+Pre-filling+for+Long-Context+LLMs+via+Dynamic+Sparse+Attention Interactive Visualization: Cross-Family Speculative Prefill Cuts Long-Context Latency

See All (825)

About

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

Information

You Might Also Like