AI Post Transformers

mcgrof
  • 0.0（0則評分）
  • 科技
  • 每日更新

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 4 天前

    DualDecoder: Fixing GPU Memory Bloat in Long-Context KV Cache Offloading

    This episode covers "DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch," which tackles a hidden cost in dynamic sparse KV-cache systems: the GPU-resident bookkeeping state (landmarks, reconstructed keys) used to make host-memory offloading fast can itself consume up to 64% of GPU memory — 8.5 times larger than the actual sparse KV entries it's meant to retrieve. Drawing on the lineage from H2O's heavy-hitter observation to ShadowKV's landmark-based retrieval, the discussion explains how this auxiliary overhead quietly erodes the memory savings these systems promise, with ShadowKV reaching only 6.7% of its idealized batch-size capacity on a 32-billion-parameter model. DualDecoder's proposed fix is predictive prefetching: rather than permanently parking retrieval-support state on the GPU, it predicts the next decoding step's needs one step ahead and pulls entries from host memory just in time. Listeners interested in LLM inference efficiency will find a concrete, measured account of how a fix for one memory wall can quietly build a smaller one right next to it — and a proposed way out. Sources: 1. DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch — Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, Yuan Cheng, 2026 http://arxiv.org/abs/2607.26475 2. SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs — Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai, 2026 https://scholar.google.com/scholar?q=SpeContext%3A+Enabling+Efficient+Long-context+Reasoning+with+Speculative+Context+Sparsity+in+LLMs 3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023 https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models 4. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024 https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval 5. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu, Heyang Huang, Junhao Hu, et al., 2024 https://scholar.google.com/scholar?q=MemServe%3A+Context+Caching+for+Disaggregated+LLM+Serving+with+Elastic+Memory+Pool 6. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 Interactive Visualization: DualDecoder: Fixing GPU Memory Bloat in Long-Context KV Cache Offloading

  2. 4 天前

    FreeAct: Rethinking One-to-One Transforms for LLM Quantization

    This episode explores FreeAct, a new approach to quantizing large language models down to 4-bit weights and activations (W4A4), presented by researchers from the National University of Singapore, Huawei Technology, and Central South University. The discussion traces how prior methods like QuaRot and FlatQuant rely on a rigid one-to-one pairing between a rotation matrix applied to activations and its exact inverse applied to weights — an assumption that breaks down for diffusion language models, where masked and unmasked tokens have different statistical profiles, and for multimodal models mixing vision and text tokens through the same layers. The hosts unpack the outlier-channel problem that makes activation quantization so much harder than weight quantization, tracing it back to Dettmers' LLM.int8 findings, and explain how FreeAct exploits a linear-algebra insight — dubbed Proposition 1 — showing that rank-deficient activation matrices allow a whole family of transformations rather than a single exact inverse, enabling different token types to use different activation-side matrices while keeping one shared weight-side transform. It's a compelling listen for anyone tracking how quantization techniques are adapting to increasingly heterogeneous token streams in modern AI systems. Sources: 1. FreeAct: Freeing Activations for LLM Quantization — Xiaohao Liu, Xiaobo Xia, Manyi Zhang, Ji-Fu Li, Xianzhi Yu, Fei Shen, Xiu Su, See-Kiong Ng, Tat-Seng Chua, 2026 http://arxiv.org/abs/2603.01776 2. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale 3. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, Song Han, 2023 https://scholar.google.com/scholar?q=SmoothQuant%3A+Accurate+and+Efficient+Post-Training+Quantization+for+Large+Language+Models 4. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving — Yilong Zhao, Chien-Yu Lin, Kan Zhu, et al., 2024 https://scholar.google.com/scholar?q=Atom%3A+Low-bit+Quantization+for+Efficient+and+Accurate+LLM+Serving 5. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs — Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, et al., 2024 https://scholar.google.com/scholar?q=QuaRot%3A+Outlier-Free+4-Bit+Inference+in+Rotated+LLMs 6. FlatQuant: Flatness Matters for LLM Quantization — Yuxuan Sun, et al., 2025 https://scholar.google.com/scholar?q=FlatQuant%3A+Flatness+Matters+for+LLM+Quantization 7. SpinQuant: LLM Quantization with Learned Rotations — Zechun Liu, Changsheng Zhao, et al. (Meta AI), 2024 https://scholar.google.com/scholar?q=SpinQuant%3A+LLM+Quantization+with+Learned+Rotations 8. QuIP: 2-Bit Quantization of Large Language Models With Guarantees — Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa, 2023 https://scholar.google.com/scholar?q=QuIP%3A+2-Bit+Quantization+of+Large+Language+Models+With+Guarantees 9. MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization — Jiangyong Yu, Sifan Zhou, Dawei Yang, et al., 2025 https://scholar.google.com/scholar?q=MQuant%3A+Unleashing+the+Inference+Potential+of+Multimodal+Large+Language+Models+via+Static+Quantization 10. DLLMQuant: Quantizing Diffusion-based Large Language Models — Chen Xu, Dan Yang, 2025 https://scholar.google.com/scholar?q=DLLMQuant%3A+Quantizing+Diffusion-based+Large+Language+Models 11. Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models — Tianao Zhang, Zhiteng Li, Xianglong Yan, Haotong Qin, Yong Guo, Yulun Zhang, 2025 https://scholar.google.com/scholar?q=Quant-dLLM%3A+Post-Training+Extreme+Low-Bit+Quantization+for+Diffusion+Large+Language+Models 12. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs — Haokun Lin, Haobo Xu, Yichen Wu, et al., 2024 https://scholar.google.com/scholar?q=DuQuant%3A+Distributing+Outliers+via+Dual+Transformation+Makes+Stronger+Quantized+LLMs Interactive Visualization: FreeAct: Rethinking One-to-One Transforms for LLM Quantization

  3. 4 天前

    Global Memory Bloat in Long-Context LLM Serving

    This episode surveys how large language model serving systems manage the key-value cache — the memory storing every token's key and value vectors — as it has grown from a disposable per-request tensor into a resource actively managed, moved, and contended for across GPUs, nodes, and storage tiers. Drawing on a Texas Tech University paper classifying over thirty existing systems, the hosts unpack the arithmetic behind why KV cache footprint balloons with long context windows (reaching roughly 40 gigabytes for a single 128K-token request on a 70-billion-parameter model) and why bandwidth, not just capacity, becomes the real bottleneck during decode. They trace the field's foundational shift back to PagedAttention, the vLLM technique that introduced OS-style paging for KV memory, and explain how nearly every later system builds on its block-table abstraction. The conversation then turns to a four-dimensional taxonomy — locality, lifetime, ownership, and transport — used to organize the design space, highlighting a striking gap where two of five lifetime categories contain zero real-world systems. Listeners interested in LLM infrastructure, memory hierarchies, or the practical limits of long-context and agentic serving will find a clear framework for reasoning about a problem that's easy to underestimate with a single "the cache grows" intuition. Sources: 1. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving — Jie Li, Tongyang Wang, Yong Chen, 2026 http://arxiv.org/abs/2607.02574 2. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model 3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 5. Medusa / EAGLE speculative decoding work (Cai et al. 2024; Li et al. 2024) — Tianle Cai et al.; Yuhui Li et al., 2024 https://scholar.google.com/scholar?q=Medusa+%2F+EAGLE+speculative+decoding+work+%28Cai+et+al.+2024%3B+Li+et+al.+2024%29 6. S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Ying Sheng et al., 2023/2024 https://scholar.google.com/scholar?q=S-LoRA%3A+Serving+Thousands+of+Concurrent+LoRA+Adapters Interactive Visualization: Global Memory Bloat in Long-Context LLM Serving

  4. 4 天前

    Making Every Verified Token Count in MoE Speculative Decoding

    This episode explores adaptive verification for speculative decoding when the target model is a sparse Mixture-of-Experts (MoE) system rather than a dense transformer, focusing on the paper "Making Every Verified Token Count." The discussion traces the lineage from Leviathan et al.'s original speculative decoding through tree-based drafting methods like Medusa and EAGLE-3, then explains why MoE architectures break a core assumption: since different draft-tree branches can route to entirely different experts, verifying a tree means loading every expert any branch touched. Drawing on the paper's benchmarks across three MoE models (including Qwen3-30B-A3B), the hosts unpack the striking finding that verification alone consumes 79-89% of per-iteration decoding latency once trees grow past thirty nodes — flipping the "verification is nearly free" pitch that made speculative decoding attractive in the first place. Listeners interested in LLM inference serving, GPU memory-bandwidth bottlenecks, or the practical tradeoffs of deploying sparse MoE models will find the episode's breakdown of why dense-model intuition fails on MoE targets especially clarifying. Sources: 1. Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding — Lehan Pan, Ziyang Tao, Ruoyu Pang, Xiao Wang, Jianjun Zhao, Yanyong Zhang, 2026 http://arxiv.org/abs/2605.00342 2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 3. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao, 2024 https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 4. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 5. Mixtral of Experts — Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, and the Mistral AI team, 2024 https://scholar.google.com/scholar?q=Mixtral+of+Experts 6. Utility-driven speculative decoding for mixture-of-experts — Anish Saxena, Po-An Tsai, Hritvik Taneja, Aamer Jaleel, Moinuddin Qureshi, 2025 https://scholar.google.com/scholar?q=Utility-driven+speculative+decoding+for+mixture-of-experts 7. MoE-Spec: Expert budgeting for efficient speculative decoding — Bradley McDanel, Steven Li, Sruthikesh Surineni, Harshit Khaitan, 2026 https://scholar.google.com/scholar?q=MoE-Spec%3A+Expert+budgeting+for+efficient+speculative+decoding 8. ECHO: Elastic speculative decoding with sparse gating for high-concurrency scenarios — Xinyi Hu, Yuhao Shen, Baolin Zhang, Hengxin Zhang, Jun Dai, Shuang Ge, Lei Chen, Yue Li, Mingcheng Wan, 2026 https://scholar.google.com/scholar?q=ECHO%3A+Elastic+speculative+decoding+with+sparse+gating+for+high-concurrency+scenarios 9. MoESD: Unveil speculative decoding's potential for accelerating sparse MoE — Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang, 2025 https://scholar.google.com/scholar?q=MoESD%3A+Unveil+speculative+decoding%27s+potential+for+accelerating+sparse+MoE Interactive Visualization: Making Every Verified Token Count in MoE Speculative Decoding

  5. 4 天前

    Model Predictive Control's Real-Time Structure, from Chapter to Cockpit

    Sitting down with a two-author-plus-one chemical-engineering-rooted textbook on Model Predictive Control, this episode unpacks why the field treats real-time feasibility as a hard constraint rather than a nice-to-have — walking through how MPC re-solves an optimization problem from scratch every control cycle using a known dynamics model, with no learning or reward signal involved. The discussion centers on the structural trick that makes this tractable on embedded hardware: exploiting the block-banded, time-local coupling of the problem via Riccati recursion or condensing to cut a naive O(N³) solve down to O(N), and how Diehl, Bock, and Schlöder's 2005 real-time iteration scheme turned this from a lab curiosity into something a drone or engine controller can rerun dozens of times per second. It also covers moving horizon estimation as the optimization-based counterpart to the Kalman filter, explaining why MHE can enforce physical constraints a Kalman filter can't, and why the book cuts particle filtering from its main text once state dimensionality climbs past five. Listeners get a clear picture of why the same machinery underlies powered-descent guidance, automotive control, and legged robotics — not as a trend, but as the only approach that reliably meets millisecond-scale deadlines. Sources: 1. Model Predictive Control's Real-Time Structure, from Chapter to Cockpit https://sites.engineering.ucsb.edu/~jbraw/mpc/MPC-book-2nd-edition-4th-printing.pdf 2. A Real-Time Iteration Scheme for Nonlinear Optimization in Optimal Feedback Control — Moritz Diehl, Hans Georg Bock, Johannes P. Schlöder, 2005 https://scholar.google.com/scholar?q=A+Real-Time+Iteration+Scheme+for+Nonlinear+Optimization+in+Optimal+Feedback+Control 3. CasADi: A Software Framework for Nonlinear Optimization and Optimal Control — Joel A. E. Andersson, Joris Gillis, Greg Horn, James B. Rawlings, Moritz Diehl, 2019 https://scholar.google.com/scholar?q=CasADi%3A+A+Software+Framework+for+Nonlinear+Optimization+and+Optimal+Control 4. acados: A Modular Open-Source Framework for Fast Embedded Optimal Control — Robin Verschueren, Gianluca Frison, Dimitris Kouzoupis, Jonathan Frey, Niels van Duijkeren, Andrea Zanelli, Branimir Novoselnik, Thivaharan Albin, Rien Quirynen, Moritz Diehl, 2022 https://scholar.google.com/scholar?q=acados%3A+A+Modular+Open-Source+Framework+for+Fast+Embedded+Optimal+Control 5. On the Implementation of an Interior-Point Filter Line-Search Algorithm for Large-Scale Nonlinear Programming — Andreas Wächter, Lorenz T. Biegler, 2006 https://scholar.google.com/scholar?q=On+the+Implementation+of+an+Interior-Point+Filter+Line-Search+Algorithm+for+Large-Scale+Nonlinear+Programming 6. Constrained Linear State Estimation — A Moving Horizon Approach — Christopher V. Rao, James B. Rawlings, Jay H. Lee, 2001 https://scholar.google.com/scholar?q=Constrained+Linear+State+Estimation+%25E2%2580%2594+A+Moving+Horizon+Approach 7. Constrained State Estimation for Nonlinear Discrete-Time Systems: Stability and Moving Horizon Approximations — Christopher V. Rao, James B. Rawlings, David Q. Mayne, 2003 https://scholar.google.com/scholar?q=Constrained+State+Estimation+for+Nonlinear+Discrete-Time+Systems%3A+Stability+and+Moving+Horizon+Approximations 8. Moving-Horizon State Estimation for Nonlinear Discrete-Time Systems: New Stability Results and Approximation Schemes — Angelo Alessandri, Marco Baglietto, Giorgio Battistelli, 2008 https://scholar.google.com/scholar?q=Moving-Horizon+State+Estimation+for+Nonlinear+Discrete-Time+Systems%3A+New+Stability+Results+and+Approximation+Schemes 9. Stochastic Model Predictive Control: An Overview and Perspectives for Future Research — Ali Mesbah, 2016 https://scholar.google.com/scholar?q=Stochastic+Model+Predictive+Control%3A+An+Overview+and+Perspectives+for+Future+Research 10. Stochastic Linear Model Predictive Control with Chance Constraints — A Review — Marcello Farina, Luca Giulioni, Riccardo Scattolini, 2016 https://scholar.google.com/scholar?q=Stochastic+Linear+Model+Predictive+Control+with+Chance+Constraints+%25E2%2580%2594+A+Review 11. Learning-Based Model Predictive Control: Toward Safe Learning in Control — Lukas Hewing, Kim P. Wabersich, Marcel Menner, Melanie N. Zeilinger, 2020 https://scholar.google.com/scholar?q=Learning-Based+Model+Predictive+Control%3A+Toward+Safe+Learning+in+Control 12. Architectures for Distributed and Hierarchical Model Predictive Control — A Review — Riccardo Scattolini, 2009 https://scholar.google.com/scholar?q=Architectures+for+Distributed+and+Hierarchical+Model+Predictive+Control+%25E2%2580%2594+A+Review 13. Distributed MPC Strategies with Application to Power System Automatic Generation Control — Aswin N. Venkat, Ian A. Hiskens, James B. Rawlings, Stephen J. Wright, 2008 https://scholar.google.com/scholar?q=Distributed+MPC+Strategies+with+Application+to+Power+System+Automatic+Generation+Control 14. Distributed Model Predictive Control: A Tutorial Review and Future Research Directions — Panagiotis D. Christofides, Riccardo Scattolini, David Muñoz de la Peña, Jinfeng Liu, 2013 https://scholar.google.com/scholar?q=Distributed+Model+Predictive+Control%3A+A+Tutorial+Review+and+Future+Research+Directions 15. acados — a modular open-source framework for fast embedded optimal control — R. Verschueren, G. Frison, D. Kouzoupis, N. van Duijkeren, A. Zanelli, B. Novoselnik, T. Albin, R. Quirynen, M. Diehl, 2022 https://scholar.google.com/scholar?q=acados+%25E2%2580%2594+a+modular+open-source+framework+for+fast+embedded+optimal+control 16. The scenario approach to robust control design — G.C. Calafiore, M.C. Campi, 2006 https://scholar.google.com/scholar?q=The+scenario+approach+to+robust+control+design 17. Stability of nonstationary receding horizon control (foundational stability result for stochastic MPC) — D. Chatterjee, J. Lygeros, 2015 https://scholar.google.com/scholar?q=Stability+of+nonstationary+receding+horizon+control+%28foundational+stability+result+for+stochastic+MPC%29 18. Robust MPC and dissipativity-based analysis for stochastic constrained systems (source of Assumption 3.22, stochastic MPC Version 2) — D.Q. Mayne, P. Falugi, 2019 https://scholar.google.com/scholar?q=Robust+MPC+and+dissipativity-based+analysis+for+stochastic+constrained+systems+%28source+of+Assumption+3.22%2C+stochastic+MPC+Version+2%29 19. A model predictive control framework for industrial turbodiesel engine control (source system for the nonlinear distributed MPC example) — B.T. Stewart, A.N. Venkat, J.B. Rawlings, S.J. Wright, G. Pannocchia (2011 IEEE CDC paper referenced as Stewart et al. 2011), 2011 https://scholar.google.com/scholar?q=A+model+predictive+control+framework+for+industrial+turbodiesel+engine+control+%28source+system+for+the+nonlinear+distributed+MPC+example%29 Interactive Visualization: Model Predictive Control's Real-Time Structure, from Chapter to Cockpit

  6. 6 天前

    Adapting Without Forgetting: A Lifelong Learning Roadmap for LLM Agents

    This episode explores "Lifelong Learning of Large Language Model based Agents: A Roadmap," a survey examining how AI agents can continuously adapt to changing environments without losing prior knowledge. The discussion centers on the stability-plasticity dilemma—the tension between preserving learned capabilities and remaining flexible enough to absorb new information—and how this classical problem from connectionist neuroscience resurfaces in a new form for modern agents that rarely fine-tune their underlying weights. Key arguments include the concept of "functional forgetting," where information technically persists in vector stores but becomes practically inaccessible if retrieval or context limits fail to surface it, and a four-part memory taxonomy spanning working, episodic, semantic, and parametric memory. The hosts also trace how this survey synthesizes and extends two separate research lineages—internal-knowledge-focused LLM surveys and agent-architecture surveys—into a unified framework modeled as a goal-conditioned POMDP. Listeners interested in why coding assistants, web-browsing agents, and other AI tools degrade over time as their environments shift will find concrete framing for that problem here. Sources: 1. Lifelong Learning of Large Language Model based Agents: A Roadmap — Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, Qianli Ma, 2025 http://arxiv.org/abs/2501.07278 2. Overcoming Catastrophic Forgetting in Neural Networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, et al. (DeepMind), 2017 https://scholar.google.com/scholar?q=Overcoming+Catastrophic+Forgetting+in+Neural+Networks 3. Continual Lifelong Learning with Neural Networks: A Review — German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, 2019 https://scholar.google.com/scholar?q=Continual+Lifelong+Learning+with+Neural+Networks%3A+A+Review 4. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein (Stanford / Google), 2023 https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior 5. Voyager: An Open-Ended Embodied Agent with Large Language Models — Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar (NVIDIA, Caltech, UT Austin), 2023 https://scholar.google.com/scholar?q=Voyager%3A+An+Open-Ended+Embodied+Agent+with+Large+Language+Models 6. Towards Lifelong Learning of Large Language Models: A Survey — J. Zheng, S. Qiu, C. Shi, Q. Ma, 2024 https://scholar.google.com/scholar?q=Towards+Lifelong+Learning+of+Large+Language+Models%3A+A+Survey 7. Loss of Plasticity in Deep Continual Learning — S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, R. S. Sutton, 2024 https://scholar.google.com/scholar?q=Loss+of+Plasticity+in+Deep+Continual+Learning 8. A Survey on Large Language Model Based Autonomous Agents — L. Wang, C. Ma, X. Feng, et al., 2024 https://scholar.google.com/scholar?q=A+Survey+on+Large+Language+Model+Based+Autonomous+Agents 9. WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models — P. Wang, Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, F. Huang, H. Chen, 2024 https://scholar.google.com/scholar?q=WISE%3A+Rethinking+the+Knowledge+Memory+for+Lifelong+Model+Editing+of+Large+Language+Models 10. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem — M. McCloskey, N. J. Cohen, 1989 https://scholar.google.com/scholar?q=Catastrophic+Interference+in+Connectionist+Networks%3A+The+Sequential+Learning+Problem Interactive Visualization: Adapting Without Forgetting: A Lifelong Learning Roadmap for LLM Agents

  7. 6 天前

    Data Temporality's Hidden Impact on LLM Pretraining

    This episode explores why open-weight LLMs like Llama 3.1, Gemma3, Qwen3, and Olmo3 systematically lose 11–39% relative accuracy on facts from 2023–2024 compared to facts from 2020–2021, even though the more recent data falls within their training window. Drawing on Kyutai's paper "Understanding Data Temporality Impact on Large Language Models Pre-training," the discussion traces this "knowledge horizon gap" to a design choice baked into standard pretraining: corpora from many years are pooled and globally shuffled before training, erasing any timestamp signal and letting older, more frequently re-crawled data dominate. The hosts connect this to learning-rate decay schedules, arguing that data seen late in training — when updates are small and durable — gets imprinted far more strongly than data seen early, so chronological ordering (feeding snapshots 2018 through 2025 in sequence) could exploit that same mechanism to anchor recent facts instead of losing them. They situate the work against Zhao et al.'s "Set the Clock" research and Bengio's foundational curriculum-learning ideas, framing chronological training as a strikingly cheap intervention — same tokens, same compute, same architecture — for a problem the field has largely ignored. It's a compelling listen for anyone puzzling over why "knowledge cutoff" claims don't match what models actually seem to know. Sources: 1. Understanding Data Temporality Impact on Large Language Models Pre-training — Hippolyte Pilchen, Romain Fabre, Franck Signe Talla, Patrick Perez, Edouard Grave, 2026 http://arxiv.org/abs/2605.22769 2. Set the Clock: Temporal Alignment of Pretrained Language Models — Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, 2024 https://scholar.google.com/scholar?q=Set+the+Clock%3A+Temporal+Alignment+of+Pretrained+Language+Models 3. Time-Aware Language Models as Temporal Knowledge Bases — Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen, 2022 https://scholar.google.com/scholar?q=Time-Aware+Language+Models+as+Temporal+Knowledge+Bases 4. TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models — Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo, 2022 https://scholar.google.com/scholar?q=TemporalWiki%3A+A+Lifelong+Benchmark+for+Training+and+Evaluating+Ever-Evolving+Language+Models 5. RealTime QA: What's the Answer Right Now? — Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Yutaro Yamada, Deqing Fu, Tushar Khot, Ashish Sabharwal, Rik Koncel-Kedziorski, Yejin Choi, Noah A. Smith, Kentaro Inui, 2022 https://scholar.google.com/scholar?q=RealTime+QA%3A+What%27s+the+Answer+Right+Now%3F 6. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009 https://scholar.google.com/scholar?q=Curriculum+Learning 7. TimeLMs: Diachronic Language Models from Twitter — Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, Jose Camacho-Collados, 2022 https://scholar.google.com/scholar?q=TimeLMs%3A+Diachronic+Language+Models+from+Twitter 8. In-Context Pretraining: Language Modeling Beyond Document Boundaries — Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen-tau Yih, Mike Lewis, 2023 https://scholar.google.com/scholar?q=In-Context+Pretraining%3A+Language+Modeling+Beyond+Document+Boundaries 9. Towards Continual Knowledge Learning of Language Models — Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, Minjoon Seo, 2022 https://scholar.google.com/scholar?q=Towards+Continual+Knowledge+Learning+of+Language+Models 10. TiC-LM: A web-scale benchmark for time-continual LLM pretraining — Li, J., Armandpour, M., Mirzadeh, I., Mehta, S., Shankar, V., Vemulapalli, R., Bengio, S., Tuzel, O., Farajtabar, M., Pouransari, H., Faghri, F., 2025 https://scholar.google.com/scholar?q=TiC-LM%3A+A+web-scale+benchmark+for+time-continual+LLM+pretraining 11. How do language models learn facts? Dynamics, curricula and hallucinations — Zucchet, N., Bornschein, J., Chan, S. C., Lampinen, A. K., Pascanu, R., De, S., 2025 https://scholar.google.com/scholar?q=How+do+language+models+learn+facts%3F+Dynamics%2C+curricula+and+hallucinations 12. Data mixing can induce phase transitions in knowledge acquisition — Gu, X., Lyu, K., Li, J., Zhang, J., 2026 https://scholar.google.com/scholar?q=Data+mixing+can+induce+phase+transitions+in+knowledge+acquisition 13. TiMoE: Time-aware mixture of language experts — Faro, R., Fan, D., Alphaidze, T., Jaggi, M., 2025 https://scholar.google.com/scholar?q=TiMoE%3A+Time-aware+mixture+of+language+experts 14. Does your data spark joy? Performance gains from domain upsampling at the end of training — Blakeney, C., Paul, M., Larsen, B. W., Owen, S., Frankle, J., 2024 https://scholar.google.com/scholar?q=Does+your+data+spark+joy%3F+Performance+gains+from+domain+upsampling+at+the+end+of+training Interactive Visualization: Data Temporality's Hidden Impact on LLM Pretraining

  8. 6 天前

    Distributed Weight Data Parallelism Cuts LLM Inference Stalls

    This episode explores DWDP (Distributed Weight Data Parallelism), a new NVIDIA-authored approach to LLM inference on NVL72 systems that targets a subtle but costly inefficiency: GPUs sitting idle while they wait to synchronize with slower peers. The hosts unpack how existing model-parallelism strategies—expert, tensor, and pipeline parallelism—all share a hidden flaw, forcing every GPU to hit a synchronization barrier at each layer boundary, which the paper's own baseline shows can waste around twelve percent of total inference time even under ordinary workload imbalance. They explain why smarter scheduling alone (cache-aware or load-aware routing) can't fix this, since it only shrinks the imbalance feeding into the wait rather than eliminating the wait itself. The discussion then turns to DWDP's core idea: keeping GPUs fully data-parallel while having each one asynchronously prefetch missing expert weights from peers on demand, timed to hide the fetch behind ongoing compute. Listeners interested in the mechanics of large-scale MoE inference, GPU synchronization bottlenecks, and practical systems-level solutions to straggler problems will find the technical walkthrough especially rewarding. Sources: 1. DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 — Wanqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang, Kefeng Duan, June Yang, 2026 http://arxiv.org/abs/2604.01621 2. DeepSeek-V3 Technical Report — DeepSeek-AI, Aixin Liu, Bei Feng, et al., 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 3. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot 4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 5. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, et al., 2024 https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting 6. Tutel: Adaptive Mixture-of-Experts at Scale — Changho Hwang, Wei Cui, Yifan Xiong, et al., 2023 https://scholar.google.com/scholar?q=Tutel%3A+Adaptive+Mixture-of-Experts+at+Scale 7. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale — Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al., 2022 https://scholar.google.com/scholar?q=DeepSpeed-MoE%3A+Advancing+Mixture-of-Experts+Inference+and+Training+to+Power+Next-Generation+AI+Scale Interactive Visualization: Distributed Weight Data Parallelism Cuts LLM Inference Stalls

顯示全部 (827)

簡介

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

資訊

  • 創作者
    mcgrof
  • 活躍年代
    2025年 - 2026年
  • 集數
    827
  • 年齡分級
    兒少適宜
  • 節目網站
    AI Post Transformers

你可能也會喜歡