AI Post Transformers

mcgrof
  • 3.7 (3)
  • Technology
  • Daily

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 1d ago

    Context Compaction Cuts Costs: "You Don't Need Long Context

    This episode examines a November 2026 NVIDIA paper arguing that long context windows are largely unnecessary, because later tokens cost far more to generate than early ones. Measured H100 throughput on open-weight models puts serving a 256K window at 3.6 to 7 times the cost of a 16K window. The discussion separates the paper's three kinds of evidence: measured serving throughput, benchmark accuracy results, and dollar savings from a fitted "estimated optimized retail cost" curve, which is a synthetic price and not a real bill. It walks through the three agent regimes: full-history accumulation, compaction at a token threshold, and single-action compaction. It also covers the proposed Semantic State Space Model, which moves a Mamba-style recurrence outside the network into readable text state. The hosts note that the SSSM is only outlined, not trained. They also note the paper doesn't cite MemGPT, its closest conceptual ancestor. Listeners get a clear guide to which of the paper's cost-cutting claims are measured and which are modeled. Sources: 1. You don't need long context — Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov, 2026 http://arxiv.org/abs/2611.05306 2. Efficiently Modeling Long Sequences with Structured State Spaces (S4) — Albert Gu, Karan Goel, Christopher Ré, 2021 (ICLR 2022) https://scholar.google.com/scholar?q=Efficiently+Modeling+Long+Sequences+with+Structured+State+Spaces+%28S4%29 3. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023 https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces 4. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 5. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024) https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts 6. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024 https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F 7. Recurrent Memory Transformer / Scaling Transformer to 1M tokens and beyond with RMT — Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev, 2022-2023 https://scholar.google.com/scholar?q=Recurrent+Memory+Transformer+%2F+Scaling+Transformer+to+1M+tokens+and+beyond+with+RMT 8. Mamba: Linear-Time Sequence Modeling with Selective State Spaces; Mamba-2 / Transformers are SSMs — Albert Gu, Tri Dao, 2023-2024 https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces%3B+Mamba-2+%2F+Transformers+are+SSMs 9. Repeat After Me: Transformers are Better than State Space Models at Copying — Samy Jelassi, David Brandfonbrener, Sham Kakade, Eran Malach, 2024 https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying 10. Context Rot / long-context degradation studies (e.g., Chroma context-rot report; NoLiMa: Long-Context Evaluation Beyond Literal Matching) — Chroma research; Ali Modarressi et al., 2025 https://scholar.google.com/scholar?q=Context+Rot+%2F+long-context+degradation+studies+%28e.g.%2C+Chroma+context-rot+report%3B+NoLiMa%3A+Long-Context+Evaluation+Beyond+Literal+Matching%29 11. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 12. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 13. Effective context engineering for AI agents / Claude Code auto-compact; Cognition 'Don't build multi-agents' context engineering notes — Anthropic; Cognition, 2025 https://scholar.google.com/scholar?q=Effective+context+engineering+for+AI+agents+%2F+Claude+Code+auto-compact%3B+Cognition+%27Don%27t+build+multi-agents%27+context+engineering+notes 14. Ring Attention, Infini-attention, and Transformer-XL style recurrence — Hao Liu et al.; Tsendsuren Munkhdalai et al.; Zihang Dai et al., 2019-2024 https://scholar.google.com/scholar?q=Ring+Attention%2C+Infini-attention%2C+and+Transformer-XL+style+recurrence 15. Recursive Language Models / agentic reading of long inputs through code and REPL tools — Alex Zhang, Omar Khattab and others, 2025 https://scholar.google.com/scholar?q=Recursive+Language+Models+%2F+agentic+reading+of+long+inputs+through+code+and+REPL+tools 16. On the Computational Complexity of Self-Attention — Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, Chinmay Hegde, 2023 https://scholar.google.com/scholar?q=On+the+Computational+Complexity+of+Self-Attention Interactive Visualization: Context Compaction Cuts Costs: "You Don't Need Long Context

  2. 1d ago

    FastKVComp: Checking Whether High-Throughput KV Compression Pays Off

    This episode examines FastKVComp, a system that speeds up KV-cache compression for disaggregated LLM serving, where the cache built during prefill must cross a 100 to 400 gigabit link to the decode GPU. It lays out the break-even rule that compression pays only if encode plus decode time is less than the wire time saved. The compression pipeline is a randomized Hadamard transform, low-bit quantization and a lossless codec. Only quantization loses information, and the paper reports no accuracy numbers. The speedups come from fusing the transform and quantization into one tiled kernel that avoids a full-size intermediate tensor, and from replacing nvCOMP ANS with the LC codec. LC compresses less but runs faster. The discussion questions the headline claims of a 5.9x faster kernel, over 500 GB/s throughput and 2.04x better job completion time. It notes that the throughput unit is undefined and appears to be raw input bytes, and that the baseline comes from the same research group. Listeners get a practical way to check whether a compression speedup holds up on fast networks. Sources: 1. Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving — Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao, 2026 http://arxiv.org/abs/2611.04382 2. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 3. Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs (LC framework line of work) — Noushin Azami, Alex Fallin, Martin Burtscher (Texas State University), 2025 https://scholar.google.com/scholar?q=Efficient+Lossless+Compression+of+Scientific+Floating-Point+Data+on+CPUs+and+GPUs+%28LC+framework+line+of+work%29 4. DFloat11: Lossless Compression of LLMs for Efficient GPU Inference (and ZipNN: Lossless Compression for AI Models) — DFloat11: Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava. ZipNN: Moshik Hershcovitch et al. (IBM), 2024–2025 https://scholar.google.com/scholar?q=DFloat11%3A+Lossless+Compression+of+LLMs+for+Efficient+GPU+Inference+%28and+ZipNN%3A+Lossless+Compression+for+AI+Models%29 5. nvCOMP / GPU lossless codecs (Bitcomp, GDeflate, ANS) and the cuSZ/FZ-GPU line from the same group as KVServe — NVIDIA nvCOMP team; cuSZ/FZ-GPU: Dingwen Tao and collaborators, 2020–2024 https://scholar.google.com/scholar?q=nvCOMP+%2F+GPU+lossless+codecs+%28Bitcomp%2C+GDeflate%2C+ANS%29+and+the+cuSZ%2FFZ-GPU+line+from+the+same+group+as+KVServe 6. SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving — Guo, Y., Joshi, S., 2026 https://scholar.google.com/scholar?q=SplitZip%3A+Ultra+fast+lossless+KV+compression+for+disaggregated+LLM+serving 7. KVServe: Service-aware KV cache compression for communication-efficient disaggregated LLM serving — Liu, Z., Ma, X., Luo, D., et al., 2026 https://scholar.google.com/scholar?q=KVServe%3A+Service-aware+KV+cache+compression+for+communication-efficient+disaggregated+LLM+serving 8. Efficient lossless compression of scientific floating-point data on CPUs and GPUs (LC / FZ-style pipelines) — Azami, N., Fallin, A., Burtscher, M., 2025 https://scholar.google.com/scholar?q=Efficient+lossless+compression+of+scientific+floating-point+data+on+CPUs+and+GPUs+%28LC+%2F+FZ-style+pipelines%29 9. NVIDIA nvCOMP (ANS, Bitcomp, GDeflate, LZ4) documentation and benchmarks — NVIDIA, 2023-2026 https://scholar.google.com/scholar?q=NVIDIA+nvCOMP+%28ANS%2C+Bitcomp%2C+GDeflate%2C+LZ4%29+documentation+and+benchmarks 10. Lossless compression of BF16 weights: DFloat11 and ZipNN — Zhang et al. (DFloat11); Hershcovich et al. (ZipNN), 2025 https://scholar.google.com/scholar?q=Lossless+compression+of+BF16+weights%3A+DFloat11+and+ZipNN 11. Mooncake: Trading more storage for less computation — Qin, R., et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation 12. KVQuant and KIVI — Hooper et al. 2024; Liu et al. 2024, 2024 https://scholar.google.com/scholar?q=KVQuant+and+KIVI Interactive Visualization: FastKVComp: Checking Whether High-Throughput KV Compression Pays Off

  3. 1d ago

    Nemotron-Labs-Diffusion: One Model, Three Decoding Modes

    This episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel decoding, and self-speculation. In self-speculation, the diffusion pathway drafts tokens and the causal pathway verifies them, so no separate drafter is needed. It first explains why single-user decoding is memory-bound, then covers the prior approaches: masked and block diffusion, speculative decoding, and multi-token-prediction and Eagle3 drafters. A central theme is that the headline claims mix different metrics. Tokens per forward pass, acceptance rate and real hardware throughput are not interchangeable. A self-speculation cycle costs two forward passes, so true tokens per pass is roughly half the raw acceptance count. The episode also covers the training recipe, which uses a noisy block-bidirectional stream alongside a strictly causal clean stream. It notes that the "speed-of-light" analysis is an oracle upper bound, not a deployable sampler. Listeners get a clear framework for reading speedup claims in LLM inference papers critically. Sources: 1. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, Pavlo Molchanov, 2026 http://arxiv.org/abs/2607.05722 2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (ICML 2023) https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (and the earlier EAGLE / EAGLE-2) — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024-2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test+%28and+the+earlier+EAGLE+%2F+EAGLE-2%29 4. Large Language Diffusion Models (LLaDA) — Shen Nie, Fengqi Zhu, et al., 2025 https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models+%28LLaDA%29 5. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LMs) — Marianne Arriola, Subham Sekhar Sahoo, et al., 2025 (ICLR 2025) https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LMs%29 6. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, et al. (Meta FAIR), 2024 https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction 7. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li et al., 2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 8. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM) — Marianne Arriola et al., 2025 https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LM%29 9. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation — Shuang Cheng et al., 2025 https://scholar.google.com/scholar?q=SDAR%3A+A+Synergistic+Diffusion-AutoRegression+Paradigm+for+Scalable+Sequence+Generation 10. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding — Chengyue Wu et al., 2025 https://scholar.google.com/scholar?q=Fast-dLLM%3A+Training-free+Acceleration+of+Diffusion+LLM+by+Enabling+KV+Cache+and+Parallel+Decoding 11. Dream 7B: Diffusion Large Language Models — Jiacheng Ye et al., 2025 https://scholar.google.com/scholar?q=Dream+7B%3A+Diffusion+Large+Language+Models 12. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai et al., 2024 https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 13. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 14. Self Speculative Decoding / Draft & Verify (layer skipping) — Jun Zhang et al., 2023 https://scholar.google.com/scholar?q=Self+Speculative+Decoding+%2F+Draft+%26+Verify+%28layer+skipping%29 15. The Pitfalls of Next-Token Prediction — Gregor Bachmann, Vaishnavh Nagarajan, 2024 https://scholar.google.com/scholar?q=The+Pitfalls+of+Next-Token+Prediction 16. SPEED-Bench — Cited as [1] in the paper, 2026 https://scholar.google.com/scholar?q=SPEED-Bench 17. Diffusion Language Models are Super Data Learners — Jinjie Ni et al., 2025 https://scholar.google.com/scholar?q=Diffusion+Language+Models+are+Super+Data+Learners Interactive Visualization: Nemotron-Labs-Diffusion: One Model, Three Decoding Modes

  4. 2d ago

    Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving

    This episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue ordering and KV cache restoration decisions. It explains why first-come-first-served, shortest-job-first and earliest-deadline-first each fail. The failures are head-of-line blocking, starved long prompts, and an inability to tell a 50 ms job from a 20 s job with the same deadline. The discussion covers goodput, tiered prefix caching across GPU memory, CPU DRAM and NVMe, and chunked prefill. It also covers the paper's motivating findings: aggregate SLO attainment hides starved classes, deep-tier restores such as a 64,000-token NVMe fetch can blow the latency target, and most violations come from queueing rather than a lack of budget. Listeners get a clear walkthrough of the scheduling vocabulary, Jain's fairness index and the budget estimator. They also get a caveat that the headline cluster results come from an extended simulator fed with GB200 NVL72 data-movement profiles rather than a real deployment. Sources: 1. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse, 2026 http://arxiv.org/abs/2608.06557 2. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022 https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models 3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 4. Fast Distributed Inference Serving for Large Language Models (FastServe) — Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin, 2023 https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models+%28FastServe%29 5. Efficient LLM Scheduling by Learning to Rank — Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang, 2024 https://scholar.google.com/scholar?q=Efficient+LLM+Scheduling+by+Learning+to+Rank 6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 7. Fairness in Serving Large Language Models — Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica, 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models 8. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems — Raj Jain, Dah-Ming Chiu, William R. Hawe, 1984 https://scholar.google.com/scholar?q=A+Quantitative+Measure+of+Fairness+and+Discrimination+for+Resource+Allocation+in+Shared+Computer+Systems 9. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services — Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, Mosharaf Chowdhury, 2024 https://scholar.google.com/scholar?q=Andes%3A+Defining+and+Enhancing+Quality-of-Experience+in+LLM-Based+Text+Streaming+Services 10. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal, Kedia, Panwar, Mohan, Kwatra, Gulavani, Tumanov, Ramjee, 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference 11. Fairness in Serving Large Language Models (Virtual Token Counter, VTC) — Sheng, Cao, Li, Zhuang, Zhang, Gonzalez, Stoica et al., 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models+%28Virtual+Token+Counter%2C+VTC%29 12. QoServe: Breaking the Silos of LLM Inference Serving — Goel, Mohan, Kwatra, Anupindi, Ramjee, 2026 https://scholar.google.com/scholar?q=QoServe%3A+Breaking+the+Silos+of+LLM+Inference+Serving 13. SLOs-Serve: Optimized Serving of Multi-SLO LLMs — Chen, Jia, Khan, Krishnamurthy, Gibbons, 2025 https://scholar.google.com/scholar?q=SLOs-Serve%3A+Optimized+Serving+of+Multi-SLO+LLMs 14. JITServe: SLO-aware LLM Serving with Imprecise Request Information — Zhang, Wu, Mu, Ning, Liu, Sarda, Lee, Lai, 2025 https://scholar.google.com/scholar?q=JITServe%3A+SLO-aware+LLM+Serving+with+Imprecise+Request+Information 15. FastServe: Iteration-Level Preemptive Scheduling for LLM Inference (MLFQ) — Wu, Zhong, Zhang, Liu, Liu, Sun, Huang, Liu, Jin, 2026 (NSDI) https://scholar.google.com/scholar?q=FastServe%3A+Iteration-Level+Preemptive+Scheduling+for+LLM+Inference+%28MLFQ%29 16. Mooncake: Trading More Storage for Less Computation — Qin, Li, He, Cui, Ren, Zhang, Wu, Zheng, Xu, 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation 17. Tutti: Making SSD-backed KV Cache Practical for Long-Context LLM Serving — Qiu, Hu, Wang, Zhu, Yan, Chen, Xu, Chen, Zhang, 2026 https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-backed+KV+Cache+Practical+for+Long-Context+LLM+Serving 18. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Liu, Cheng, Yao, An, Chen, Feng, Huang, Shen, Zhang, Du, Jiang, 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 19. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment (and least-laxity-first literature, e.g. Mok 1983) — Liu and Layland (1973); Mok, 1973 / 1983 https://scholar.google.com/scholar?q=Scheduling+Algorithms+for+Multiprogramming+in+a+Hard-Real-Time+Environment+%28and+least-laxity-first+literature%2C+e.g.+Mok+1983%29 20. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving — Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+LLM+Serving Interactive Visualization: Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving

  5. 3d ago

    Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search

    This episode examines ANNS-AMP, a June 2026 paper proposing adaptive mixed-precision computing for approximate nearest neighbor search. It starts with the IVF-PQ index pipeline: cluster locating, residual calculation, lookup-table construction, distance summation and top-k selection. The paper reports that over 99.9% of distance arithmetic in the cluster-locating and table-construction stages goes to vectors that are not neighbors. The proposed fix is a small runtime predictor that picks the bit-width for each cluster and each query, paired with a bit-serial, bit-plane near-memory accelerator so that fewer bits mean less compute and less memory traffic. The discussion also covers why existing index tuning and early-termination methods miss this redundancy. It questions whether flexible-precision hardware can succeed now that fixed INT8, FP8 and FP4 formats have taken over in neural networks. A running thread asks how much of the reported speed and energy gain comes from the precision idea and how much from the custom silicon. Sources: 1. ANNS-AMP: Accelerating Approximate Nearest Neighbor Search via Adaptive Mixed-Precision Computing — Mingkai Chen, Cheng Liu, Shengwen Liang, Lei Zhang, Xiaowei Li, Huawei Li, 2026 http://arxiv.org/abs/2606.07156 2. Mixed Precision Training — Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, et al., 2018 https://scholar.google.com/scholar?q=Mixed+Precision+Training 3. HAQ: Hardware-Aware Automated Quantization with Mixed Precision — Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han, 2019 https://scholar.google.com/scholar?q=HAQ%3A+Hardware-Aware+Automated+Quantization+with+Mixed+Precision 4. HAWQ: Hessian Aware Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019 https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+Aware+Quantization+of+Neural+Networks+with+Mixed-Precision 5. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016 https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing 6. Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks — Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, Hadi Esmaeilzadeh, 2018 https://scholar.google.com/scholar?q=Bit+Fusion%3A+Bit-Level+Dynamically+Composable+Architecture+for+Accelerating+Deep+Neural+Networks 7. BISMO: A Scalable Bit-Serial Matrix Multiplication Overlay for Reconfigurable Computing — Yaman Umuroglu, Lahiru Rasnayake, Magnus Sjalander, 2018 https://scholar.google.com/scholar?q=BISMO%3A+A+Scalable+Bit-Serial+Matrix+Multiplication+Overlay+for+Reconfigurable+Computing 8. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011 https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search 9. Billion-scale similarity search with GPUs — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017 (IEEE Transactions on Big Data, 2019) https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs 10. Accelerating Large-Scale Inference with Anisotropic Vector Quantization — Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar, 2020 https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Inference+with+Anisotropic+Vector+Quantization 11. Co-design Hardware and Algorithm for Vector Search (FANNS) — Wenqi Jiang, Shigang Li, Yu Zhu, Johannes de Fine Licht, Zhenhao He, Runbin Shi, Cédric Renggli, Shuai Zhang, Theodoros Rekatsas, Torsten Hoefler, Gustavo Alonso, 2023 https://scholar.google.com/scholar?q=Co-design+Hardware+and+Algorithm+for+Vector+Search+%28FANNS%29 12. ANSMET: Approximate Nearest Neighbor Search with Near-Memory Processing and Hybrid Early Termination — Yiwei Li, Yuxin Jin, Boyu Tian, Huanchen Zhang, Mingyu Gao, 2025 https://scholar.google.com/scholar?q=ANSMET%3A+Approximate+Nearest+Neighbor+Search+with+Near-Memory+Processing+and+Hybrid+Early+Termination 13. ANNA: Specialized Architecture for Approximate Nearest Neighbor Search — Yejin Lee, Hyunji Choi, Sunhong Min, Hyunseung Lee, Sangwon Beak, Dawoon Jeong, Jae W. Lee, Tae Jun Ham, 2022 https://scholar.google.com/scholar?q=ANNA%3A+Specialized+Architecture+for+Approximate+Nearest+Neighbor+Search 14. JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping — Zihan Liu, Wentao Ni, Jingwen Leng, Yu Feng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, Yuhao Zhu, 2024 https://scholar.google.com/scholar?q=JUNO%3A+Optimizing+High-Dimensional+Approximate+Nearest+Neighbour+Search+with+Sparsity-Aware+Algorithm+and+Ray-Tracing+Core+Mapping 15. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search — Jianyang Gao, Cheng Long, 2024 https://scholar.google.com/scholar?q=RaBitQ%3A+Quantizing+High-Dimensional+Vectors+with+a+Theoretical+Error+Bound+for+Approximate+Nearest+Neighbor+Search 16. High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operations (ADSampling) — Jianyang Gao, Cheng Long, 2023 https://scholar.google.com/scholar?q=High-Dimensional+Approximate+Nearest+Neighbor+Search%3A+with+Reliable+and+Efficient+Distance+Comparison+Operations+%28ADSampling%29 17. Quicker ADC: Unlocking the Hidden Potential of Product Quantization with SIMD (and Fast Scan variants in Faiss) — Fabien André, Anne-Marie Kermarrec, Nicolas Le Scouarnec, 2019 https://scholar.google.com/scholar?q=Quicker+ADC%3A+Unlocking+the+Hidden+Potential+of+Product+Quantization+with+SIMD+%28and+Fast+Scan+variants+in+Faiss%29 Interactive Visualization: Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor Search

  6. 3d ago

    Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit

    This episode examines "All for 1-Bit," a paper claiming to compress LLM linear weights to exactly one bit each, metadata included, without retraining. It builds on double binary factorization and challenges earlier "1-bit" post-training methods, which land at roughly 2.5 to 3.5 effective bits per weight once scales, masks and outlier indices are counted. The discussion covers why bandwidth-bound decoding motivates extreme quantization, how a product of two binary matrices adds capacity without adding bits, and how null-space reasoning and Shapley-value layer allocation are meant to spend a fixed rank budget well. Along the way it separates post-training from quantization-aware training and sets up the key skeptical test: whether "genuine" and "usable" both hold, given that a score near chance (about 25 on the zero-shot scale) means a broken model. Listeners get a clear explanation of what "bits per weight" really measures and a framework for judging low-bit quantization claims. Sources: 1. All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs — Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang, 2026 http://arxiv.org/abs/2609.06161 2. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs — Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi, 2024 https://scholar.google.com/scholar?q=BiLLM%3A+Pushing+the+Limit+of+Post-Training+Quantization+for+LLMs 3. ARB-LLM: Alternating Refined Binarizations for Large Language Models — Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Zhongchao Shi, Linghe Kong, Yulun Zhang, Xiaokang Yang, 2024 https://scholar.google.com/scholar?q=ARB-LLM%3A+Alternating+Refined+Binarizations+for+Large+Language+Models 4. OneBit: Towards Extremely Low-bit Large Language Models — Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che, 2024 https://scholar.google.com/scholar?q=OneBit%3A+Towards+Extremely+Low-bit+Large+Language+Models 5. BitNet: Scaling 1-bit Transformers for Large Language Models / The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei (BitNet); Shuming Ma et al. (b1.58), 2023 / 2024 https://scholar.google.com/scholar?q=BitNet%3A+Scaling+1-bit+Transformers+for+Large+Language+Models+%2F+The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits 6. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 7. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression — Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang, 2024 https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+Large+Language+Model+Compression 8. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models — Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, Tat-Seng Chua, 2024 https://scholar.google.com/scholar?q=AlphaEdit%3A+Null-Space+Constrained+Knowledge+Editing+for+Language+Models 9. Training Networks in Null Space of Feature Covariance for Continual Learning — Shipeng Wang, Xiaorong Li, Jian Sun, Zongben Xu, 2021 https://scholar.google.com/scholar?q=Training+Networks+in+Null+Space+of+Feature+Covariance+for+Continual+Learning 10. A Unified Approach to Interpreting Model Predictions (SHAP) — Scott M. Lundberg, Su-In Lee, 2017 https://scholar.google.com/scholar?q=A+Unified+Approach+to+Interpreting+Model+Predictions+%28SHAP%29 11. Data Shapley: Equitable Valuation of Data for Machine Learning — Amirata Ghorbani, James Zou, 2019 https://scholar.google.com/scholar?q=Data+Shapley%3A+Equitable+Valuation+of+Data+for+Machine+Learning 12. HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks — Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, Kurt Keutzer, 2020 https://scholar.google.com/scholar?q=HAWQ-V2%3A+Hessian+Aware+trace-Weighted+Quantization+of+Neural+Networks 13. SqueezeLLM: Dense-and-Sparse Quantization — Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer, 2023 https://scholar.google.com/scholar?q=SqueezeLLM%3A+Dense-and-Sparse+Quantization 14. Addition is almost all you need: Compressing neural networks with double binary factorization — Vladimír Boža, Vladimír Macko, 2025 https://scholar.google.com/scholar?q=Addition+is+almost+all+you+need%3A+Compressing+neural+networks+with+double+binary+factorization 15. LittleBit: Ultra Low-Bit Quantization via Latent Factorization — Lee et al., 2026 https://scholar.google.com/scholar?q=LittleBit%3A+Ultra+Low-Bit+Quantization+via+Latent+Factorization 16. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma et al., 2024 https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29 17. FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation — Chuang Ma et al., 2024 https://scholar.google.com/scholar?q=FBI-LLM%3A+Scaling+Up+Fully+Binarized+LLMs+from+Scratch+via+Autoregressive+Distillation 18. HBLLM: Haar-based approach for accurate structured 1-bit quantized LLMs — Chen et al., 2025 https://scholar.google.com/scholar?q=HBLLM%3A+Haar-based+approach+for+accurate+structured+1-bit+quantized+LLMs 19. AQLM: Extreme Compression of LLMs via Additive Quantization — Egiazarian et al., 2024 https://scholar.google.com/scholar?q=AQLM%3A+Extreme+Compression+of+LLMs+via+Additive+Quantization 20. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks — Tseng et al., 2024 https://scholar.google.com/scholar?q=QuIP%23%3A+Even+Better+LLM+Quantization+with+Hadamard+Incoherence+and+Lattice+Codebooks 21. QTIP: Quantization with Trellises and Incoherence Processing — Tseng et al., 2024 https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing 22. Extreme Compression of Large Language Models via Additive Quantization / PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression — Malinovskii et al., 2024 https://scholar.google.com/scholar?q=Extreme+Compression+of+Large+Language+Models+via+Additive+Quantization+%2F+PV-Tuning%3A+Beyond+Straight-Through+Estimation+for+Extreme+LLM+Compression 23. SVD-LLM: Truncation-aware Singular Value Decomposition for LLM Compression — Xin Wang et al., 2024 https://scholar.google.com/scholar?q=SVD-LLM%3A+Truncation-aware+Singular+Value+Decomposition+for+LLM+Compression 24. BitNet a4.8 / bitnet.cpp: Efficient edge inference for ternary LLMs — Microsoft Research, 2024-2025 https://scholar.google.com/scholar?q=BitNet+a4.8+%2F+bitnet.cpp%3A+Efficient+edge+inference+for+ternary+LLMs 25. ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization — Zechun Liu et al., 2025 https://scholar.google.com/scholar?q=ParetoQ%3A+Scaling+Laws+in+Extremely+Low-bit+LLM+Quantization Interactive Visualization: Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-Bit

  7. 3d ago

    PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference

    This episode examines PagedMoE, a system for running a huge mixture-of-experts model (Qwen3.5-397B-A17B) on a single PC with expert weights stored on an SSD. Cache misses leave decoding stalled on storage nearly half the time, and giving each token a different bit-width per expert would normally make that worse. Treating every expert-and-precision pair as its own cache object drops the hit rate from 57% to 35%. The paper's fix is to store each expert as nested refinement units, so a resident low-precision copy can be topped up with only the missing bits, or reused with no I/O at all. The discussion separates this idea from the prior work it borrows from (Matryoshka Quantization, Any-Precision LLM, SliceMoE, HOBBIT). It also splits the reported 1.56x speedup into the part from requesting fewer bits and the part from sharing cached bits across precisions. Listeners get a clear walk through the binary-tree codebook construction, the quantization-error numbers, and the limits of the evidence, such as a single-layer headroom study and no head-to-head comparison with SliceMoE. Sources: 1. PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — Yutong Lin, Yingdi Shan, Shaoyuan Chen, Yuanyong Chen, Yongwei Wu, Mingxing Zhang, 2026 http://arxiv.org/abs/2611.03173 2. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023 https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading 3. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina, 2024 (arXiv; later revisions) https://scholar.google.com/scholar?q=MoE-Infinity%3A+Efficient+MoE+Inference+on+Personal+Machines+with+Sparsity-Aware+Expert+Cache 4. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu, 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 5. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, et al. (Apple), 2023 https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory 6. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang, Jiacheng Liu, Xiaofeng Hou, et al., 2024 https://scholar.google.com/scholar?q=HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 7. Mixture Compressor for Mixture-of-Experts LLMs Gains More (MC) — Wei Huang, Yue Liao, Jianhui Liu, et al., 2024 https://scholar.google.com/scholar?q=Mixture+Compressor+for+Mixture-of-Experts+LLMs+Gains+More+%28MC%29 8. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 9. Microscaling Data Formats for Deep Learning (OCP MX specification) — Bita Darvish Rouhani et al. (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm), 2023 https://scholar.google.com/scholar?q=Microscaling+Data+Formats+for+Deep+Learning+%28OCP+MX+specification%29 10. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs — Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee, 2024 https://scholar.google.com/scholar?q=Any-Precision+LLM%3A+Low-Cost+Deployment+of+Multiple%2C+Different-Sized+LLMs 11. Matryoshka Quantization — Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, Aditya Kusupati, 2025 https://scholar.google.com/scholar?q=Matryoshka+Quantization 12. QTIP: Quantization with Trellises and Incoherence Processing — Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa, 2024 https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing 13. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016 https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing 14. SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference — Yuseon Choi, Sangjin Kim, Jungjun Oh, Gwangtae Park, Byeongcheol Kim, Hoi-Jun Yoo, 2026 https://scholar.google.com/scholar?q=SliceMoE%3A+Bit-Sliced+Expert+Caching+under+Miss-Rate+Constraints+for+Efficient+MoE+Inference 15. D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving — Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo, 2025 https://scholar.google.com/scholar?q=D%C2%B2MoE%3A+Dual+Routing+and+Dynamic+Scheduling+for+Efficient+On-Device+MoE-based+LLM+Serving 16. DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge — Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng, 2026 https://scholar.google.com/scholar?q=DyMoE%3A+Dynamic+Expert+Orchestration+with+Mixed-Precision+Quantization+for+Efficient+MoE+Inference+on+Edge 17. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design — Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, Dahua Lin, 2025 https://scholar.google.com/scholar?q=MxMoE%3A+Mixed-precision+Quantization+for+MoE+with+Accuracy+and+Performance+Co-Design 18. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs — Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, Jingtong Hu, 2026 https://scholar.google.com/scholar?q=GEMQ%3A+Global+Expert-Level+Mixed-Precision+Quantization+for+MoE+LLMs 19. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models — Hongtao Chen, Weiyu Xie, Boxin Zhang, et al., 2025 https://scholar.google.com/scholar?q=KTransformers%3A+Unleashing+the+Full+Potential+of+CPU%2FGPU+Hybrid+Inference+for+MoE+Models 20. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2025 https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge 21. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 22. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 23. Speculative MoE / expert prefetching works (e.g., Pre-gated MoE, ProMoE, AdapMoE) — Various, 2024–2025 https://scholar.google.com/scholar?q=Speculative+MoE+%2F+expert+prefetching+works+%28e.g.%2C+Pre-gated+MoE%2C+ProMoE%2C+AdapMoE%29 Interactive Visualization: PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference

  8. 4d ago

    ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs

    This episode examines ECHO, a system from Shanghai Jiao Tong University and collaborators (OSDI 2026) that offloads the KV cache to host memory for serving sparse-attention models like DeepSeek-V3.2. It explains why sparse attention cuts compute but not storage. Cache size still grows linearly with context, so GPU memory limits batch size and leaves the sparse kernels underused. It then covers why naive recall over PCIe is slow: the top-k token set is known only after the lightning indexer has scored every earlier token. Prior prefetchers such as InfiniGen and FreeKV guess which tokens to fetch, which is lossy. ECHO instead guesses only when to fetch and then guarantees a recall of any missed token. To make this work under CUDA graph execution, it keeps its cache-management state in fixed-size integer tensors on the GPU. The discussion also stresses that the predictability ECHO relies on comes from one trained indexer, not from attention in general. It ends by questioning whether "lossless" is only a design property or something the paper actually measures. Sources: 1. ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs https://www.usenix.org/system/files/osdi26-liu-guangda.pdf 2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management 4. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, et al., 2024 https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs 5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022 https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness 7. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024 https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision 8. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention 9. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, et al. (DeepSeek-AI and collaborators), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 10. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Liu et al., 2025 https://scholar.google.com/scholar?q=FreeKV%3A+Boosting+KV+Cache+Retrieval+for+Efficient+LLM+Inference 11. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Chen et al., 2024 https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction 12. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025 https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving 13. ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp — Chen et al., 2025 https://scholar.google.com/scholar?q=ESS%3A+An+Offload-Centric+Latent-Cache+Management+Architecture+for+DeepSeek-V3.2-Exp 14. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 15. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (NSA) — Yuan et al. (DeepSeek-AI), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention+%28NSA%29 16. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Cheng et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 17. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference Interactive Visualization: ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs

See All (1K)

Ratings & Reviews

3.7
out of 5
3 Ratings

About

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

Information

You Might Also Like