AI Post Transformers

mcgrof

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 20 h fa

    Discretizing Reward Models for RL Alignment

    This episode explores the paper Discretizing Reward Models and its argument that smooth decimal reward scores can be misleading for reinforcement learning alignment, because policies learn to exploit tiny, often meaningless differences instead of genuine quality. It explains why reward models are used for fuzzy goals like helpfulness and honesty, then digs into reward hacking, equivalence classes of equally valid answers, and the distinction between a model’s ability to separate good from bad responses versus its tendency to invent rankings among ties. The discussion also covers benchmarks such as the Ties setting and the paper’s core proposal: replacing continuous scores with a small number of ordinal reward buckets built from uncertainty estimates, pairwise equivalence judgments, and hierarchical clustering. Listeners would find it interesting because it connects an abstract modeling choice to a practical alignment problem facing modern language-model training, while also examining why the field currently seems more convinced by the diagnosis than by large-scale adoption of this exact fix. Sources: 1. Discretizing Reward Models — Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal, Tongshuang Wu, Graham Neubig, Yuning Mao, 2026 http://arxiv.org/abs/2606.21795 2. Deep Reinforcement Learning from Human Preferences — Paul Christiano, Jan Leike, Tom B. Brown, Shane Legg, Dario Amodei, 2017 https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+from+Human+Preferences 3. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback — Stephen Casper, Xander Davies, Claudia Shi, Jeremy Scheurer, Dylan Hadfield-Menell, et al., 2023 https://scholar.google.com/scholar?q=Open+Problems+and+Fundamental+Limitations+of+Reinforcement+Learning+from+Human+Feedback 4. RewardBench 2: Advancing Reward Model Evaluation — Saumya Malik, Valentina Pyatkin, Sander Land, Nathan Lambert, Noah A. Smith, Hannaneh Hajishirzi, 2025 https://scholar.google.com/scholar?q=RewardBench+2:+Advancing+Reward+Model+Evaluation 5. Discretizing Reward Models — Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal, Tongshuang Wu, Graham Neubig, Yuning Mao, 2026 https://scholar.google.com/scholar?q=Discretizing+Reward+Models 6. How to Evaluate Reward Models for RLHF — Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph Gonzalez, Ion Stoica, 2024 https://scholar.google.com/scholar?q=How+to+Evaluate+Reward+Models+for+RLHF 7. What Makes a Reward Model a Good Teacher? An Optimization Perspective — Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora, 2025 https://scholar.google.com/scholar?q=What+Makes+a+Reward+Model+a+Good+Teacher?+An+Optimization+Perspective 8. The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models — Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen, 2024 https://scholar.google.com/scholar?q=The+Accuracy+Paradox+in+RLHF:+When+Better+Reward+Models+Don't+Yield+Better+Language+Models 9. Validating LLM-as-a-Judge Systems under Rating Indeterminacy — Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alexandra Chouldechova, 2025 https://scholar.google.com/scholar?q=Validating+LLM-as-a-Judge+Systems+under+Rating+Indeterminacy 10. Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback — Amirhossein Afsharrad, Ruida Zhou, Luca Viano, Sanjay Lall, Mohammad Ghavamzadeh, 2026 https://scholar.google.com/scholar?q=Beyond+Binary+Preferences:+A+Principled+Framework+for+Reward+Modeling+with+Ordinal+Feedback 11. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts — Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang, 2024 https://scholar.google.com/scholar?q=Interpretable+Preferences+via+Multi-Objective+Reward+Modeling+and+Mixture-of-Experts 12. AI Post Transformers: Split Personality Training Reveals Latent Knowledge — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-08-split-personality-training-reveals-laten-c84616.mp3 13. AI Post Transformers: Robots Need More Than VLAs and World Models — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-10-robots-need-more-than-vlas-and-world-mod-cdab8b.mp3

  2. 20 h fa

    Program-as-Weights for Compiling Fuzzy Functions

    This episode explores Program-as-Weights, a paper that asks whether a natural-language description of a fuzzy software task can be compiled once into a small reusable neural artifact instead of sending every request to a larger remote model. It explains the paper’s architecture in concrete terms: a 4B pseudo-compiler rewrites the task and generates example I/O pairs, a trained 4B compiler plus LoRA mapper turns that specification into adapter weights, and a frozen Qwen3-0.6B interpreter runs the task locally on new inputs. The discussion focuses on why this matters for real problems like log triage, malformed JSON repair, and intent-based reranking, highlighting the promised gains in cost, latency, privacy, offline use, and reproducibility. It also digs into the paper’s broader claim, debating whether this is truly a new programming paradigm or a sharp repackaging of PEFT and LoRA-based adaptation, which makes the episode interesting for listeners thinking about practical deployment rather than just model benchmarks. Sources: 1. Program-as-Weights: A Programming Paradigm for Fuzzy Functions — Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng, 2026 http://arxiv.org/abs/2607.02512 2. HyperNetworks — David Ha, Andrew Dai, Quoc V. Le, 2016 https://scholar.google.com/scholar?q=HyperNetworks 3. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al., 2021 https://scholar.google.com/scholar?q=LoRA:+Low-Rank+Adaptation+of+Large+Language+Models 4. Learning to Compile Programs to Neural Networks — Logan Weber, Jesse Michel, Alex Renda, Michael Carbin, 2024 https://scholar.google.com/scholar?q=Learning+to+Compile+Programs+to+Neural+Networks 5. Text-to-LoRA: Instant Transformer Adaption — Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange, 2025 https://scholar.google.com/scholar?q=Text-to-LoRA:+Instant+Transformer+Adaption 6. SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass — Y. Liu, X. Wang, Y. Mao, Y. Gelberg, H. Maron, and M. Zhang, 2026 https://scholar.google.com/scholar?q=SHINE:+A+Scalable+In-Context+Hypernetwork+for+Mapping+Context+to+LoRA+in+a+Single+Pass 7. Doc-to-LoRA: Learning to Instantly Internalize Contexts — R. Charakorn, E. Cetin, S. Uesaka, and R. T. Lange, 2026 https://scholar.google.com/scholar?q=Doc-to-LoRA:+Learning+to+Instantly+Internalize+Contexts 8. Latent Context Compilation: Distilling Long Context into Compact Portable Memory — Z. Li, Y. Zhou, and Q. Xu, 2026 https://scholar.google.com/scholar?q=Latent+Context+Compilation:+Distilling+Long+Context+into+Compact+Portable+Memory 9. The Alchemist: Automated Labeling 500x Cheaper than LLM Data Annotators — T. Huang, C. Cao, V. Bhargava, and F. Sala, 2024 https://scholar.google.com/scholar?q=The+Alchemist:+Automated+Labeling+500x+Cheaper+than+LLM+Data+Annotators 10. Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models — Vladimir Araujo, Marie-Francine Moens, Tinne Tuytelaars, 2024 https://arxiv.org/abs/2408.09053 11. X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design — Eric L. Buehler, Markus J. Buehler, 2024 https://arxiv.org/abs/2402.07148 12. CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learning — Yu Feng, Zhen Tian, Yifan Zhu, Zongfu Han, Haoran Luo, Guangwei Zhang, Meina Song, 2024 https://arxiv.org/abs/2407.21043 13. Gradient Projection For Continual Parameter-Efficient Tuning — Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Wensheng Zhang, Zhi Han, Yuan Xie, 2024 https://arxiv.org/abs/2405.13383 14. Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading — Minrui Xu, Dusit Niyato, Christopher G. Brinton, 2025 https://arxiv.org/abs/2501.14205 15. AI Post Transformers: SGLang for Faster Structured LLM Programs — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-06-sglang-for-faster-structured-llm-program-c59f1c.mp3 16. AI Post Transformers: OpenSkill for Open-World Self-Evolution in LLM Agents — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-08-openskill-for-open-world-self-evolution-19762a.mp3 17. AI Post Transformers: Learning Facts at Scale with Active Reading — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-25-learning-facts-at-scale-with-active-read-161bea.mp3 18. AI Post Transformers: Fine-Tuning LLMs for Human Behavior Prediction — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-21-fine-tuning-llms-for-human-behavior-pred-c79163.mp3

  3. 1 g fa

    AIConfigurator for Cross-Framework LLM Serving

    This episode explores AIConfigurator, a NVIDIA-led system for optimizing LLM serving configurations across frameworks such as TensorRT-LLM, vLLM, SGLang, and NVIDIA’s internal stack without brute-force benchmarking. It explains why operators should care more about TTFT, TPOT, and goodput than raw tokens-per-second, and unpacks the real serving tradeoffs around prefill/decode disaggregation, hybrid tensor/pipeline/expert parallelism, CUDA graphs, KV-cache sizing, and token limits. The discussion argues that the paper’s main contribution is a calibrated, framework-agnostic performance model built from primitive costs like GEMMs, attention, communication, and memory operations, then combined with backend-specific scheduling behavior to search thousands of deployment choices quickly. It is especially interesting for listeners who want a concrete view of LLM deployment economics: how to translate hardware budgets and latency targets into practical, high-performing serving setups without wasting days of GPU tuning. Sources: 1. AIConfigurator for Cross-Framework LLM Serving https://arxiv.org/pdf/2601.06288 2. LLM Inference Serving: Survey of Recent Advances and Opportunities — Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari, 2024 https://scholar.google.com/scholar?q=LLM+Inference+Serving:+Survey+of+Recent+Advances+and+Opportunities 3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 4. Vidur: A Large-Scale Simulation Framework For LLM Inference — Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, Alexey Tumanov, 2024 https://scholar.google.com/scholar?q=Vidur:+A+Large-Scale+Simulation+Framework+For+LLM+Inference 5. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving — Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, et al., 2026 https://scholar.google.com/scholar?q=AIConfigurator:+Lightning-Fast+Configuration+Optimization+for+Multi-Framework+LLM+Serving 6. Apex: An Extensible and Dynamism-aware Simulator for Automated Parallel Execution in LLM Serving — Yi-Chien Lin et al., 2024 https://scholar.google.com/scholar?q=Apex:+An+Extensible+and+Dynamism-aware+Simulator+for+Automated+Parallel+Execution+in+LLM+Serving 7. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong et al., 2024 https://scholar.google.com/scholar?q=DistServe:+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 8. Splitwise: Efficient Generative LLM Inference using Phase Splitting — Pratyush Patel et al., 2024 https://scholar.google.com/scholar?q=Splitwise:+Efficient+Generative+LLM+Inference+using+Phase+Splitting 9. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024 https://scholar.google.com/scholar?q=Mooncake:+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal et al., 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 11. KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference — Huan Yang et al., 2025 https://arxiv.org/abs/2503.16525 12. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management — Haoyu Zheng et al., 2026 https://arxiv.org/abs/2605.06472 13. PyGraph: Robust Compiler Support for CUDA Graphs in PyTorch — Abhishek Ghosh et al., 2025 https://arxiv.org/abs/2503.19779 14. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang et al., 2025 https://arxiv.org/abs/2508.01989 15. Enhancing LLM Efficiency: Targeted Pruning for Prefill-Decode Disaggregation in Inference — Hao Zhang et al., 2025 https://arxiv.org/abs/2509.04467 16. Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts — Xuan-Phi Nguyen et al., 2026 https://arxiv.org/abs/2601.17111 17. AI Post Transformers: LLMServingSim 2.0 for Disaggregated LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-27-llmservingsim-20-for-disaggregated-llm-s-05c04b.mp3 18. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp3 19. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3 20. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3

  4. 1 g fa

    Nemotron-TwoTower for Parallel Diffusion Language Modeling

    This episode explores Nemotron-TwoTower, an NVIDIA paper that tries to keep the text quality of autoregressive language models while reducing the one-token-at-a-time decoding bottleneck through block-wise diffusion generation. It explains how diffusion language modeling works in practice: future token blocks begin as noisy or masked guesses and are iteratively refined in parallel, rather than emitted strictly one token at a time. The discussion focuses on the paper’s core architectural idea of splitting responsibilities between a frozen pretrained causal context tower and a separate trainable denoiser tower, including layer-aligned cross-attention, reused KV caches and Mamba states, and confidence-based early token commitment. Listeners would find it interesting because it gets beyond benchmark hype and examines the real systems tradeoff the paper is making: higher throughput through heavier refinement steps, balanced against serving complexity, multiple denoising passes, and the risk of losing autoregressive-level reliability. Sources: 1. Nemotron-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context — Fitsum Reda, John Kamalu, Roger Waleffe, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, 2026 http://arxiv.org/abs/2606.26493 2. Structured Denoising Diffusion Models in Discrete State-Spaces (https://arxiv.org/abs/2107.03006) — Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, 2021 https://scholar.google.com/scholar?q=Structured+Denoising+Diffusion+Models+in+Discrete+State-Spaces+(https://arxiv.org/abs/2107.03006) 3. Simple and Effective Masked Diffusion Language Models (https://arxiv.org/abs/2406.07524) — Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Volodymyr Kuleshov, 2024 https://scholar.google.com/scholar?q=Simple+and+Effective+Masked+Diffusion+Language+Models+(https://arxiv.org/abs/2406.07524) 4. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (https://arxiv.org/abs/2503.09573) — Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Volodymyr Kuleshov, 2025 https://scholar.google.com/scholar?q=Block+Diffusion:+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+(https://arxiv.org/abs/2503.09573) 5. Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (https://arxiv.org/abs/2510.22852) — Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov, 2025 https://scholar.google.com/scholar?q=Encoder-Decoder+Diffusion+Language+Models+for+Efficient+Training+and+Inference+(https://arxiv.org/abs/2510.22852) 6. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models — Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov, 2025 https://scholar.google.com/scholar?q=Block+Diffusion:+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models 7. Encoder-Decoder Diffusion Language Models for Efficient Training and Inference — Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov, 2025 https://scholar.google.com/scholar?q=Encoder-Decoder+Diffusion+Language+Models+for+Efficient+Training+and+Inference 8. Simple and Effective Masked Diffusion Language Models — Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, Volodymyr Kuleshov, 2024 https://scholar.google.com/scholar?q=Simple+and+Effective+Masked+Diffusion+Language+Models 9. Large Language Diffusion Models — Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li, 2025 https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models 10. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data — Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, Chongxuan Li, 2024 https://scholar.google.com/scholar?q=Your+Absorbing+Discrete+Diffusion+Secretly+Models+the+Conditional+Distributions+of+Clean+Data 11. AI Post Transformers: NeurIPS 2025: Large Language Diffusion Models — Hal Turing & Dr. Ada Shannon, 2025 https://podcast.do-not-panic.com/episodes/neurips-2025-large-language-diffusion-models/ 12. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp3 13. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3 14. AI Post Transformers: Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/draft-verify-lossless-large-language-model-acceleration-via-self-speculative-dec/ 15. AI Post Transformers: JETSPEC and Parallel Tree Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-27-jetspec-and-parallel-tree-speculative-de-3d144c.mp3

  5. 4 gg fa

    DART Speeds Up Speculative LLM Decoding

    This episode explores the DART paper as a practical attempt to make speculative decoding deliver real end-to-end speedups for memory-bound LLM inference. It explains how exact draft-and-verify decoding works, why accepted chunk length only matters when the drafter is cheap enough, and how DART differs from Medusa and EAGLE by reusing target-model hidden states to predict several future tokens in parallel with a diffusion-inspired draft stage. The discussion focuses on DART’s mechanics, including multi-layer state reuse, masked future slots, N-gram-guided pruning, and a shifted-logit design that makes the first drafted token especially important because an early mistake invalidates the rest of the chunk. Listeners would find it interesting because it connects model architecture choices to real serving constraints like latency, batching, and GPU efficiency, showing where theoretical decoding gains do and do not survive in production. Sources: 1. DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference — Fuliang Liu, Xue Li, Ketai Zhao, Yinxi Gao, Ziyan Zhou, Zhonghui Zhang, Zhibin Wang, Wanchun Dou, Sheng Zhong, Chen Tian, 2026 http://arxiv.org/abs/2601.19278 2. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding — Hemeng Xia, Zijian Wu, Chunxi Zhang, Yonggan Fu, Haoran Sun, Zhicong Liu, Ping Luo, 2024 https://arxiv.org/abs/2401.07851 3. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://arxiv.org/abs/2211.17192 4. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024 https://arxiv.org/abs/2401.15077 5. Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion — Jacob K. Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto, 2024 https://arxiv.org/abs/2408.05636 6. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025 https://scholar.google.com/scholar?q=EAGLE-3:+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 7. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao, 2024 https://scholar.google.com/scholar?q=Medusa:+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 8. DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding — Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, Jun Wang, 2025 https://scholar.google.com/scholar?q=DiffuSpec:+Unlocking+Diffusion+Language+Models+for+Speculative+Decoding 9. SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding — Jameson Sandler, Jacob K. Christopher, Thomas Hartvigsen, Nando Fioretto, 2025 https://scholar.google.com/scholar?q=SpecDiff-2:+Scaling+Diffusion+Drafter+Alignment+For+Faster+Speculative+Decoding 10. Speculative Decoding: Performance or Illusion? — Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, Alvin Cheung, 2026 https://scholar.google.com/scholar?q=Speculative+Decoding:+Performance+or+Illusion? 11. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3 12. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 13. AI Post Transformers: InfiniGen for Efficient Long-Context LLM Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-18-infinigen-for-efficient-long-context-llm-143d77.mp3 14. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp3

  6. 4 gg fa

    SALCA for Sparse Long-Context Decoding

    This episode explores why long-context LLM decoding becomes memory-bandwidth bound: once prompt prefill is done, each new token must repeatedly scan an ever-growing KV cache, making inference limited more by data movement than raw compute. It explains sparse attention as the idea that only a small fraction of prior tokens matter for each step, and uses top-k recall to frame the core challenge of preserving the right token ranking while cutting memory traffic. The discussion centers on Salca’s main argument: a sparsity-aware accelerator can make sparse decoding practical by combining dominant-channel feature selection with asymmetric ultra-low-bit query/key prediction, reducing predictor traffic to roughly one-eighth of a standard 4-bit filtering baseline. A listener would find it interesting because it connects transformer inference theory, serving-system bottlenecks, and custom chip design into a concrete case for faster, more energy-efficient long-context generation. Sources: 1. SALCA for Sparse Long-Context Decoding https://arxiv.org/pdf/2604.24820 2. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019 https://scholar.google.com/scholar?q=Fast+Transformer+Decoding:+One+Write-Head+is+All+You+Need 3. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jeff Dean, et al., 2022 https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference 4. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 5. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024 https://scholar.google.com/scholar?q=QUEST:+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 6. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning — Hanrui Wang, Zhekai Zhang, Song Han, 2020 https://scholar.google.com/scholar?q=SpAtten:+Efficient+Sparse+Attention+Architecture+with+Cascade+Token+and+Head+Pruning 7. Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention — Zhe Zhou, Junlin Liu, Zhenyu Gu, Guangyu Sun, 2021 https://scholar.google.com/scholar?q=Energon:+Towards+Efficient+Acceleration+of+Transformers+Using+Dynamic+Sparse+Attention 8. S2-Attention: Hardware-Aware Context Sharding Among Attention Heads — Xihui Lin, Yunan Zhang, Suyu Ge, Liliang Ren, Barun Patra, Vishrav Chaudhary, Hao Peng, Xia Song, 2024 https://scholar.google.com/scholar?q=S2-Attention:+Hardware-Aware+Context+Sharding+Among+Attention+Heads 9. SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining — Yifan Zhang, Zunhai Su, Shuhao Hu, Rui Yang, Wei Wu, Yulei Qian, Yuchen Xie, Xunliang Cai, 2026 https://scholar.google.com/scholar?q=SnapMLA:+Efficient+Long-Context+MLA+Decoding+via+Hardware-Aware+FP8+Quantized+Pipelining 10. ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs (https://arxiv.org/abs/2602.07721) — Yanlin Qi et al., 2026 https://scholar.google.com/scholar?q=ParisKV:+Fast+and+Drift-Robust+KV-Cache+Retrieval+for+Long-Context+LLMs+(https://arxiv.org/abs/2602.07721) 11. LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences (https://arxiv.org/abs/2510.11292) — Wenbo Wu et al., 2025 https://scholar.google.com/scholar?q=LouisKV:+Efficient+KV+Cache+Retrieval+for+Long+Input-Output+Sequences+(https://arxiv.org/abs/2510.11292) 12. Efficient Low Rank Attention for Long-Context Inference in Large Language Models (https://arxiv.org/abs/2510.23649) — Tenghui Li et al., 2025 https://scholar.google.com/scholar?q=Efficient+Low+Rank+Attention+for+Long-Context+Inference+in+Large+Language+Models+(https://arxiv.org/abs/2510.23649) 13. LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference (https://arxiv.org/abs/2503.08879) — Guangtao Wang et al., 2025 https://scholar.google.com/scholar?q=LLMs+Know+What+to+Drop:+Self-Attention+Guided+KV+Cache+Eviction+for+Efficient+Long-Context+Inference+(https://arxiv.org/abs/2503.08879) 14. Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs (https://arxiv.org/abs/2602.05191) — Wentao Ni et al., 2026 https://scholar.google.com/scholar?q=Double-P:+Hierarchical+Top-P+Sparse+Attention+for+Long-Context+LLMs+(https://arxiv.org/abs/2602.05191) 15. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (https://arxiv.org/abs/2502.00299) — Xiang Liu et al., 2025 https://scholar.google.com/scholar?q=ChunkKV:+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference+(https://arxiv.org/abs/2502.00299) 16. ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (https://arxiv.org/abs/2405.14256) — Yefei He et al., 2024 https://scholar.google.com/scholar?q=ZipCache:+Accurate+and+Efficient+KV+Cache+Quantization+with+Salient+Token+Identification+(https://arxiv.org/abs/2405.14256) 17. Accurate KV Cache Quantization with Outlier Tokens Tracing (https://arxiv.org/abs/2505.10938) — Yi Su et al., 2025 https://scholar.google.com/scholar?q=Accurate+KV+Cache+Quantization+with+Outlier+Tokens+Tracing+(https://arxiv.org/abs/2505.10938) 18. A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (https://arxiv.org/abs/2410.01485) — Suyu Ge et al., 2024 https://scholar.google.com/scholar?q=A+Little+Goes+a+Long+Way:+Efficient+Long+Context+Training+and+Inference+with+Partial+Contexts+(https://arxiv.org/abs/2410.01485) 19. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3 20. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp3 21. AI Post Transformers: MiniMax Sparse Attention at Million-Token Scale — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-13-minimax-sparse-attention-at-million-toke-300108.mp3 22. AI Post Transformers: ScoutAttention for Efficient KV Cache Offloading — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-24-scoutattention-for-efficient-kv-cache-of-b26699.mp3 23. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 24. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3

  7. 4 gg fa

    SERA: Repository Specialization for Open Coding Agents

    This episode explores SERA, a method for specializing open-weight coding agents to individual repositories so they learn local APIs, naming conventions, refactor habits, and test idioms as model behavior rather than prompt context. It contrasts that idea with repository-aware retrieval, arguing that while RAG updates faster, weight adaptation could better capture the diffuse, codebase-specific patterns that matter for agentic tasks like searching, planning, editing, and validating changes. The discussion focuses on SERA’s soft-verification pipeline: a teacher model generates repository-grounded edit trajectories and synthetic pull request descriptions, then a second rollout regenerates the patch and keeps examples only when the two edits overlap enough at the line level. A listener would find it interesting because it gets into the practical tradeoff at the heart of coding agents: whether cheaper agreement-based filtering can make repo specialization useful without the heavy infrastructure cost of full execution-based verification. Sources: 1. SERA: Soft-Verified Efficient Repository Agents — Ethan Shen, Daniel Tormoen, Saurabh Shah, Ali Farhadi, Tim Dettmers, 2026 http://arxiv.org/abs/2601.20789 2. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation — Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, Weizhu Chen, 2023 https://scholar.google.com/scholar?q=RepoCoder:+Repository-Level+Code+Completion+Through+Iterative+Retrieval+and+Generation 3. RepoFusion: Training Code Models to Understand Your Repository — Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, Torsten Scholak, 2023 https://scholar.google.com/scholar?q=RepoFusion:+Training+Code+Models+to+Understand+Your+Repository 4. Customizing an LLM for Enterprise Software Engineering — Aditya Kini, Satish Chandra, Milad Hashemi, Saksham Thakur, Aditya Pandey, Vincent Nguyen, et al., 2026 https://scholar.google.com/scholar?q=Customizing+an+LLM+for+Enterprise+Software+Engineering 5. SERA: Soft-Verified Efficient Repository Agents — Ethan Shen, Daniel Tormoen, Saurabh Shah, Ali Farhadi, Tim Dettmers, 2026 https://scholar.google.com/scholar?q=SERA:+Soft-Verified+Efficient+Repository+Agents 6. CodeT: Code Generation with Generated Tests — Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen, 2022 https://scholar.google.com/scholar?q=CodeT:+Code+Generation+with+Generated+Tests 7. LEVER: Learning to Verify Language-to-Code Generation with Execution — Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin, 2023 https://scholar.google.com/scholar?q=LEVER:+Learning+to+Verify+Language-to-Code+Generation+with+Execution 8. SWE-smith: Scaling Data for Software Engineering Agents — John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, et al., 2025 https://scholar.google.com/scholar?q=SWE-smith:+Scaling+Data+for+Software+Engineering+Agents 9. R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents — N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica, 2025 https://scholar.google.com/scholar?q=R2E-Gym:+Procedural+Environments+and+Hybrid+Verifiers+for+Scaling+Open-Weights+SWE+Agents 10. RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing — Y. Xie, A. Xie, D. Sheth, P. Liu, D. Fried, and C. P. Rosé, 2025 https://scholar.google.com/scholar?q=RepoST:+Scalable+Repository-Level+Coding+Environment+Construction+with+Sandbox+Testing 11. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents — I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel, 2025 https://scholar.google.com/scholar?q=SWE-rebench:+An+Automated+Pipeline+for+Task+Collection+and+Decontaminated+Evaluation+of+Software+Engineering+Agents 12. CodeRAG-Bench: Can Retrieval Augment Code Generation? — Z. Z. Wang, A. Asai, X. V. Yu, F. F. Xu, Y. Xie, G. Neubig, and D. Fried, 2024 https://scholar.google.com/scholar?q=CodeRAG-Bench:+Can+Retrieval+Augment+Code+Generation? 13. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces — M. A. Merrill et al., 2026 https://scholar.google.com/scholar?q=Terminal-Bench:+Benchmarking+Agents+on+Hard,+Realistic+Tasks+in+Command+Line+Interfaces 14. Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks — A. Chandra, A. Agrawal, A. Hosseini, S. Fischmeister, R. Agarwal, N. Goyal, and A. Courville, 2026 https://scholar.google.com/scholar?q=Shape+of+Thought:+When+Distribution+Matters+More+than+Correctness+in+Reasoning+Tasks 15. GenX: Mastering Code and Test Generation with Execution Feedback — Nan Wang et al., 2024 https://scholar.google.com/scholar?q=GenX:+Mastering+Code+and+Test+Generation+with+Execution+Feedback 16. Enhancing LLM-Based Code Translation with Verified Multi-Semantic Representations — Yufu Wang et al., 2026 https://scholar.google.com/scholar?q=Enhancing+LLM-Based+Code+Translation+with+Verified+Multi-Semantic+Representations 17. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback — Shihan Dou et al., 2024 https://scholar.google.com/scholar?q=StepCoder:+Improve+Code+Generation+with+Reinforcement+Learning+from+Compiler+Feedback 18. Execution-based Code Generation using Deep Reinforcement Learning — Parshin Shojaee et al., 2023 https://scholar.google.com/scholar?q=Execution-based+Code+Generation+using+Deep+Reinforcement+Learning 19. Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering — Yoseph Berhanu Alebachew et al., 2026 https://scholar.google.com/scholar?q=Beyond+Code+Snippets:+Benchmarking+LLMs+on+Repository-Level+Question+Answering 20. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp3 21. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp3 22. AI Post Transformers: Trace Rewriting Against Unauthorized LLM Distillation — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-trace-rewriting-against-unauthorized-llm-306357.mp3 23. AI Post Transformers: Learning Facts at Scale with Active Reading — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-25-learning-facts-at-scale-with-active-read-161bea.mp3

  8. 5 gg fa

    Cache-Resident LLM Inference in GB-Scale Caches

    This episode explores a 2026 paper on cache-resident LLM inference, asking whether modern CPUs with gigabyte-scale last-level caches can cut decoding latency by keeping model weights on-chip instead of repeatedly fetching them from DRAM. It explains why autoregressive decoding is often memory-bound rather than compute-bound, then breaks down the paper’s main design ideas: separating weight-heavy projections and feed-forward work from attention and KV-cache handling, and using fine-grained static scheduling to reduce synchronization overhead. The discussion gets concrete about the system architecture on AMD EPYC 9684X machines, including dual-socket role separation, INT8 weights and KV caches, and locality-aware placement of weight shards and activations. A listener would find it interesting because it gives a sharp, skeptical look at where CPU-based LLM serving might genuinely improve throughput and time-per-output-token, while also arguing that this is a targeted systems win rather than a replacement for GPU-first inference. Sources: 1. Cache-Resident LLM Inference in GB-Scale Last-Level Caches — Wanning Zhang, Tongzhou Gu, Marco Canini, Ceyu Xu, Jian Weng, 2026 http://arxiv.org/abs/2606.25353 2. LLM Inference Serving: Survey of Recent Advances and Opportunities — Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari, 2024 https://scholar.google.com/scholar?q=LLM+Inference+Serving:+Survey+of+Recent+Advances+and+Opportunities 3. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Hao Zhang, Ion Stoica, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 4. Splitwise: Efficient generative LLM inference using phase splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, Ricardo Bianchini, 2023 https://scholar.google.com/scholar?q=Splitwise:+Efficient+generative+LLM+inference+using+phase+splitting 5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Joseph E. Gonzalez, Percy Liang, Christopher Re, Ion Stoica, Ce Zhang, et al., 2023 https://scholar.google.com/scholar?q=FlexGen:+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 6. Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks — Charles Eckert, Xiaowei Wang, Jingcheng Wang, Arun Subramaniyan, Ravi Iyer, Dennis Sylvester, David Blaauw, Reetuparna Das, 2018 https://scholar.google.com/scholar?q=Neural+Cache:+Bit-Serial+In-Cache+Acceleration+of+Deep+Neural+Networks 7. Proximu$: Efficiently Scaling DNN Inference in Multi-core CPUs through Near-Cache Compute — Anant V. Nori, Rahul Bera, Shankar Balachandran, Joydeep Rakshit, Om J. Omer, Avishaii Abuhatzera, Belliappa Kuttanna, Sreenivas Subramoney, 2020 https://scholar.google.com/scholar?q=Proximu$:+Efficiently+Scaling+DNN+Inference+in+Multi-core+CPUs+through+Near-Cache+Compute 8. Inference Performance Optimization for Large Language Models on CPUs — Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui, Weifei Yu, Yi Xie, 2024 https://scholar.google.com/scholar?q=Inference+Performance+Optimization+for+Large+Language+Models+on+CPUs 9. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs — Yuzhuang Xu, Xu Han, Yuxuan Li, Wanxiang Che, 2026 https://scholar.google.com/scholar?q=ArcLight:+A+Lightweight+LLM+Inference+Architecture+for+Many-Core+CPUs 10. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2025 https://scholar.google.com/scholar?q=vAttention:+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention 11. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 12. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe:+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+Large+Language+Model+Serving 13. WaferLLM: Large Language Model Inference at Wafer Scale — Congjie He, Yeqi Huang, Pei Mu, Ziming Miao, Jilong Xue, Lingxiao Ma, Fan Yang, Luo Mai, 2025 https://scholar.google.com/scholar?q=WaferLLM:+Large+Language+Model+Inference+at+Wafer+Scale 14. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2025 https://scholar.google.com/scholar?q=T-MAC:+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge 15. Compute Or Load KV Cache? Why Not Both? — Shuowei Jin et al., 2024 https://arxiv.org/abs/2410.03065 16. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025 https://arxiv.org/abs/2507.07400 17. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon et al., 2024 https://arxiv.org/abs/2405.12981 18. QCQA: Quality and Capacity-aware Grouped Query Attention — Vinay Joshi et al., 2024 https://arxiv.org/abs/2406.10247 19. Beyond KV Caching: Shared Attention for Efficient LLMs — Bingli Liao and Danilo Vasconcellos Vargas, 2024 https://arxiv.org/abs/2407.12866 20. ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive — Xinhao Luo et al., 2025 https://arxiv.org/abs/2508.18850 21. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3 22. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 23. AI Post Transformers: ScoutAttention for Efficient KV Cache Offloading — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-24-scoutattention-for-efficient-kv-cache-of-b26699.mp3 24. AI Post Transformers: CacheFlow and 3D-Parallel KV Cache Restoration — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-01-cacheflow-and-3d-parallel-kv-cache-resto-8db883.mp3 25. AI Post Transformers: VeriCache: Lossless LLM Inference from Lossy KV Caches — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-vericache-lossless-llm-inference-from-lo-df9daf.mp3 26. AI Post Transformers: Harvest: Borrowing Peer GPU Memory for LLMs — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-02-harvest-borrowing-peer-gpu-memory-for-ll-e9e54f.mp3

Elenco completo (752)

Descrizione

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

Informazioni

  • Creatore
    mcgrof
  • Anni di attività
    2025 - 2026
  • Puntate
    752
  • Classificazione
    Contenuti adatti a tutti
  • Sito web del podcast
    AI Post Transformers

Potrebbero piacerti anche…