AI Post Transformers

mcgrof

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. قبل ١٩ ساعة

    Puzzle Compression for Hybrid MoE LLMs

    This episode explores NVIDIA’s Nemotron-Labs-3-Puzzle-75B-A9B, a July 2026 paper on compressing a hybrid Mamba-attention mixture-of-experts reasoning model after training instead of building a smaller model from scratch. It explains why hybrid MoE systems are harder to shrink than dense transformers, focusing on routing, active expert budgets, Mamba state, and long-context memory costs, and walks through the paper’s iterative Puzzle method of pruning, distilling, and recovery in staged rounds. The discussion highlights the headline result: a parent model reduced from 120.7B total parameters and 12.8B active per token to 75.3B total and 9.3B active, while reportedly delivering about 2x higher interactive throughput on an 8xB200 server and raising million-token concurrency on a single H100 from one request to eight. Listeners would find it interesting because it digs into whether those gains reflect a real quality-efficiency advance for long-context serving or depend heavily on extra tricks such as quantization, multi-token prediction, and substantial post-training recovery compute. Sources: 1. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv, 2026 http://arxiv.org/abs/2607.04371 2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021 https://scholar.google.com/scholar?q=Switch+Transformers:+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity 3. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber, Barak Lenz, Hofit Bata, et al., 2024 https://scholar.google.com/scholar?q=Jamba:+A+Hybrid+Transformer-Mamba+Language+Model 4. Puzzle: Distillation-Based NAS for Inference-Optimized LLMs — Akhiad Bercovich, Tomer Ronen, Talor Abramovich, et al., 2024 https://scholar.google.com/scholar?q=Puzzle:+Distillation-Based+NAS+for+Inference-Optimized+LLMs 5. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs — Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, et al., 2026 https://scholar.google.com/scholar?q=Nemotron-Labs-3-Puzzle-75B-A9B:+Compressing+Hybrid+MoE+LLMs 6. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — NVIDIA et al., 2026 https://scholar.google.com/scholar?q=Nemotron+3+Super:+Open,+Efficient+Mixture-of-Experts+Hybrid+Mamba-Transformer+Model+for+Agentic+Reasoning 7. Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration — Akhiad Bercovich et al., 2026 https://scholar.google.com/scholar?q=Extending+Puzzle+for+Mixture-of-Experts+Reasoning+Models+with+Application+to+GPT-OSS+Acceleration 8. Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control — Ali Taghibakhshi et al., 2026 https://scholar.google.com/scholar?q=Star+Elastic:+Many-in-One+Reasoning+LLMs+with+Efficient+Budget+Control 9. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts — Venmugil Elango et al., 2026 https://scholar.google.com/scholar?q=LatentMoE:+Toward+Optimal+Accuracy+per+FLOP+and+Parameter+in+Mixture+of+Experts 10. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding — Talor Abramovich et al., 2026 https://scholar.google.com/scholar?q=SPEED-Bench:+A+Unified+and+Diverse+Benchmark+for+Speculative+Decoding 11. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024 https://arxiv.org/abs/2401.06066 12. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling — Liliang Ren et al., 2024 https://arxiv.org/abs/2406.07522 13. Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning — Zhen Li et al., 2025 https://arxiv.org/abs/2501.03035 14. ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference — Yesheng Liang et al., 2025 https://arxiv.org/abs/2511.10645 15. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu et al., 2023 https://arxiv.org/abs/2310.07240 16. KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving — Zedong Liu et al., 2026 https://arxiv.org/abs/2605.13734 17. AI Post Transformers: Ministral 3: Cascade Distillation for Long-Context Multimodal Models — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-cascade-distillation-for-long-context-mu-0ebd1a.mp3 18. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3 19. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp3 20. AI Post Transformers: AIConfigurator for Cross-Framework LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-07-02-aiconfigurator-for-cross-framework-llm-s-b39139.mp3 21. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3 22. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp3

  2. قبل يوم واحد

    Gemma 4 Open-Weight Multimodal Reasoning

    This episode explores Google DeepMind’s Gemma 4 Technical Report by separating the family’s headline claims into distinct pieces: base architecture, post-training, multimodality, sparse routing, and extra test-time compute from “thinking mode.” It explains, in plain language, how the lineup mixes very different design bets, including encoder-free image and audio inputs in the 12B model, a sparse MoE setup in the 26B-A4B, and long-context efficiency tricks such as p-RoPE, speculative decoding, and key-as-value reuse to cut KV-cache costs. The discussion argues that Gemma 4 is not one breakthrough but a bundle of science and deployment choices, which matters when judging what actually drives quality, latency, and cost. Listeners would find it interesting because it ties those design choices to concrete benchmark jumps in reasoning, coding, and vision performance while showing how much of the improvement may come from the inference stack rather than a single model innovation. Sources: 1. Gemma 4 Technical Report — Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamed, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chrysovalantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Glenn Cameron, Charlotte Caucheteux, Garima Chadha, Jetha Chan, Aditya Chawla, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu-hsiang Chien, Nikolai Chinaev, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodhia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Takumi Fujimoto, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhin, Dongdong Li, Qiujia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O'Donnell, Kate Olszewska, Bernett Orlando, Wanqiong Pan, Rina Panigrahy, Unnati Parekh, Chunjong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Yuan Shangguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Chetan Tekur, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wassenberg, John Wieting, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, Chen Zou, 2026 http://arxiv.org/abs/2607.02770 2. Round and Round We Go! What Makes Rotary Positional Encodings Useful? — Federico Barbero et al., 2025 https://scholar.google.com/scholar?q=Round+and+Round+We+Go!+What+Makes+Rotary+Positional+Encodings+Useful? 3. Do Transformers Need Three Projections? Systematic Study of QKV Variants — A. Kayyam, A. M. Gopal, Michael A. Lewis, 2026 https://scholar.google.com/scholar?q=Do+Transformers+Need+Three+Projections?+Systematic+Study+of+QKV+Variants 4. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 5. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yinghui Li, Fandong Wei, Cheng Zhang, Hongyang Zhang, 2024 https://scholar.google.com/scholar?q=EAGLE:+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty 6. RULER: What's the Real Context Size of Your Long-Context Language Models? — C.-P. Hsieh et al., 2024 https://scholar.google.com/scholar?q=RULER:+What's+the+Real+Context+Size+of+Your+Long-Context+Language+Models? 7. OpenAI o1 System Card — OpenAI, 2024 https://scholar.google.com/scholar?q=OpenAI+o1+System+Card 8. Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning — Xinghao Chen et al., 2025 https://arxiv.org/abs/2505.16782 9. Latent Chain-of-Thought for Visual Reasoning — Guohao Sun et al., 2025 https://arxiv.org/abs/2510.23925 10. Sliding Window Attention Adaptation — Yijiong Yu et al., 2025 https://arxiv.org/abs/2512.10411 11. Short window attention enables long-term memorization — Loic Cabannes et al., 2025 https://arxiv.org/abs/2509.24552 12. Can I Buy Your KV Cache? — Luoyuan Zhang, 2026 https://arxiv.org/abs/2606.13361 13. VCoder: Versatile Vision Encoders for Multimodal Large Language Models — Jitesh Jain, Jianwei Yang, Humphrey Shi, 2023 https://arxiv.org/abs/2312.14233 14. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3 15. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3 16. AI Post Transformers: JETSPEC and Parallel Tree Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-27-jetspec-and-parallel-tree-speculative-de-3d144c.mp3 17. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp3 18. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp3 19. AI Post Transformers: LPU Chip for Low-Latency LLM Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-20-lpu-chip-for-low-latency-llm-inference-be13c3.mp3

  3. قبل يوم واحد

    Hierarchical Sparse Attention for Infinite Context

    This episode explores the HiLS paper, which tackles the central long-context transformer problem: how to preserve normal-context quality while avoiding the exploding compute and KV-cache costs of dense attention at extreme sequence lengths. It explains the paper’s core mechanism of hierarchical sparse attention, where the model learns summary keys for context chunks, retrieves the most relevant chunks for each query, attends within them, and then keeps the retrieval scores in the forward pass so the chunk selector is trained directly by language-model loss. The discussion contrasts this with older sparse schemes, sliding-window attention, and positional stretching methods like YaRN, arguing that better retrieval inside attention matters as much as longer positional extrapolation or extra continued pretraining. Listeners would find it interesting because it connects the architecture details to concrete 7B OLMo 3 results on long-context benchmarks like RULER and LongBench, framing HiLS as a serious attempt to reach million-token-class context without paying full dense-attention cost. Sources: 1. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling — Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang, 2026 http://arxiv.org/abs/2607.02980 2. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — Ofir Press, Noah A. Smith, Mike Lewis, 2021 https://scholar.google.com/scholar?q=Train+Short,+Test+Long:+Attention+with+Linear+Biases+Enables+Input+Length+Extrapolation 3. Extending Context Window of Large Language Models via Positional Interpolation — Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, 2023 https://scholar.google.com/scholar?q=Extending+Context+Window+of+Large+Language+Models+via+Positional+Interpolation 4. YaRN: Efficient Context Window Extension of Large Language Models — Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole, 2023 https://scholar.google.com/scholar?q=YaRN:+Efficient+Context+Window+Extension+of+Large+Language+Models 5. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024 https://scholar.google.com/scholar?q=RULER:+What's+the+Real+Context+Size+of+Your+Long-Context+Language+Models? 6. Random-Access Infinite Context Length for Transformers — Amirkeivan Mohtashami and Martin Jaggi, 2023 https://scholar.google.com/scholar?q=Random-Access+Infinite+Context+Length+for+Transformers 7. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al., 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention:+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 8. Hardware-Aligned Hierarchical Sparse Attention for Efficient Long-Term Memory Access — Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu, 2026 https://scholar.google.com/scholar?q=Hardware-Aligned+Hierarchical+Sparse+Attention+for+Efficient+Long-Term+Memory+Access 9. DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention — Yuxiang Huang et al., 2026 https://scholar.google.com/scholar?q=DashAttention:+Differentiable+and+Adaptive+Sparse+Hierarchical+Attention 10. Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models — Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, and Yucheng Lu, 2026 https://scholar.google.com/scholar?q=Understanding+and+Improving+Length+Generalization+in+Hierarchical+Sparse+Attention+Models 11. RingAttention with Blockwise Transformers for Near-Infinite Context — Hao Liu, Matei Zaharia, and Pieter Abbeel, 2024 https://scholar.google.com/scholar?q=RingAttention+with+Blockwise+Transformers+for+Near-Infinite+Context 12. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling — Wenhao An et al. (MiniCPM Team), 2026 https://arxiv.org/abs/2602.11761 13. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection — Dongwon Jo et al., 2026 https://arxiv.org/abs/2602.03216 14. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention — Huiqiang Jiang et al., 2024 https://arxiv.org/abs/2407.02490 15. Long-Context Generalization with Sparse Attention — Pavlo Vasylenko, Marcos Treviso, Andre F. T. Martins, 2025 https://arxiv.org/abs/2506.16640 16. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025 https://arxiv.org/abs/2502.00299 17. AI Post Transformers: Long Context Pre-Training with Lighthouse Attention — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-13-long-context-pre-training-with-lighthous-e85bbe.mp3 18. AI Post Transformers: MiniMax Sparse Attention at Million-Token Scale — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-13-minimax-sparse-attention-at-million-toke-300108.mp3 19. AI Post Transformers: MiA-Signature and Global Activation for Long Context — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-13-mia-signature-and-global-activation-for-5ad62f.mp3 20. AI Post Transformers: AllMem for Efficient Long-Context Modeling — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-12-allmem-for-efficient-long-context-modeli-7474db.mp3 21. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp3

  4. قبل يوم واحد

    When Combining Language Models Stops Helping

    This episode explores Josef Chen’s paper on when combining language models actually improves accuracy, focusing on the difference between pairwise error correlation and the more decisive co-failure rate, beta: the chance that every model in a pool fails on the same query. It explains why beta sets a hard ceiling for routing, voting, cascades, and post-training Mixture-of-Agents systems, and why the real gain over a strong single model only exists on queries where that model fails but another succeeds. The discussion walks through results from a 15-model routing setup and a 67-model frontier-model study, showing that even calibrated copula-based estimates systematically understate shared failure and that learned routers capture only a small fraction of the available oracle gain. A listener would find it interesting because it cuts through ensemble hype with a concrete argument about when multi-model orchestration is worth the added cost and complexity, plus a practical way to estimate headroom before building a router at all. Sources: 1. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026 http://arxiv.org/abs/2606.27288 2. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion — Dongfu Jiang, Xiang Ren, Bill Yuchen Lin, 2023 https://scholar.google.com/scholar?q=LLM-Blender:+Ensembling+Large+Language+Models+with+Pairwise+Ranking+and+Generative+Fusion 3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024 https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities 4. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? — Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin, 2025 https://scholar.google.com/scholar?q=Rethinking+Mixture-of-Agents:+Is+Mixing+Different+Large+Language+Models+Beneficial? 5. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models — Josef Chen, 2026 https://scholar.google.com/scholar?q=When+Does+Combining+Language+Models+Help?+A+Co-Failure+Ceiling+on+Routing,+Voting,+and+Mixture-of-Agents+Across+67+Frontier+Models 6. A Unified Approach to Routing and Cascading for LLMs — Jasper Dekoninck, Maximilian Baader, and Martin Vechev, 2024 https://scholar.google.com/scholar?q=A+Unified+Approach+to+Routing+and+Cascading+for+LLMs 7. When Does Confidence-Based Cascade Deferral Suffice? — Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar, 2023 https://scholar.google.com/scholar?q=When+Does+Confidence-Based+Cascade+Deferral+Suffice? 8. Correlated Errors in Large Language Models — Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg, 2025 https://scholar.google.com/scholar?q=Correlated+Errors+in+Large+Language+Models 9. Don't Always Pick the Highest-Performing Model: An Information-Theoretic View of LLM Ensemble Selection — Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu, 2026 https://scholar.google.com/scholar?q=Don't+Always+Pick+the+Highest-Performing+Model:+An+Information-Theoretic+View+of+LLM+Ensemble+Selection 10. PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier — Yuhua Jiang et al., 2025 https://arxiv.org/abs/2506.10406 11. S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ruotian Ma et al., 2025 https://arxiv.org/abs/2502.12853 12. Small Language Models Need Strong Verifiers to Self-Correct Reasoning — Yunxiang Zhang et al., 2024 https://arxiv.org/abs/2404.17140 13. CP-Router: An Uncertainty-Aware Router Between LLM and LRM — Jiayuan Su et al., 2025 https://arxiv.org/abs/2505.19970 14. Leveraging Uncertainty Estimation for Efficient LLM Routing — Tuo Zhang et al., 2025 https://arxiv.org/abs/2502.11021 15. Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization — Yu-Neng Chuang et al., 2025 https://arxiv.org/abs/2502.04428 16. Wisdom and Delusion of LLM Ensembles for Code Generation and Repair — Fernando Vallecillos Ruiz et al., 2025 https://arxiv.org/abs/2510.21513 17. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang et al., 2026 https://arxiv.org/abs/2602.03794 18. LLM Chemistry Estimation for Multi-LLM Recommendation — Huascar Sanchez and Briland Hitaj, 2025 https://arxiv.org/abs/2510.03930 19. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp3 20. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp3

  5. قبل يوم واحد

    Why GLU Variants Improve Transformer Feed-Forward Layers

    This episode explores Noam Shazeer’s 2020 paper on replacing the Transformer’s standard feed-forward network with gated linear unit variants such as GLU, Bilinear, ReGLU, GEGLU, and SwiGLU. It explains why this seemingly small change matters, walking through the role of the per-token MLP in a Transformer and how multiplicative gating can change feature processing without altering the broader encoder-decoder architecture. The discussion focuses on the paper’s T5-style sequence-to-sequence setup, including span-corruption pretraining on C4, and on the key methodological choice to shrink gated-layer width so parameter count and FLOPs stay roughly matched with the baseline. Listeners would find it interesting because the episode connects a clean, tightly controlled ablation to a design idea that later had an outsized influence on modern Transformer architectures, while also highlighting the limits of what the experiment actually proves. Sources: 1. GLU Variants Improve Transformer — Noam Shazeer, 2020 http://arxiv.org/abs/2002.05202 2. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016 https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks 3. GLU Variants Improve Transformer — Noam Shazeer, 2020 https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer 4. PaLM: Scaling Language Modeling with Pathways — Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al., 2022 https://scholar.google.com/scholar?q=PaLM:+Scaling+Language+Modeling+with+Pathways 5. Gemma 2: Improving Open Language Models at a Practical Size — Gemma Team, including Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and collaborators, 2024 https://scholar.google.com/scholar?q=Gemma+2:+Improving+Open+Language+Models+at+a+Practical+Size 6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Colin Raffel et al., 2019 https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer 7. Do Transformer Modifications Transfer Across Implementations and Applications? — Sharan Narang et al., 2021 https://scholar.google.com/scholar?q=Do+Transformer+Modifications+Transfer+Across+Implementations+and+Applications? 8. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021 https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories 9. Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers — Zihan Qiu, Zeyu Huang, Youcheng Huang, Jie Fu, 2024 https://scholar.google.com/scholar?q=Empirical+Study+on+Updating+Key-Value+Memories+in+Transformer+Feed-forward+Layers 10. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024 https://scholar.google.com/scholar?q=ReLU^2+Wins:+Discovering+Efficient+Activation+Functions+for+Sparse+LLMs 11. Spark Transformer: Reactivating Sparsity in FFN and Attention — Chong You et al., 2025 https://scholar.google.com/scholar?q=Spark+Transformer:+Reactivating+Sparsity+in+FFN+and+Attention 12. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 13. AI Post Transformers: RoBERTa: Robustly Optimized BERT Pretraining Approach — Hal Turing & Dr. Ada Shannon, Wed, https://podcast.do-not-panic.com/episodes/roberta-robustly-optimized-bert-pretraining-approach/ 14. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp3 15. AI Post Transformers: Unified Neural Scaling Laws Across Regimes — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-07-unified-neural-scaling-laws-across-regim-292e2d.mp3

  6. قبل يومين

    SiLU Activations for Replay-Free Atari Reinforcement Learning

    This episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to unpack older RL concepts like function approximation, TD(lambda), and on-policy learning for listeners coming from modern deep learning. The discussion walks through the paper’s results on SZ-Tetris, 10x10 Tetris, and Atari-style settings, highlighting that dSiLU and mixed SiLU/dSiLU networks outperformed ReLU-based alternatives in several configurations. Listeners would find it interesting because it challenges the idea that replay buffers and DQN-style machinery are the only serious path for high-dimensional RL, and shows how a seemingly small architectural choice can reshape learning dynamics. Sources: 1. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017 http://arxiv.org/abs/1702.03118 2. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units — Dan Hendrycks, Kevin Gimpel, 2016 https://scholar.google.com/scholar?q=Bridging+Nonlinearities+and+Stochastic+Regularizers+with+Gaussian+Error+Linear+Units 3. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017 https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning 4. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017 https://scholar.google.com/scholar?q=Searching+for+Activation+Functions 5. GLU Variants Improve Transformer — Noam Shazeer, 2020 https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer 6. Learning to Predict by the Methods of Temporal Differences — Richard S. Sutton, 1988 https://scholar.google.com/scholar?q=Learning+to+Predict+by+the+Methods+of+Temporal+Differences 7. True Online Temporal-Difference Learning — Harm van Seijen, A. Rupam Mahmood, Patrick M. Pilarski, Marlos C. Machado, Richard S. Sutton, 2015 https://scholar.google.com/scholar?q=True+Online+Temporal-Difference+Learning 8. High-Dimensional Continuous Control Using Generalized Advantage Estimation — John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015 https://scholar.google.com/scholar?q=High-Dimensional+Continuous+Control+Using+Generalized+Advantage+Estimation 9. Multi-step Reinforcement Learning: A Unifying Algorithm — Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, Richard S. Sutton, 2017 https://scholar.google.com/scholar?q=Multi-step+Reinforcement+Learning:+A+Unifying+Algorithm 10. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013 https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning 11. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016 https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning 12. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017 https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms 13. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022 https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback 14. Human-level control through deep reinforcement learning — Volodymyr Mnih et al., 2015 https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning 15. Deep reinforcement learning with double q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2015 https://scholar.google.com/scholar?q=Deep+reinforcement+learning+with+double+q-learning 16. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2016 https://scholar.google.com/scholar?q=Prioritized+Experience+Replay 17. High-dimensional function approximation for knowledge-free reinforcement learning: a case study in SZ-Tetris — Wojciech Jaskowski, Maciej Szubert, Pawel Liskowski, Krzysztof Krawiec, 2015 https://scholar.google.com/scholar?q=High-dimensional+function+approximation+for+knowledge-free+reinforcement+learning:+a+case+study+in+SZ-Tetris 18. Approximate dynamic programming finally performs well in the game of tetris — Victor Gabillon, Mohammad Ghavamzadeh, Bruno Scherrer, 2013 https://scholar.google.com/scholar?q=Approximate+dynamic+programming+finally+performs+well+in+the+game+of+tetris 19. Replay across Experiments: A Natural Extension of Off-Policy RL — Dhruva Tirumala et al., 2023 https://arxiv.org/abs/2311.15951 20. Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning — Theo Vincent et al., 2024 https://arxiv.org/abs/2405.16195 21. A Survey of Temporal Credit Assignment in Deep Reinforcement Learning — Eduardo Pignatelli et al., 2023 https://arxiv.org/abs/2312.01072 22. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3 23. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3

  7. قبل يومين

    Swish: Self-Gated Activation Beyond ReLU

    This episode explores the 2017 Swish paper and asks whether a simple self-gated activation, `x * sigmoid(x)`, can outperform ReLU without changing the surrounding network architecture. It explains why activation functions matter for gradient flow and deep optimization, focusing on Swish’s smooth, non-monotonic behavior and its ability to attenuate rather than discard negative inputs. The discussion walks through results on CIFAR, ImageNet, and machine translation, highlighting modest but real gains in deeper vision models, including roughly 0.9-point and 0.6-point improvements on ImageNet benchmarks. It also gives a critical read of the evidence, noting that Swish is not a universal win and raises practical questions around tuning, sparsity, hardware efficiency, compression, and whether its legacy matters more as part of broader gating mechanisms than as a standalone ReLU replacement. Sources: 1. Swish: a Self-Gated Activation Function — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017 http://arxiv.org/abs/1710.05941v1 2. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015 https://scholar.google.com/scholar?q=Delving+Deep+into+Rectifiers:+Surpassing+Human-Level+Performance+on+ImageNet+Classification 3. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) — Djork-Arne Clevert, Thomas Unterthiner, Sepp Hochreiter, 2015 https://scholar.google.com/scholar?q=Fast+and+Accurate+Deep+Network+Learning+by+Exponential+Linear+Units+(ELUs) 4. Gaussian Error Linear Units (GELUs) — Dan Hendrycks, Kevin Gimpel, 2016 https://scholar.google.com/scholar?q=Gaussian+Error+Linear+Units+(GELUs) 5. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017 https://scholar.google.com/scholar?q=Searching+for+Activation+Functions 6. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2016 https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks 7. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017 https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning 8. GLU Variants Improve Transformer — Noam Shazeer, 2020 https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer 9. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017 https://scholar.google.com/scholar?q=Attention+Is+All+You+Need 10. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models — Iman Mirzadeh et al., 2023 https://arxiv.org/abs/2310.04564 11. ReLU^2 Wins: Discovering Efficient Activation Functions for Sparse LLMs — Zhengyan Zhang et al., 2024 https://arxiv.org/abs/2402.03804 12. Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts — Huy Nguyen, Nhat Ho, Alessandro Rinaldo, 2024 https://arxiv.org/abs/2405.13997 13. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free — Zihan Qiu et al., 2025 https://arxiv.org/abs/2505.06708 14. A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU — Andrea Belano et al., 2024 https://arxiv.org/abs/2412.06321 15. AI Post Transformers: Adam: A Method for Stochastic Optimization — Hal Turing & Dr. Ada Shannon, 2025 https://podcast.do-not-panic.com/episodes/adam-a-method-for-stochastic-optimization/ 16. AI Post Transformers: PALOMA: Benchmarking Language Model Fit Across Domains — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-23-paloma-benchmarking-language-model-fit-a-360060.mp3

  8. قبل يومين

    Verbalizable Representations and the Global Workspace

    This episode explores Anthropic’s paper on whether language models contain a privileged “verbalizable” subspace, functionally similar to a global workspace, whose contents can be reported, reasoned over, and deliberately controlled. It draws a clear line between access consciousness and phenomenal consciousness, then focuses on the paper’s mechanistic proposal: a Jacobian-based “J-space” that identifies internal directions causally poised to become language rather than merely easy to decode. The discussion highlights intervention results showing that swapping or ablating directions such as France/China or Soccer/Rugby changes later reports and multi-step reasoning, with broader examples in code bug detection, prompt-injection recognition, and protein-function judgments. Listeners would find it interesting because it turns a consciousness-adjacent question into a concrete engineering argument about whether models have a small, reusable internal workspace that shapes what they know, say, and do. Sources: 1. Verbalizable Representations and the Global Workspace https://transformer-circuits.pub/2026/workspace/index.html 2. Verbalizable Representations Form a Global Workspace in Language Models — Wes Gurnee, Nicholas Sofroniew, Jack Lindsey, Adam Pearce, Mateusz Piotrowski, et al., 2026 https://scholar.google.com/scholar?q=Verbalizable+Representations+Form+a+Global+Workspace+in+Language+Models 3. Eliciting Latent Predictions from Transformers with the Tuned Lens — Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, Jacob Steinhardt, 2023 https://scholar.google.com/scholar?q=Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens 4. Sparse Autoencoders Find Highly Interpretable Features in Language Models — Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, Lee Sharkey, 2023 https://scholar.google.com/scholar?q=Sparse+Autoencoders+Find+Highly+Interpretable+Features+in+Language+Models 5. Do Activation Verbalization Methods Convey Privileged Information? — Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, Byron C. Wallace, 2026 https://scholar.google.com/scholar?q=Do+Activation+Verbalization+Methods+Convey+Privileged+Information? 6. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, Tom Henighan, 2026 https://scholar.google.com/scholar?q=Scaling+Monosemanticity:+Extracting+Interpretable+Features+from+Claude+3+Sonnet 7. Implicit Representations of Meaning in Neural Language Models — Belinda Z. Li, Maxwell Nye, Jacob Andreas, 2021 https://scholar.google.com/scholar?q=Implicit+Representations+of+Meaning+in+Neural+Language+Models 8. On the Biology of a Large Language Model — Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al., 2025 https://scholar.google.com/scholar?q=On+the+Biology+of+a+Large+Language+Model 9. A Neuronal Model of a Global Workspace in Effortful Cognitive Tasks — Stanislas Dehaene, Serge Kerszberg, Jean-Pierre Changeux, 1998 https://scholar.google.com/scholar?q=A+Neuronal+Model+of+a+Global+Workspace+in+Effortful+Cognitive+Tasks 10. Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding — Haolin Chen et al., 2024 https://scholar.google.com/scholar?q=Language+Models+are+Hidden+Reasoners:+Unlocking+Latent+Reasoning+Capabilities+via+Self-Rewarding 11. Efficient Post-Training Refinement of Latent Reasoning in Large Language Models — Xinyuan Wang et al., 2025 https://scholar.google.com/scholar?q=Efficient+Post-Training+Refinement+of+Latent+Reasoning+in+Large+Language+Models 12. SeLaR: Selective Latent Reasoning in Large Language Models — Renyu Fu and Guibo Luo, 2026 https://scholar.google.com/scholar?q=SeLaR:+Selective+Latent+Reasoning+in+Large+Language+Models 13. Measuring Faithfulness in Chain-of-Thought Reasoning — Tamera Lanham et al., 2023 https://scholar.google.com/scholar?q=Measuring+Faithfulness+in+Chain-of-Thought+Reasoning 14. Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps — Martin Tutek et al., 2025 https://scholar.google.com/scholar?q=Measuring+Chain+of+Thought+Faithfulness+by+Unlearning+Reasoning+Steps 15. Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models — Richard J. Young, 2026 https://scholar.google.com/scholar?q=Why+Models+Know+But+Don't+Say:+Chain-of-Thought+Faithfulness+Divergence+Between+Thinking+Tokens+and+Answers+in+Open-Weight+Reasoning+Models 16. Steering Language Models With Activation Engineering — Alexander Matt Turner et al., 2023 https://scholar.google.com/scholar?q=Steering+Language+Models+With+Activation+Engineering 17. Improving Instruction-Following in Language Models through Activation Steering — Alessandro Stolfo et al., 2024 https://scholar.google.com/scholar?q=Improving+Instruction-Following+in+Language+Models+through+Activation+Steering 18. Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering — Marco Valentino et al., 2025 https://scholar.google.com/scholar?q=Mitigating+Content+Effects+on+Reasoning+in+Language+Models+through+Fine-Grained+Activation+Steering 19. AI Post Transformers: How Models Detect Hidden Activation Steering — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-08-how-models-detect-hidden-activation-stee-577f73.mp3 20. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp3 21. AI Post Transformers: Why Transformers Fail at Counting — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-08-why-transformers-fail-at-counting-137924.mp3 22. AI Post Transformers: RAPTOR: Stable Concept Directions From Logistic Probes — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-08-raptor-stable-concept-directions-from-lo-b37365.mp3 Interactive Visualization: Verbalizable Representations and the Global Workspace

مشاهدة الكل (٧٦٠)

حول

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

المعلومات

  • صناع العمل
    mcgrof
  • سنوات النشاط
    ٢٠٢٥ - ٢٠٢٦
  • الحلقات
    ٧٦٠
  • التقييم
    ملائم
  • موقع البرنامج على الويب
    AI Post Transformers

قد يعجبك أيضًا