AI Post Transformers

mcgrof
  • 3.7 (3)
  • Technology
  • Updated Daily

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

  1. 1d ago

    Semantic Cache Distillation: Solving Semantic Drift in KV Cache Transfer

    This episode examines Semantic Cache Distillation, a technique for reusing KV caches across producer and consumer transformers that share architecture but have different fine-tuned weights. The discussion covers why prefill-decode disaggregation splits compute-bound and memory-bandwidth-bound phases across separate machines, and how naively shipping raw or compressed KV caches between differently-weighted models causes "semantic drift" — a small per-layer mismatch that compounds through deep residual networks and degrades generation quality. The hosts unpack the paper's REUSE mechanism, which uses paired producer-consumer KV traces and low-rank SVD factorization to build a shared latent code, letting a lightweight encoder-decoder pair reconstruct usable cache states instead of forcing a full recompute. Real-world motivations include LoRA-adapter fleets sharing a base model and draft-verifier pairs in speculative decoding. Listeners interested in LLM serving infrastructure will find the reported 2.65x time-to-first-token speedup, and the underlying cross-model cache reconstruction problem, a concrete look at an underexplored bottleneck in production inference systems. Sources: 1. Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching — Qianli Ma, Zhiqing Tang, Hanshuai Cui, Zhi Yao, Weijia Jia, 2026 http://arxiv.org/abs/2606.07684 2. S-LoRA: Serving Thousands of Concurrent LoRA Adapters — Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica, 2024 https://scholar.google.com/scholar?q=S-LoRA%3A+Serving+Thousands+of+Concurrent+LoRA+Adapters 3. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 4. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+with+Cached+Knowledge+Fusion 5. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 6. DroidSpeak: KV cache sharing for cross-LLM communication and multi-LLM serving — Liu, Y., Huang, Y., Yao, J., Feng, S., Gu, Z., Du, K., Li, H., Cheng, Y., Jiang, J., Lu, S., et al., 2024 https://scholar.google.com/scholar?q=DroidSpeak%3A+KV+cache+sharing+for+cross-LLM+communication+and+multi-LLM+serving 7. Cache-to-Cache: Direct semantic communication between large language models — Fu, T., Min, Z., Zhang, H., Yan, J., Dai, G., Ouyang, W., and Wang, Y., 2026 https://scholar.google.com/scholar?q=Cache-to-Cache%3A+Direct+semantic+communication+between+large+language+models 8. S-LoRA: Scalable serving of thousands of LoRA adapters — Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al., 2024 https://scholar.google.com/scholar?q=S-LoRA%3A+Scalable+serving+of+thousands+of+LoRA+adapters 9. EAGLE: Speculative sampling requires rethinking feature uncertainty — Li, Y., Wei, F., Zhang, C., and Zhang, H., 2024 https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+sampling+requires+rethinking+feature+uncertainty 10. Mooncake: A KVCache-centric disaggregated architecture for LLM serving — Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X., 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+disaggregated+architecture+for+LLM+serving Interactive Visualization: Semantic Cache Distillation: Solving Semantic Drift in KV Cache Transfer

  2. 2d ago

    Deep Drone Acrobatics: Vision-Only Zero-Shot Sim-to-Real Flight

    This episode explores "Deep Drone Acrobatics," which trains a quadrotor to fly extreme maneuvers — a Power Loop, Barrel Roll, and Matty Flip — using only an onboard camera and IMU, with no external motion capture. The discussion centers on how the policy is trained entirely in simulation via DAgger imitation learning, where a privileged model-predictive controller with perfect ground-truth state acts as an expert that a vision-limited student imitates, rather than through reinforcement learning or reward shaping. A key focus is the sim-to-real gap: at high accelerations, motion blur degrades vision-based state estimation, so the paper's "input abstraction" approach feeds the network geometry-based feature tracks instead of raw pixels, drawing on prior work showing that shared abstractions between simulated and real observations shrink the performance gap. The conversation also traces the paper's intellectual lineage, connecting it to "Does Computer Vision Matter for Action?" and "Learning by Cheating," while highlighting why acrobatic flight is a harder version of the sim-to-real problem than driving, since a flipping drone has no margin for hesitation. Listeners interested in robotics, sim-to-real transfer, or imitation learning will find a concrete, technically grounded case study of zero-shot policy transfer under extreme physical constraints. Sources: 1. Deep Drone Acrobatics: Vision-Only Zero-Shot Sim-to-Real Flight https://roboticsproceedings.org/rss16/p040.pdf 2. Learning by Cheating — Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl, 2019 https://scholar.google.com/scholar?q=Learning+by+Cheating 3. Deep Drone Racing: From Simulation to Reality with Domain Randomization — Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, Davide Scaramuzza, 2020 https://scholar.google.com/scholar?q=Deep+Drone+Racing%3A+From+Simulation+to+Reality+with+Domain+Randomization 4. Does computer vision matter for action? — Brady Zhou, Philipp Krähenbühl, Vladlen Koltun, 2019 https://scholar.google.com/scholar?q=Does+computer+vision+matter+for+action%3F 5. Driving Policy Transfer via Modularity and Abstraction — Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, Vladlen Koltun, 2018 https://scholar.google.com/scholar?q=Driving+Policy+Transfer+via+Modularity+and+Abstraction 6. Agile Autonomous Driving Using End-to-End Deep Imitation Learning — Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, Byron Boots, 2018 https://scholar.google.com/scholar?q=Agile+Autonomous+Driving+Using+End-to-End+Deep+Imitation+Learning 7. A reduction of imitation learning and structured prediction to no-regret online learning — Stéphane Ross, Geoffrey Gordon, Drew Bagnell, 2011 https://scholar.google.com/scholar?q=A+reduction+of+imitation+learning+and+structured+prediction+to+no-regret+online+learning Interactive Visualization: Deep Drone Acrobatics: Vision-Only Zero-Shot Sim-to-Real Flight

  3. 2d ago

    Imitative Reinforcement Learning for UCAV Pursuit-Lock-Launch Dogfights

    This episode examines "An Imitative Reinforcement Learning Framework for Pursuit-Lock-Launch Missions," which trains an unmanned combat aerial vehicle to complete a three-stage dogfighting task by blending TD3-style actor-critic reinforcement learning with a behavior-cloning term drawn from expert trajectories generated in the authors' own simulator. The discussion covers why sparse-reward, multistage combat tasks make good RL benchmarks despite the setting, the classic tradeoffs between pure reinforcement learning (sample inefficiency) and pure imitation learning (compounding error and drifting off-distribution), and how combining both aims to get faster, more reliable learning than either alone. It also flags a notable gap in the paper: it never benchmarks against DAgger, the standard fix for imitation learning's distribution-shift problem, raising open questions about whether the reported near-100% success rate reflects a genuinely better architecture or simply a weak baseline comparison. Listeners interested in robotics, RL/imitation-learning hybrids, or how combat-style testbeds get used for general control research will find the critique of the experimental design as engaging as the headline results. Sources: 1. An Imitative Reinforcement Learning Framework for Pursuit-Lock-Launch Missions — Siyuan Li, Rongchang Zuo, Bofei Liu, Yaoyu He, Peng Liu, Yingnan Zhao, 2024 http://arxiv.org/abs/2406.11562 2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning — Stéphane Ross, Geoffrey Gordon, Drew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning 3. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning 4. End to End Learning for Self-Driving Cars — Mariusz Bojarski et al. (NVIDIA), 2016 https://scholar.google.com/scholar?q=End+to+End+Learning+for+Self-Driving+Cars 5. Autonomous Air Combat Maneuvering Decision Making with Deep Reinforcement Learning — Wang et al. (multiple independent groups have published under similar titles), 2019 https://scholar.google.com/scholar?q=Autonomous+Air+Combat+Maneuvering+Decision+Making+with+Deep+Reinforcement+Learning 6. Alpha Dogfight Trials public results and analysis (DARPA program summaries) — DARPA / Heron Systems and other competing teams, 2020 https://scholar.google.com/scholar?q=Alpha+Dogfight+Trials+public+results+and+analysis+%28DARPA+program+summaries%29 7. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model — Julian Schrittwieser et al. (DeepMind), 2020 https://scholar.google.com/scholar?q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model 8. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine, 2018 https://scholar.google.com/scholar?q=Soft+Actor-Critic%3A+Off-Policy+Maximum+Entropy+Deep+Reinforcement+Learning+with+a+Stochastic+Actor 9. Deep Q-learning from Demonstrations — Todd Hester et al. (DeepMind), 2018 https://scholar.google.com/scholar?q=Deep+Q-learning+from+Demonstrations 10. A Minimalist Approach to Offline Reinforcement Learning (TD3+BC) — Scott Fujimoto, Shixiang Shane Gu, 2021 https://scholar.google.com/scholar?q=A+Minimalist+Approach+to+Offline+Reinforcement+Learning+%28TD3%2BBC%29 11. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey Gordon, Drew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29 12. Discovering Expert-Level Air Combat Knowledge via Deep Excitatory-Inhibitory Factorized Reinforcement Learning — Hai Yin Piao, Shengqi Yang, Hechang Chen, et al., 2024 https://scholar.google.com/scholar?q=Discovering+Expert-Level+Air+Combat+Knowledge+via+Deep+Excitatory-Inhibitory+Factorized+Reinforcement+Learning 13. Multi-Dimensional Decision-Making for UAV Air Combat Based on Hierarchical Reinforcement Learning — Jiandong Zhang, Dinghan Wang, Qiming Yang, et al., 2023 https://scholar.google.com/scholar?q=Multi-Dimensional+Decision-Making+for+UAV+Air+Combat+Based+on+Hierarchical+Reinforcement+Learning Interactive Visualization: Imitative Reinforcement Learning for UCAV Pursuit-Lock-Launch Dogfights

  4. 2d ago

    Programmatically Interpretable Reinforcement Learning: Readable Policies

    This episode examines Programmatically Interpretable Reinforcement Learning (PIRL), a 2018 framework from Rice University, Google Brain, and DeepMind researchers that forces RL policies to be expressed as short, human-readable programs rather than opaque neural network weights. The discussion centers on why formal verification—proving properties like bounded steering output in a self-driving car—is tractable for small domain-specific programs but essentially impossible for networks with millions of parameters. Using the paper's driving example, the hosts unpack "policy sketches" (a switch statement branching on track position, with PID controllers filling each branch) and Neurally Directed Program Search (NDPS), which trains a conventional deep RL policy as an oracle and then searches program space to imitate its outputs via smooth regression rather than fighting a jagged, non-differentiable reward landscape. They draw out the connection to DAgger's iterative imitation-learning approach from Ross, Gordon, and Bagnell, while flagging a subtle mismatch between matching an expert's actions and matching reward through an imitation proxy. Listeners interested in AI safety, control theory, or the tension between interpretability and performance will find the concrete TORCS driving case a clear entry point into verifiable reinforcement learning. Sources: 1. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018 http://arxiv.org/abs/1804.02477 2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning — Stéphane Ross, Geoffrey Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning 3. Is Imitation Learning the Route to Humanoid Robots? — Stefan Schaal, 1999 https://scholar.google.com/scholar?q=Is+Imitation+Learning+the+Route+to+Humanoid+Robots%3F 4. ALVINN: An Autonomous Land Vehicle in a Neural Network — Dean Pomerleau, 1989 https://scholar.google.com/scholar?q=ALVINN%3A+An+Autonomous+Land+Vehicle+in+a+Neural+Network 5. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning 6. Verifiable Reinforcement Learning via Policy Extraction — Osbert Bastani, Yewen Pu, Armando Solar-Lezama, 2018 https://scholar.google.com/scholar?q=Verifiable+Reinforcement+Learning+via+Policy+Extraction 7. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018 https://scholar.google.com/scholar?q=Programmatically+Interpretable+Reinforcement+Learning 8. Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning — Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, Sung-Hyun Son, 2020 https://scholar.google.com/scholar?q=Optimization+Methods+for+Interpretable+Differentiable+Decision+Trees+Applied+to+Reinforcement+Learning 9. Distilling a Neural Network Into a Soft Decision Tree — Nicholas Frosst, Geoffrey Hinton, 2017 https://scholar.google.com/scholar?q=Distilling+a+Neural+Network+Into+a+Soft+Decision+Tree 10. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29 11. Continuous Control with Deep Reinforcement Learning (DDPG) — Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2015 https://scholar.google.com/scholar?q=Continuous+Control+with+Deep+Reinforcement+Learning+%28DDPG%29 12. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks — Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, Mykel J. Kochenderfer, 2017 https://scholar.google.com/scholar?q=Reluplex%3A+An+Efficient+SMT+Solver+for+Verifying+Deep+Neural+Networks 13. The Sketching Approach to Program Synthesis — Armando Solar-Lezama, 2009 https://scholar.google.com/scholar?q=The+Sketching+Approach+to+Program+Synthesis 14. Syntax-Guided Synthesis (SyGuS) — Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Ghila Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, Abhishek Udupa, 2015 https://scholar.google.com/scholar?q=Syntax-Guided+Synthesis+%28SyGuS%29 Interactive Visualization: Programmatically Interpretable Reinforcement Learning: Readable Policies

  5. 3d ago

    Guided Policy Search: Teaching Neural Nets via Trajectory Optimization

    This episode examines Guided Policy Search, a 2013 method from Sergey Levine and Vladlen Koltun that lets flexible neural-network policies control robots without falling into the poor local optima that plague direct policy search over high-dimensional parameter spaces. The discussion traces the paper's teacher-student structure: differential dynamic programming (DDP), a model-based trajectory optimizer rooted in 1960s optimal control theory, generates high-reward example trajectories for specific starting conditions, and the neural-network student learns to match and generalize this behavior via policy gradients and importance sampling rather than naive imitation. A key distinction drawn out is why this differs from imitation learning approaches like DAGGER — DDP's guidance is only locally valid, so the method needs an objective built to maximize reward everywhere, not just mimic a narrow expert trajectory. The conversation connects DDP's backward pass to Bellman recursion and the broader LQR/Kalman-filter lineage, and explains how importance sampling lets the same batch of guiding samples be reused across many gradient steps, which matters when real-hardware data collection is expensive. Listeners interested in the historical roots of modern reinforcement learning — and how classical control theory was fused with neural networks years before this became standard practice — will find the episode's walkthrough of the underlying mechanics clarifying. Sources: 1. Guided Policy Search: Teaching Neural Nets via Trajectory Optimization https://proceedings.mlr.press/v28/levine13.pdf 2. Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics — Sergey Levine, Pieter Abbeel, 2014 https://scholar.google.com/scholar?q=Learning+Neural+Network+Policies+with+Guided+Policy+Search+under+Unknown+Dynamics 3. End-to-End Training of Deep Visuomotor Policies — Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel, 2016 https://scholar.google.com/scholar?q=End-to-End+Training+of+Deep+Visuomotor+Policies 4. Trust Region Policy Optimization — John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, Pieter Abbeel, 2015 https://scholar.google.com/scholar?q=Trust+Region+Policy+Optimization 5. Reinforcement Learning of Motor Skills with Policy Gradients — Jan Peters, Stefan Schaal, 2008 https://scholar.google.com/scholar?q=Reinforcement+Learning+of+Motor+Skills+with+Policy+Gradients 6. Differential Dynamic Programming — David Jacobson, David Mayne, 1970 https://scholar.google.com/scholar?q=Differential+Dynamic+Programming 7. A Generalized Iterative LQG Method for Locally-Optimal Feedback Control of Constrained Nonlinear Stochastic Systems — Emanuel Todorov, Weiwei Li, 2005 https://scholar.google.com/scholar?q=A+Generalized+Iterative+LQG+Method+for+Locally-Optimal+Feedback+Control+of+Constrained+Nonlinear+Stochastic+Systems 8. Synthesis and Stabilization of Complex Behaviors through Online Trajectory Optimization — Yuval Tassa, Tom Erez, Emanuel Todorov, 2012 https://scholar.google.com/scholar?q=Synthesis+and+Stabilization+of+Complex+Behaviors+through+Online+Trajectory+Optimization 9. Aggressive Driving with Model Predictive Path Integral Control — Grady Williams, Paul Drews, Brian Goldfain, James Rehg, Evangelos Theodorou, 2016 https://scholar.google.com/scholar?q=Aggressive+Driving+with+Model+Predictive+Path+Integral+Control 10. Eligibility Traces for Off-Policy Policy Evaluation — Doina Precup, Richard Sutton, Satinder Singh, 2000 https://scholar.google.com/scholar?q=Eligibility+Traces+for+Off-Policy+Policy+Evaluation 11. Learning from Scarce Experience — Leonid Peshkin, Christian Shelton, 2002 https://scholar.google.com/scholar?q=Learning+from+Scarce+Experience 12. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017 https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms 13. PILCO: A Model-Based and Data-Efficient Approach to Policy Search — Deisenroth, M. and Rasmussen, C., 2011 https://scholar.google.com/scholar?q=PILCO%3A+A+Model-Based+and+Data-Efficient+Approach+to+Policy+Search 14. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAGGER) — Ross, S., Gordon, G., and Bagnell, A., 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAGGER%29 15. On a Connection Between Importance Sampling and the Likelihood Ratio Policy Gradient — Tang, J. and Abbeel, P., 2010 https://scholar.google.com/scholar?q=On+a+Connection+Between+Importance+Sampling+and+the+Likelihood+Ratio+Policy+Gradient 16. Approximately Optimal Approximate Reinforcement Learning — Kakade, S. and Langford, J., 2002 https://scholar.google.com/scholar?q=Approximately+Optimal+Approximate+Reinforcement+Learning 17. SIMBICON: Simple Biped Locomotion Control — Yin, K., Loken, K., and van de Panne, M., 2007 https://scholar.google.com/scholar?q=SIMBICON%3A+Simple+Biped+Locomotion+Control Interactive Visualization: Guided Policy Search: Teaching Neural Nets via Trajectory Optimization

  6. 3d ago

    Hierarchical RL Beats an F-16 Instructor Pilot 5-0

    This episode examines "Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials," in which the PHANG-MAN agent swept a graduate of the USAF Weapons Instructor Course 5-0 in simulated dogfighting. The discussion traces the DARPA ACE program's rationale for building trust incrementally toward AI-assisted piloted aircraft, and contrasts this work with prior systems like Nick Ernest's genetic fuzzy tree ALPHA, highlighting how PHANG-MAN operates with genuinely continuous stick-and-rudder control in the high-fidelity JSBSim F-16 simulator rather than a maneuver library. It unpacks the two-layer architecture — three frozen, independently-trained low-level Soft Actor-Critic specialist policies (Control Zone, Aggressive Shooter, Conservative Shooter) governed by a higher-frequency policy selector — and explains supporting concepts like curriculum learning and maximum-entropy RL that make the training tractable. Listeners interested in reinforcement learning architecture, autonomous systems trust-building, or the gap between simulated and real-world control will find the technical breakdown of temporally-extended specialist routing especially compelling. Sources: 1. Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials — Adrian P. Pope, Jaime S. Ide, Daria Micovic, Henry Diaz, David Rosenbluth, Lee Ritholtz, Jason C. Twedt, Thayne T. Walker, Kevin Alcedo, Daniel Javorsek, 2021 http://arxiv.org/abs/2105.00990 2. Feudal Reinforcement Learning — Peter Dayan, Geoffrey Hinton, 1993 https://scholar.google.com/scholar?q=Feudal+Reinforcement+Learning 3. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition — Thomas G. Dietterich, 2000 https://scholar.google.com/scholar?q=Hierarchical+Reinforcement+Learning+with+the+MAXQ+Value+Function+Decomposition 4. FeUdal Networks for Hierarchical Reinforcement Learning — Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, Koray Kavukcuoglu, 2017 https://scholar.google.com/scholar?q=FeUdal+Networks+for+Hierarchical+Reinforcement+Learning 5. The Option-Critic Architecture — Pierre-Luc Bacon, Jean Harb, Doina Precup, 2017 https://scholar.google.com/scholar?q=The+Option-Critic+Architecture 6. Curriculum Learning — Yoshua Bengio, Jerome Louradour, Ronan Collobert, Jason Weston, 2009 https://scholar.google.com/scholar?q=Curriculum+Learning 7. Reverse Curriculum Generation for Reinforcement Learning — Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, Pieter Abbeel, 2017 https://scholar.google.com/scholar?q=Reverse+Curriculum+Generation+for+Reinforcement+Learning 8. Automatic Goal Generation for Reinforcement Learning Agents — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018 https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents 9. Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI (Christopher Berner, Greg Brockman, Brooke Chan, et al.), 2019 https://scholar.google.com/scholar?q=Dota+2+with+Large+Scale+Deep+Reinforcement+Learning 10. Reinforcement Learning with Deep Energy-Based Policies — Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine, 2017 https://scholar.google.com/scholar?q=Reinforcement+Learning+with+Deep+Energy-Based+Policies 11. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine, 2018 https://scholar.google.com/scholar?q=Soft+Actor-Critic%3A+Off-Policy+Maximum+Entropy+Deep+Reinforcement+Learning+with+a+Stochastic+Actor 12. Soft Actor-Critic Algorithms and Applications — Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine, 2018 https://scholar.google.com/scholar?q=Soft+Actor-Critic+Algorithms+and+Applications 13. Maximum Entropy Inverse Reinforcement Learning — Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, Anind K. Dey, 2008 https://scholar.google.com/scholar?q=Maximum+Entropy+Inverse+Reinforcement+Learning 14. Meta Learning Shared Hierarchies — K. Frans, J. Ho, X. Chen, P. Abbeel, J. Schulman, 2018 https://scholar.google.com/scholar?q=Meta+Learning+Shared+Hierarchies 15. Data-Efficient Hierarchical Reinforcement Learning (HIRO) — O. Nachum, S. Gu, H. Lee, S. Levine, 2018 https://scholar.google.com/scholar?q=Data-Efficient+Hierarchical+Reinforcement+Learning+%28HIRO%29 16. Genetic Fuzzy based Artificial Intelligence for Unmanned Combat Aerial Vehicle Control in Simulated Air Combat Missions — N. Ernest, D. Carroll, C. Schumacher, M. Clark, K. Cohen, G. Lee, 2016 https://scholar.google.com/scholar?q=Genetic+Fuzzy+based+Artificial+Intelligence+for+Unmanned+Combat+Aerial+Vehicle+Control+in+Simulated+Air+Combat+Missions 17. Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play — Z. Sun, H. Piao, Z. Yang, Y. Zhao, et al., 2021 https://scholar.google.com/scholar?q=Multi-agent+hierarchical+policy+gradient+for+air+combat+tactics+emergence+via+self-play 18. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World — J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, P. Abbeel, 2017 https://scholar.google.com/scholar?q=Domain+Randomization+for+Transferring+Deep+Neural+Networks+from+Simulation+to+the+Real+World Interactive Visualization: Hierarchical RL Beats an F-16 Instructor Pilot 5-0

  7. 3d ago

    MPC-Net: Learning Optimal Control via the Hamiltonian

    This episode examines MPC-Net, a 2019/2020 paper from ETH Zürich's Robotic Systems Lab that trains a fast neural policy to replace expensive model predictive control on the ANYmal quadruped, cutting per-step evaluation from 38 milliseconds to roughly 0.125 milliseconds using less than ten minutes of demonstration data. The discussion centers on why the method learns by minimizing the control Hamiltonian — the optimality condition MPC itself solves internally — rather than copying the expert's chosen actions, arguing this teaches the network the underlying reasoning rather than surface behavior. It contrasts this approach with classical Guided Policy Search, where the teacher adapts toward the student over training, versus MPC-Net's fixed, non-adaptive teacher that keeps solving the same optimal control problem regardless of the learner's progress. The hosts debate the tradeoffs of adaptive versus static teachers in imitation learning, weighing convergence speed against the validity and reusability of generated trajectories. Listeners interested in legged robotics, optimal control theory, or the mechanics of imitation learning will find a detailed technical walkthrough of how theory-grounded objectives can outperform standard behavioral cloning. Sources: 1. MPC-Net: A First Principles Guided Policy Search — Jan Carius, Farbod Farshidian, Marco Hutter, 2019 http://arxiv.org/abs/1909.05197 2. ALVINN: An Autonomous Land Vehicle in a Neural Network — Dean Pomerleau, 1989 https://scholar.google.com/scholar?q=ALVINN%3A+An+Autonomous+Land+Vehicle+in+a+Neural+Network 3. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29 4. Guided Policy Search — Sergey Levine, Vladlen Koltun, 2013 https://scholar.google.com/scholar?q=Guided+Policy+Search 5. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning 6. Learning Agile and Dynamic Motor Skills for Legged Robots — Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter, 2019 https://scholar.google.com/scholar?q=Learning+Agile+and+Dynamic+Motor+Skills+for+Legged+Robots 7. Sim-to-Real: Learning Agile Locomotion for Quadruped Robots — Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, Vincent Vanhoucke, 2018 https://scholar.google.com/scholar?q=Sim-to-Real%3A+Learning+Agile+Locomotion+for+Quadruped+Robots 8. Learning Quadrupedal Locomotion over Challenging Terrain — Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter, 2020 https://scholar.google.com/scholar?q=Learning+Quadrupedal+Locomotion+over+Challenging+Terrain 9. High-Slope Terrain Locomotion for Torque-Controlled Quadruped Robots (representative MPC/whole-body baseline) — Marco Hutter, Christian Gehring, and colleagues, ETH Zurich Robotic Systems Lab, 2016-2018 (various) https://scholar.google.com/scholar?q=High-Slope+Terrain+Locomotion+for+Torque-Controlled+Quadruped+Robots+%28representative+MPC%2Fwhole-body+baseline%29 10. Adaptive mixtures of local experts — R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, 1991 https://scholar.google.com/scholar?q=Adaptive+mixtures+of+local+experts 11. An efficient optimal planning and control framework for quadrupedal locomotion — F. Farshidian, M. Neunert, A. W. Winkler, G. Rey, J. Buchli, 2017 https://scholar.google.com/scholar?q=An+efficient+optimal+planning+and+control+framework+for+quadrupedal+locomotion Interactive Visualization: MPC-Net: Learning Optimal Control via the Hamiltonian

  8. 4d ago

    An Algorithmic Perspective on Imitation Learning

    This episode examines a 2018 survey, "An Algorithmic Perspective on Imitation Learning," which frames robot skill acquisition as an alternative to brittle manual programming or fragile reward engineering. It contrasts two core approaches: behavioral cloning, which treats the problem as supervised learning but suffers from compounding errors when the policy drifts into states the expert never demonstrated, and inverse reinforcement learning, which recovers the expert's underlying reward function before solving for a policy, trading computational cost for better generalization. Concrete examples like the ALVINN self-driving system, AlphaGo's use of expert-game pretraining, and Dynamic Movement Primitives illustrate how these ideas played out in practice, with DMPs offered as a hand-structured counterpoint to fully learned neural approaches. Listeners interested in the tradeoffs between hand-designed structure and end-to-end learning, or in how robotics tackled these problems just before deep learning reshaped the field, will find the historical framing useful for understanding today's imitation-learning methods. Sources: 1. An Algorithmic Perspective on Imitation Learning — Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, Jan Peters, 2018 http://arxiv.org/abs/1811.06711 2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29 3. End to End Learning for Self-Driving Cars — Mariusz Bojarski, et al. (NVIDIA), 2016 https://scholar.google.com/scholar?q=End+to+End+Learning+for+Self-Driving+Cars 4. Generative Adversarial Imitation Learning (GAIL) — Jonathan Ho, Stefano Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning+%28GAIL%29 5. ALVINN: An Autonomous Land Vehicle in a Neural Network — Dean Pomerleau, 1989 https://scholar.google.com/scholar?q=ALVINN%3A+An+Autonomous+Land+Vehicle+in+a+Neural+Network 6. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song, 2023 https://scholar.google.com/scholar?q=Diffusion+Policy%3A+Visuomotor+Policy+Learning+via+Action+Diffusion 7. Algorithms for Inverse Reinforcement Learning — Andrew Ng, Stuart Russell, 2000 https://scholar.google.com/scholar?q=Algorithms+for+Inverse+Reinforcement+Learning 8. Maximum Entropy Inverse Reinforcement Learning — Brian Ziebart, Andrew Maas, J. Andrew Bagnell, Anind Dey, 2008 https://scholar.google.com/scholar?q=Maximum+Entropy+Inverse+Reinforcement+Learning 9. Deep Reinforcement Learning from Human Preferences — Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, Dario Amodei, 2017 https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+from+Human+Preferences 10. Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors — Auke Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, Stefan Schaal, 2013 (Neural Computation; building on the authors' earlier 2002/2003 conference papers) https://scholar.google.com/scholar?q=Dynamical+Movement+Primitives%3A+Learning+Attractor+Models+for+Motor+Behaviors 11. Probabilistic Movement Primitives — Alexandros Paraschos, Christian Daniel, Jan Peters, Gerhard Neumann, 2013 https://scholar.google.com/scholar?q=Probabilistic+Movement+Primitives 12. Learning and Generalization of Motor Skills by Learning from Demonstration — Peter Pastor, Heiko Hoffmann, Tamim Asfour, Stefan Schaal, 2009 https://scholar.google.com/scholar?q=Learning+and+Generalization+of+Motor+Skills+by+Learning+from+Demonstration 13. Generative Adversarial Imitation Learning — J. Ho, S. Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning 14. Trust Region Policy Optimization — J. Schulman, S. Levine, P. Moritz, M. Jordan, P. Abbeel, 2015 https://scholar.google.com/scholar?q=Trust+Region+Policy+Optimization 15. Cooperative Inverse Reinforcement Learning — D. Hadfield-Menell, S. J. Russell, P. Abbeel, A. Dragan, 2016 https://scholar.google.com/scholar?q=Cooperative+Inverse+Reinforcement+Learning 16. Time-Contrastive Networks: Self-Supervised Learning from Video — P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, 2017 https://scholar.google.com/scholar?q=Time-Contrastive+Networks%3A+Self-Supervised+Learning+from+Video 17. Deep Q-learning from Demonstrations / Sun et al. on sample-complexity of imitation vs RL — W. Sun, A. Venkatraman, G. Gordon, B. Boots, J. A. Bagnell, 2017 https://scholar.google.com/scholar?q=Deep+Q-learning+from+Demonstrations+%2F+Sun+et+al.+on+sample-complexity+of+imitation+vs+RL Interactive Visualization: An Algorithmic Perspective on Imitation Learning

See All (886)

Ratings & Reviews

3.7
out of 5
3 Ratings

About

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

Information

You Might Also Like