Programmatically Interpretable Reinforcement Learning: Readable Policies

This episode examines Programmatically Interpretable Reinforcement Learning (PIRL), a 2018 framework from Rice University, Google Brain, and DeepMind researchers that forces RL policies to be expressed as short, human-readable programs rather than opaque neural network weights. The discussion centers on why formal verification—proving properties like bounded steering output in a self-driving car—is tractable for small domain-specific programs but essentially impossible for networks with millions of parameters. Using the paper's driving example, the hosts unpack "policy sketches" (a switch statement branching on track position, with PID controllers filling each branch) and Neurally Directed Program Search (NDPS), which trains a conventional deep RL policy as an oracle and then searches program space to imitate its outputs via smooth regression rather than fighting a jagged, non-differentiable reward landscape. They draw out the connection to DAgger's iterative imitation-learning approach from Ross, Gordon, and Bagnell, while flagging a subtle mismatch between matching an expert's actions and matching reward through an imitation proxy. Listeners interested in AI safety, control theory, or the tension between interpretability and performance will find the concrete TORCS driving case a clear entry point into verifiable reinforcement learning. Sources: 1. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018 http://arxiv.org/abs/1804.02477 2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning — Stéphane Ross, Geoffrey Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning 3. Is Imitation Learning the Route to Humanoid Robots? — Stefan Schaal, 1999 https://scholar.google.com/scholar?q=Is+Imitation+Learning+the+Route+to+Humanoid+Robots%3F 4. ALVINN: An Autonomous Land Vehicle in a Neural Network — Dean Pomerleau, 1989 https://scholar.google.com/scholar?q=ALVINN%3A+An+Autonomous+Land+Vehicle+in+a+Neural+Network 5. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016 https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning 6. Verifiable Reinforcement Learning via Policy Extraction — Osbert Bastani, Yewen Pu, Armando Solar-Lezama, 2018 https://scholar.google.com/scholar?q=Verifiable+Reinforcement+Learning+via+Policy+Extraction 7. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018 https://scholar.google.com/scholar?q=Programmatically+Interpretable+Reinforcement+Learning 8. Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning — Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, Sung-Hyun Son, 2020 https://scholar.google.com/scholar?q=Optimization+Methods+for+Interpretable+Differentiable+Decision+Trees+Applied+to+Reinforcement+Learning 9. Distilling a Neural Network Into a Soft Decision Tree — Nicholas Frosst, Geoffrey Hinton, 2017 https://scholar.google.com/scholar?q=Distilling+a+Neural+Network+Into+a+Soft+Decision+Tree 10. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, 2011 https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29 11. Continuous Control with Deep Reinforcement Learning (DDPG) — Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2015 https://scholar.google.com/scholar?q=Continuous+Control+with+Deep+Reinforcement+Learning+%28DDPG%29 12. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks — Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, Mykel J. Kochenderfer, 2017 https://scholar.google.com/scholar?q=Reluplex%3A+An+Efficient+SMT+Solver+for+Verifying+Deep+Neural+Networks 13. The Sketching Approach to Program Synthesis — Armando Solar-Lezama, 2009 https://scholar.google.com/scholar?q=The+Sketching+Approach+to+Program+Synthesis 14. Syntax-Guided Synthesis (SyGuS) — Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Ghila Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, Abhishek Udupa, 2015 https://scholar.google.com/scholar?q=Syntax-Guided+Synthesis+%28SyGuS%29 Interactive Visualization: Programmatically Interpretable Reinforcement Learning: Readable Policies