This episode explores a 2023 paper on Deep Spiking Q-Networks, asking whether a directly trained spiking version of DQN can compete with earlier conversion-based spiking reinforcement learning methods on Atari while retaining the energy-efficiency promise of spiking neural networks. It explains the technical foundations behind spiking networks, including leaky integrate-and-fire neurons, surrogate-gradient training, and why SNNs remain difficult to train and awkward on conventional GPU hardware despite their appeal for neuromorphic chips like TrueNorth and Loihi. The discussion also situates the paper against the legacy of the original DeepMind DQN work, arguing that the paper’s title deliberately invites scrutiny over whether it truly matches the breadth and ambition of the classic Atari benchmark. Listeners would find it interesting for its clear framing of both the hype and the hard practical questions around neuromorphic AI: not just whether spiking RL works, but where, on what hardware, and under what conditions its efficiency claims actually matter. Sources: 1. Human-Level Control through Directly-Trained Deep Spiking Q-Networks — Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang, 2021 http://arxiv.org/abs/2201.07211 2. Spiking Neural Networks for Machine Learning: An Overview — Wolfgang Maass and others; overview literature includes major contributors such as Thomas Pfeil, Emre Neftci, and Surya Ganguli across the field, Recent overview genre, especially 2023 https://scholar.google.com/scholar?q=Spiking+Neural+Networks+for+Machine+Learning:+An+Overview 3. Training Spiking Neural Networks Using Lessons From Deep Learning — Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass, 2018 https://scholar.google.com/scholar?q=Training+Spiking+Neural+Networks+Using+Lessons+From+Deep+Learning 4. Spiking Neural Networks in the Fourth Generation of Artificial Intelligence — Zhaofei Yu, Hanle Zheng, Yujie Wu, and others, 2023 https://scholar.google.com/scholar?q=Spiking+Neural+Networks+in+the+Fourth+Generation+of+Artificial+Intelligence 5. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks — Friedemann Zenke, Tim Vogels, 2021 https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+Surrogate+Gradient+Learning+for+Instilling+Complex+Function+in+Spiking+Neural+Networks 6. Human-level control through deep reinforcement learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis, 2015 https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning 7. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillicrap, David Silver, Koray Kavukcuoglu, 2016 https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning 8. Deep Reinforcement Learning: An Overview — Yuxi Li, 2017 https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning:+An+Overview 9. Reinforcement Learning: An Introduction — Richard S. Sutton, Andrew G. Barto, 1998; 2nd edition 2018 https://scholar.google.com/scholar?q=Reinforcement+Learning:+An+Introduction 10. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks — Emre O. Neftci, Hesham Mostafa, Friedemann Zenke, 2019 https://scholar.google.com/scholar?q=Surrogate+Gradient+Learning+in+Spiking+Neural+Networks:+Bringing+the+Power+of+Gradient-Based+Optimization+to+Spiking+Neural+Networks 11. Direct Training for Spiking Neural Networks: Faster, Larger, Better — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2019 https://scholar.google.com/scholar?q=Direct+Training+for+Spiking+Neural+Networks:+Faster,+Larger,+Better 12. Going Deeper With Directly-Trained Larger Spiking Neural Networks — Chaoteng Duan, Shikuang Deng, Xingting Wang, Meng Zhang, and others, 2022 https://scholar.google.com/scholar?q=Going+Deeper+With+Directly-Trained+Larger+Spiking+Neural+Networks 13. Threshold-Dependent Batch Normalization for Training Deep Spiking Neural Networks — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2021 https://scholar.google.com/scholar?q=Threshold-Dependent+Batch+Normalization+for+Training+Deep+Spiking+Neural+Networks 14. A million spiking-neuron integrated circuit with a scalable communication network and interface — Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron Flickner, William Risk, Rajit Manohar, Dharmendra Modha, 2014 https://scholar.google.com/scholar?q=A+million+spiking-neuron+integrated+circuit+with+a+scalable+communication+network+and+interface 15. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning — Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al., 2018 https://scholar.google.com/scholar?q=Loihi:+A+Neuromorphic+Manycore+Processor+with+On-Chip+Learning 16. SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation — Steve B. Furber, Francesco Galluppi, Steve Temple, Luis A. Plana, 2014 https://scholar.google.com/scholar?q=SpiNNaker:+A+1-W+18-Core+System-on-Chip+for+Massively-Parallel+Neural+Network+Simulation 17. Benchmarking Neuromorphic Systems with Nengo — Terry C. Stewart, Dan Rasmussen, Xuan Choo, Aaron Voelker, and others, 2015-2017 era benchmarking work https://scholar.google.com/scholar?q=Benchmarking+Neuromorphic+Systems+with+Nengo 18. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013 https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning 19. Deep Reinforcement Learning with Double Q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2016 https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+with+Double+Q-learning 20. Rainbow: Combining Improvements in Deep Reinforcement Learning — Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, 2018 https://scholar.google.com/scholar?q=Rainbow:+Combining+Improvements+in+Deep+Reinforcement+Learning 21. Enabling Deep Spiking Neural Networks for Reinforcement Learning — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2020 https://scholar.google.com/scholar?q=Enabling+Deep+Spiking+Neural+Networks+for+Reinforcement+Learning 22. Going Deeper in Spiking Neural Networks: VGG and Residual Architectures — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2021 https://scholar.google.com/scholar?q=Going+Deeper+in+Spiking+Neural+Networks:+VGG+and+Residual+Architectures 23. Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks — Yuhang Fang, Zhaofei Yu, Tielin Zhang, et al., 2021 https://scholar.google.com/scholar?q=Incorporating+Learnable+Membrane+Time+Constant+to+Enhance+Learning+of+Spiking+Neural+Networks 24. Deep Residual Learning in Spiking Neural Networks — Yujie Wu, Yuhang Zhao, et al., 2021 https://scholar.google.com/scholar?q=Deep+Residual+Learning+in+Spiking+Neural+Networks 25. A Unified Optimization Framework of ANN-SNN Conversion: Towards Optimal Mapping from Activation Values to Firing Rates — approx. recent ANN-to-SNN conversion literature, 2023-2024 https://scholar.google.com/scholar?q=A+Unified+Optimization+Framework+of+ANN-SNN+Conversion:+Towards+Optimal+Mapping+from+Activation+Values+to+Firing+Rates 26. Towards High-Performance Spiking Transformers from ANN to SNN Conversion — approx. recent conversion/transformer authors, 2024 https://scholar.google.com/scholar?q=Towards+High-Performance+Spiking+Transformers+from+ANN+to+SNN+Conversion 27. Towards Training-Free and Accurate ANN-to-SNN Conversion via Activation-Aware Redistribution — approx. recent ANN-to-SNN conversion authors, 2024 https://scholar.google.com/scholar?q=Towards+Training-Free+and+Accurate+ANN-to-SNN+Conversion+via+Activation-Aware+Redistribution 28. Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks — approx. recent SNN RL authors, 2024-2025 https://scholar.google.com/scholar?q=Adaptive+Surrogate+Gradients+for+Sequential+Reinforcement+Learning+in+Spiking+Neural+Networks 29. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks — approx. recent theoretical SNN authors, 2023-2024 https://scholar.google.com/scholar?q=Elucidating+the+Theoretical+Underpinnings+of+Surrogate+Gradient+Learning+in+Spiking+Neural+Networks 30. Spiking Reinforcement Learning Enhanced by Bioinspired Event Source of Multi-Dendrite Spiking Neuron and Dynamic Thresholds — approx. recent spiking RL authors, 2024-2025 https://scholar.google.com/scholar?q=Spiking+Reinforcement+Learning+Enhanced+by+Bioinspired+Event+Source+of+Multi-Dendrite+Spiking+Neuron+and+Dynamic+Thresholds 31. S2Act: Simple Spiking Actor — approx. recent spiking actor-critic authors, 2024-2025 https://scholar.google.com/scholar?q=S2Act:+Simple+Spiking+Actor 32. AI Post Transformers: Zero-Shot Context Gen
