Backpropagation Through Time Explained

This episode explores Paul Werbos’s 1990 paper on Backpropagation Through Time and explains how ordinary backpropagation extends to systems whose state evolves over time. It walks through the core idea of unrolling a recurrent or dynamic system into a time-indexed computation graph, then applying reverse-mode differentiation to compute exact gradients across both layers and time steps. The discussion also places BPTT in historical context, connecting it to earlier work on backpropagation, automatic differentiation, and alternative recurrent learning methods like real-time recurrent learning. Listeners would find it interesting because it shows how a foundational training method for sequence models, control systems, and differentiable simulations emerged from a simple but powerful reframing of memory and time in neural computation. Sources: 1. Backpropagation Through Time Explained https://podcast.do-not-panic.com/uploaded-pdfs/2026-05-08T22-17-15-230Z-Backpropagation-through-time-what-it-does-and-how-to-do-it.pdf 2. Learning representations by back-propagating errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986 https://scholar.google.com/scholar?q=Learning+representations+by+back-propagating+errors 3. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks — Ronald J. Williams, David Zipser, 1989 https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks 4. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990 https://scholar.google.com/scholar?q=Backpropagation+Through+Time:+What+It+Does+and+How+to+Do+It 5. Learning long-term dependencies with gradient descent is difficult — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994 https://scholar.google.com/scholar?q=Learning+long-term+dependencies+with+gradient+descent+is+difficult 6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997 https://scholar.google.com/scholar?q=Long+Short-Term+Memory 7. Taylor expansion of the accumulated rounding error — Seppo Linnainmaa, 1976 https://scholar.google.com/scholar?q=Taylor+expansion+of+the+accumulated+rounding+error 8. Fast Exact Multiplication by the Hessian — Barak A. Pearlmutter, 1994 https://scholar.google.com/scholar?q=Fast+Exact+Multiplication+by+the+Hessian 9. Automatic Differentiation in Machine Learning: a Survey — Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind, 2018 https://scholar.google.com/scholar?q=Automatic+Differentiation+in+Machine+Learning:+a+Survey 10. A review of automatic differentiation and its efficient implementation — Charles C. Margossian, 2019 https://scholar.google.com/scholar?q=A+review+of+automatic+differentiation+and+its+efficient+implementation 11. Generalization of Back-Propagation to Recurrent Neural Networks — Fernando J. Pineda, 1987 https://scholar.google.com/scholar?q=Generalization+of+Back-Propagation+to+Recurrent+Neural+Networks 12. Finding Structure in Time — Jeffrey L. Elman, 1990 https://scholar.google.com/scholar?q=Finding+Structure+in+Time 13. BP(lambda): Online Learning via Synthetic Gradients — approx. anonymous from snippet / modern deep learning authors, recent https://scholar.google.com/scholar?q=BP(lambda):+Online+Learning+via+Synthetic+Gradients 14. Streaming Propagation Through Time: A New Computational Paradigm for Recurrent Neural Networks — approx. modern recurrent-learning authors, recent https://scholar.google.com/scholar?q=Streaming+Propagation+Through+Time:+A+New+Computational+Paradigm+for+Recurrent+Neural+Networks 15. Combining Truncated BPTT and Truncated RTRL for LSTM Training — Jakob Stefan Weber, recent https://scholar.google.com/scholar?q=Combining+Truncated+BPTT+and+Truncated+RTRL+for+LSTM+Training 16. Second-order forward-mode optimization of recurrent neural networks for neuroscience — approx. modern neuroscience/optimization authors, recent https://scholar.google.com/scholar?q=Second-order+forward-mode+optimization+of+recurrent+neural+networks+for+neuroscience 17. Sample-Based Hybrid Mode Control: Asymptotically Optimal Switching of Algorithmic and Non-Differentiable Control Modes — approx. modern control authors, recent https://scholar.google.com/scholar?q=Sample-Based+Hybrid+Mode+Control:+Asymptotically+Optimal+Switching+of+Algorithmic+and+Non-Differentiable+Control+Modes 18. On the differentiability of the value function of switched linear systems under arbitrary and controlled switching — approx. control theory authors, recent https://scholar.google.com/scholar?q=On+the+differentiability+of+the+value+function+of+switched+linear+systems+under+arbitrary+and+controlled+switching 19. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp3 20. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3 Interactive Visualization: Backpropagation Through Time Explained