This episode explores the Relational Graph Transformer paper and asks whether transformer-based models can outperform standard graph neural networks for prediction tasks over real multi-table databases such as customers, orders, products, claims, and shipments. It explains how the method turns a relational warehouse into a heterogeneous temporal graph, then builds five-part tokens for sampled neighbors that encode row features, table type, hop distance, relative time, and a learned local-structure signal. The discussion focuses on the model’s local-global attention design, where dense attention over timestamp-safe two-hop neighborhoods is paired with learned global centroids to capture broader database patterns without full all-pairs cost. It is especially interesting because it frames both the promise and the friction of relational deep learning: strong motivation to beat hand-engineered SQL features and message-passing bottlenecks, but real skepticism about whether such graph-heavy systems are practical enough for ordinary industrial stacks. Sources: 1. Relational Graph Transformer — Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, Jure Leskovec, 2025 http://arxiv.org/abs/2505.10960 2. Heterogeneous Graph Transformer — Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun, 2020 https://scholar.google.com/scholar?q=Heterogeneous+Graph+Transformer 3. Temporal Graph Networks for Deep Learning on Dynamic Graphs — Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein, 2020 https://scholar.google.com/scholar?q=Temporal+Graph+Networks+for+Deep+Learning+on+Dynamic+Graphs 4. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, Jure Leskovec, 2023 https://scholar.google.com/scholar?q=Relational+Deep+Learning:+Graph+Representation+Learning+on+Relational+Databases 5. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Matthias Fey, Jure Leskovec, et al., 2024 https://scholar.google.com/scholar?q=RelBench:+A+Benchmark+for+Deep+Learning+on+Relational+Databases 6. Graph-Bert: Only Attention is Needed for Learning Graph Representations — Jiawei Zhang, Haopeng Zhang, Congying Xia, Li Sun, 2020 https://scholar.google.com/scholar?q=Graph-Bert:+Only+Attention+is+Needed+for+Learning+Graph+Representations 7. Do Transformers Really Perform Bad for Graph Representation? — Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu, 2021 https://scholar.google.com/scholar?q=Do+Transformers+Really+Perform+Bad+for+Graph+Representation? 8. Pure Transformers are Powerful Graph Learners — Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong, 2022 https://scholar.google.com/scholar?q=Pure+Transformers+are+Powerful+Graph+Learners 9. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs — Jinsong Chen, Kaiyuan Gao, Gaichao Li, Kun He, 2023 https://scholar.google.com/scholar?q=NAGphormer:+A+Tokenized+Graph+Transformer+for+Node+Classification+in+Large+Graphs 10. Representing Long-Range Context for Graph Neural Networks with Global Attention — Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, Ion Stoica, 2021 https://scholar.google.com/scholar?q=Representing+Long-Range+Context+for+Graph+Neural+Networks+with+Global+Attention 11. Recipe for a General, Powerful, Scalable Graph Transformer — Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini, 2022 https://scholar.google.com/scholar?q=Recipe+for+a+General,+Powerful,+Scalable+Graph+Transformer 12. Exphormer: Sparse Transformers for Graphs — Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, Ali Kemal Sinop, 2023 https://scholar.google.com/scholar?q=Exphormer:+Sparse+Transformers+for+Graphs 13. Centroid Transformers: Learning to Abstract with Attention — Lemeng Wu, Xingchao Liu, Qiang Liu, 2021 https://scholar.google.com/scholar?q=Centroid+Transformers:+Learning+to+Abstract+with+Attention 14. Learning Efficient Positional Encodings with Graph Neural Networks — Charilaos I. Kanatsoulis et al., 2025 https://arxiv.org/abs/2502.01122 15. ContextGNN: Beyond Two-Tower Recommendation Systems — Yiwen Yuan et al., 2024 https://arxiv.org/abs/2411.19513 16. RelGNN: Composite Message Passing for Relational Deep Learning — Tianlang Chen, Charilaos Kanatsoulis, Jure Leskovec, 2025 https://arxiv.org/abs/2502.06784 17. Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs — Jeongwhan Choi et al., 2025 https://arxiv.org/abs/2511.13010 18. Beyond Message Passing: Neural Graph Pattern Machine — Zehong Wang et al., 2025 https://arxiv.org/abs/2501.18739 19. Transformers Meet Relational Databases — Jakub Peleska and Gustav Sir, 2024 https://arxiv.org/abs/2412.05218 20. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan et al., 2025 https://arxiv.org/abs/2510.06377 21. Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification — Zijie Zhou et al., 2024 https://arxiv.org/abs/2412.15302 22. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3
