A response from the Derby Mill team of AI experts to Anthropic co-founder Jack Clark’s recent Substack essay, “AI systems are about to start building themselves.” In the essay, Clark predicts that “no-human-involved AI R&D - an AI system powerful enough that it could plausibly autonomously build its own successor - happens by the end of 2028.” Clark notes: “If that happens, we will cross a Rubicon into a nearly-impossible-to-forecast future." With that in mind, Ajay tees up a discussion among Rich, Niamh, Sendhil and Suzanne. Is Jack Clark correct that we’re heading toward a future of self-replicating, self-optimizing AI? What are the implications? Does that actually lead to the singularity? And should we, as Rich Sutton says, treat AI the way a parent treats a child, recognizing that mistakes are inevitable? “There’s some spookiness being implied here,” Mullainathan says, “about it coming alive and taking over a whole set of decisions we didn’t intend to cede it… and if that’s the case, that freaks me out, too.” “The thing that they’re scared about…” says Niamh Gavin. “It’s the fact that you have, in essence, an irreversible positive feedback loop of successive self-improvement cycles, that accelerate it toward what they call a singularity, whereby artificial intelligence exceeds human intelligence and control.” Are the concerns outlined by Jack Clark warranted? Or are the fears about a harmful singularity overblown? The Derby Mill experts provide their views in our latest episode. Finally, another fascinating thread in the episode comes from Suzanne Gildert. “It all comes down to the reward function,” says Gildert. “So there’s this thing that [AI is] still limited by us telling them what to do, because we’re the ones who want something… But that’s eventually going to break because the way the whole reward function in ML works at the moment — it’s all based on our economy… It’s based on people exchanging money for goods and services. All that’s going to break if people can’t work anymore... So that’s why I think we really have to understand what the reward function should be, because the way we’re implementing it now is not going to work beyond the point where [AI] can do all human labour for us.” HOSTS AND PANELLISTS Ajay Agrawal, co-founder and partner, Intrepid Growth Partners Richard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta Sendhil Mullainathan, senior advisor, Intrepid Growth Partners, MacArthur Genius grant recipient and professor, MIT Niamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist and CEO, Emergent Platforms Suzanne Gildert, CEO, Nirvanic Consciousness Technologies, quantum physicist, co-founder of Sanctuary AI and Kindred LINKS The Jack Clark Substack post that triggered the discussion. Jack Clark is on X @jackclarkSF Subscribe to The Derby Mill Series at our Substack (main site) or on YouTube, Spotify or Apple Podcasts. We post highlights from the show on YouTube. Derby Mill is created by the team at Intrepid Growth Partners and produced by Ghost Bureau. DISCUSSION POINTS 00:00 Cold open 01:06 Jack Clark post 02:56 Rich Sutton reaction 08:37 Sendhil’s reaction 16:22 Suzanne’s reaction 18:14 Niamh’s reaction 23:48 Wrap up DISCLAIMER The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insights.intrepidgp.com
