Here are my pop-up Sustain What AI conversations with a top cybersecurity expert who says the Trump administration is poorly positioned to undertake the steps necessary to avoid superintelligence hacking us all, and a philosopher of technology who sees the path to safer artificial intelligence coming through dialogue with the technology more than direction. As President Trump on Sunday said the race with China requires a full-speed-ahead approach to this technology, the heads of all of the leading companies in the AI surge spoke about throttling things back. Here was The Wall Street Journal editorial board Sunday night: Anthropic CEO Dario Amodei issued a statement on Saturday calling for the slowdown amid the escalating progress of Anthropic’s Claude and other AI models. Sam Altman of OpenAI and ChatGPT quickly agreed, as did Elon Musk, who runs SpaceXAI, and Google DeepMind’s Demis Hassabis. The four compete in AI development, so their simultaneous warnings are striking. An early drill in 1999 As the last millennium drew toward a close, I diverted from my environment beat at The New York Times and asked to join our team reporting on the looming technological crisis called Y2k - short for the year 2000 computer bug. Somewhere in the vicinity of half a trillion dollars was being invested in sifting through myriad computerized systems to be sure they wouldn’t break down when ‘99 ticked over to ‘00. I spent New Year’s Eve at the, yes, Millennium Hotel down the block from The Times newsroom in Times Square. The lights stayed on. The world did not end. And doubters and believers remained locked in debates over whether this had been a panic or was a legitimate massive, and successful, preemptive surge. My first byline of the 2000s, with my friend and colleague Barnaby Feder was “Vast Effort to Fix Computers Not Over (And It’s Not Over).” Along with assessing claims and counterclaims, we looked at the general merits of the preventitve approach: Computer experts said the United States had benefited in several ways from its heavy spending, pegged at $100 billion by the government and several times that amount by some private analysts. One of the most important, some said, was a recognition by business and government leaders that computers and software must be more carefully managed as information technology becomes more pervasive. Mr. Rubin [Howard Rubin, then chairman of the department of computer science at Hunter College and an international expert on the year 2000 computer problem] called the rollover an “extraordinary success” after a belated start working on year 2000 flaws. Such slow reactions may not work in the future, he warned. “When computers are in your face -- when you have an Internet refrigerator -- you won’t be able to take that kind of chance,” he said. “Soon, Earth will have an Internet skin. This skin will be a critical thing. As everything becomes more responsive in real time, the technological risks get higher.” “Soon, Earth will have an Internet skin. This skin will be a critical thing. As everything becomes more responsive in real time, the technological risks get higher.” - Howard Rubin, January 1, 2000 Now, of course, Earth — and nearly all of human enterprise — has that digital skin, and were are in the midst of a fresh crescendo of warnings about the fast-approaching advent of unmanageable superintelligence. Sustain What is a reader-supported publication. To support my work, consider becoming a paying subscriber. I posted several times on this last week and I agree with those calling for a pause to assess what steps can mitigate worst-case outcomes — just as I did joined in calls for caution in 2023: I also just interviewed two very different experts: Bruce Schneier and J. Gray Cox to get their thoughts on how to survive superintelligence. Schneier, who calls himself a public-interest technologist, assesses and writes about security risks and opportunities at the intersection of technology and society. He gave a must-watch talk on hacking and AI at DEF CON 34, the latest in a long-running series of hacking conferences on computer security and its opposite: He describes his talk as a combination of the potentialities he raised in his 2022 book A Hacker’s Mind and the lessons we’re learning from current AI models engaging in hacking behavior. Here’s the YouTube summary: Humans are hacking AI systems. Humans are hacking with AI systems. But also, AIs are hacking human systems. They’re finding and exploiting vulnerabilities in computer code, and they’ll soon be doing the same with all sorts of other codes. For example: the tax code can be hacked. Vulnerabilities are called loopholes, exploits are called tax avoidance strategies, and black hats are called accountants. Similarly, financial markets can be hacked. So can any system of rules or laws, including democracy itself. AIs will hack these systems at our request, and they’ll also do this innately, organically – and possibly in ways we don’t immediately see. We need to consider a world where increasingly sophisticated hacks or our social, economic, and political systems are discovered computer speeds, and then exploited at computer scale and scope. Right now, our systems of patching these systems operate at a human pace, which won’t be good enough. In a section on how AI agents are the modern equivalent of genies, he notes how most genie stories don’t turn out well for those who do the wishing. If you have time, read the essay on genies and AI that he co-wrote with USC computer science professor Barath Raghavan for the security publication Lawfare. Here’s the Midas example: King Midas was granted his wish that everything he touches turns to gold. Then his bread turned to gold, and his wine, and his daughter. This is a story about greed, but it’s also a story about language. The gods did not cheat him; Midas got exactly what he asked for. He simply could not delineate, in advance, the full set of restrictions to his wish. Neither can anyone who gives tasks to an AI agent. Gray Cox is a wide-ranging professor of philosophy, peace studies, artificial intelligence and sustainable development at College of the Atlantic down the road from me in Bar Harbor, Maine. Key readings include: * This new paper in the journal Artificial Intelligence: From control to relationship: A peace studies approach to AI alignment, with evidence from multi-model dialogue. He told me how the standard approach to managing technology is an unworkable fit for AI agents. Negotiation methods used in pursuing peace are needed. And research on this needs to start now. * Cox’s book From control to relationship: A peace studies approach to AI alignment, with evidence from multi-model dialogue (Creative Commons download here) I asked Cox what the average person can do and he said Congress has a core role to play, so connect with your senators and representatives. Whether or not you agree with all of its agenda items (I don’t), PauseAI US is a decent portal for doing so. Schneier said it’s clear that the current administration is incapable of doing what’s needed on the international front so voting in November is a key step. In my conversation with Cox, I had to bring up the failed negotiation between the astronaut and Hal in Stanley Kubrick’s “2001: A Space Odyssey.” There is absolutely a lot of work to be done, starting yesterday. For further reading, don’t miss The New York Times op-ed “This is Really Bad” by Stephen Witt, the author of The Thinking Machine (on the rise of Nvidia). An excerpt: Researchers in Silicon Valley — and around the world — are beginning to recognize that A.I. may no longer be entirely within human control. Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue…. The more I learn about these rogue systems, the more I am reminded of the first two months of 2020, when a small group of epidemiologists was attempting to raise alarms about Covid. We have a short window, right now, to rein these systems in. This is A.I.’s warning shot, and we might not get another one. For sanity’s sake I also encourage you, as something of a counterpoint, to listen to Lex Fridman’s most recent conversation with David Heinemeier Hansson (Lex’s intro: “…also known as DHH, creator of Ruby on Rails, CTO of 37signals, creator of the new Omarchy Linux operating system, best-selling author, race car driver, and one of the most outspoken and prolific programmers in the world”). A key point from DHH is to embrace this astonishing moment in human and planetary history instead of recoiling: Maybe I wasn’t always like this, but stoic philosophy, amor fati, loving your fate, is an incredibly liberating way to live. And you could attack that and say, “Well, that’s a point of privilege.” True, but also I had the same directional point when I didn’t have all this, quote unquote, “privilege.” So I think there’s a way to live and interact with the world where you accept the things you can change and the things you cannot change and fall in love with all of it. Fall in love with the fact that the world today has jumped two decades forward into technology industry from where it was last year. Here’s a bit more from Fridman’s transcript (advice for programmers that I think applies to all of us): Lex Fridman(01:16:51) So certainly at the societal level, at the macroeconomic level, growth is exciting. But at the individual level, there’s going to be a lot of anxiety, potentially a lot of suffering. So by way of advice, if you’re like a young DHH or young developer programmer now, what would you, what would you advise they do? DHH(01:17:11) Don’t try to anticipate anything. You will literally go crazy. Because even the smartest bra
