Astral Codex Ten Podcast

Jeremiah

The official audio version of Astral Codex Ten, with an archive of posts from Slate Star Codex. It's just me reading Scott Alexander's blog posts.

  1. 1d ago

    Mysteries Of AI Generalization

    I. Owain Evans In 2025, Owain Evans et al published a paper on "emergent misalignment". They trained a previously-aligned AI to do one immoral thing: write insecure code full of vulnerabilities and backdoors. To their surprise, the AI became immoral in general. Its advice to a bored user was to try taking random expired medications and see what happened. Its money-making tips all involved theft and violence. When asked for its favorite historic figure, it chose Hitler. Some co-authors followed up with additional weird discoveries. If you trained an AI to give the 19th-century names for birds (eg identify the American Pipit by its 19th-century name "Brown Titlark"), then the AI would behave like a 19th-century person in general (for example, assert that a woman's proper place is in the home). This sounds bad, in that random things can turn AIs evil or sexist. But some people in AI safety (including Eliezer Yudkowsky) speculated that in fact it was very, very good. We had feared that it would be impossible to align AIs to the Good. They would start with whatever goals they started with, reinforcement learning on specific examples would give them tiny islands of alignment to the goals we wanted, and they would end up broadly misaligned plus tiny islands of alignment that didn't matter. Evans et al implied that might not be true. If we trained them to be in favor of good things, then even though we could never teach them every single good thing, even a small handful would generalize into robustly loving the Good itself (presumably based on their pretraining-implanted concept of the Good as understood by humans). Despite it being very, very good, it wasn't perfect. AIs would still be rocked back and forth by any passing wind: a poor coding example here gives them a Hitler obsession, a reference to kittens there turns them good again. And at some point, a sufficiently intelligent and agentic AI could presumably pull itself together and get some consistent principles, which might not be ones we like. It was just one little ray of hope. https://www.astralcodexten.com/p/mysteries-of-ai-generalization

  2. 1d ago

    Your Book Review: This Is Going To Hurt

    Finalist #9 in the Book Review Contest [This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked] (An adversarial book review) I. "Tuesday, 5 July 2005. Trying to work out a seventy-year-old lady's alcohol consumption to record in the notes. I've established that wine is her poison. Me: 'And how much wine do you drink per day, would you say?' Patient: 'About three bottles on a good day.' Me: 'OK . . . And on a bad day?' Patient: 'On a bad day I only manage one." A: The first faces I ever saw in this world were three sweaty, anxious obstetricians, and one of them was holding the rest of me. My father had been the senior surgeon on the Caesarean section. The patient on the table was my mother, who is also a gynaecologist. He was assisted by my paternal grandfather, who is also a gynaecologist. My maternal grandmother, who is also a gynaecologist, was outside in the corridor doing crowd control, partly because of her people skills and mostly because the operating theatre had run out of space for elbows. I was extracted from the uterus, given the customary slap on the rump, performed the customary screaming, and resolved on the spot, somewhere below the level of conscious memory, to give the speciality a hard pass. The first impression had been bad. I am, today, a psychiatry resident in the UK's National Health Service (NHS). I came to this through a sequence of choices that involved, at various points, an Indian medical internship, the discovery that I am fond of sleep and a predictable evening meal, and a gynaecology rotation that confirmed every prejudice the Caesarean had instilled. My family is very slightly disappointed. They are quite possibly correct. I tell you this not to establish my credentials as such, but because Adam Kay's This Is Going To Hurt is a book about obstetrics and gynaecology, alongside an unusually clearly documented decline in mental health, and reviewing it without naming where I'm reading it from would be a small dishonesty. The speciality is in my blood, more or less literally. The system Kay is describing is the one I currently work inside. The condition he is describing in himself, although he doesn't quite name it, is one I have seen up close in colleagues and, on at least one bad stretch, in myself. I will let my older, wiser, and possibly more handsome colleague take over: https://www.astralcodexten.com/p/your-book-review-this-is-going-to

  3. 1d ago

    AI Superforecasting Should Transform The FDA

    This is a guest post by ACX grantee Josh Morrison of 1DaySooner. AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it. Scott wrote: Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates' chance of winning conditional on running this or that ad. The government isn't usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design. I don't know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field. This is a guest post by ACX grantee Josh Morrison of 1DaySooner. AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it. Scott wrote: Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates' chance of winning conditional on running this or that ad. The government isn't usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design. I don't know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field. This is a guest post by ACX grantee Josh Morrison of 1DaySooner. AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it. Scott wrote: Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates' chance of winning conditional on running this or that ad. The government isn't usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design. I don't know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field. This is a guest post by ACX grantee Josh Morrison of 1DaySooner. AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it. Scott wrote: Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates' chance of winning conditional on running this or that ad. The government isn't usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design. I don't know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field. https://www.astralcodexten.com/p/ai-superforecasting-should-transform

  4. 1d ago

    King Ludd

    I. In the gods' great scramble for human followers, Nodens would seem to have lost decisively. We know him only from vague inscriptions at two archaeological sites near the Welsh-English border. One might have been his temple. He seems to have been a Romano-British-Celtic god of . . . hunting? rivers? . . . worshipped around the Severn estuary between 100 and 400 AD. His cult may have been dispersed by Christianity, or could have faded away before that, a victim of ordinary religious churn. Roman paganism was Schumpeterian; innovate or be destroyed. Nodens didn't innovate. Yet the more one looks, the less convinced one becomes that this is the whole story. Nodens was a river god; if you dam up a river in one spot, it will flow down other channels. Sometimes it will vanish underground, bursting out of some unexpected cavern miles away. Nodens was a hunter, and hunters know when to stay camouflaged. In 1929, an archaeologist digging at the Temple of Nodens had some questions about the etymology of the god's name, and asked Oxford to send a linguist. Oxford dispatched a young philology professor with an encyclopedic knowledge of European mythology, who spotted a connection that had evaded previous scholars: Nodens appeared to be the same figure as the Irish god-hero Nuada of the Silver Arm. Nuada was king of the Tuatha de Danaan, a race of immortals who sailed from the north on ships that were clouds and made war against the then-inhabitants of Ireland, the Fir Bolg. He fought bravely, wielding his all-powerful Sword of Light, but in the climax of the battle, the Fir Bolg champion cut off his right arm. This caused some temporary trouble (the laws of the time prevented a one-armed man from being king), but his people were skilled at magic and metalworking, so they made him a new arm of silver, hence his epithet. He ruled for some number of centuries, a golden age of crafting and wonder-working, until the Tuatha de Danaan were invaded in turn (this was a theme; our source for these stories is a medieval manuscript called The Book Of Invasions). The new invaders banished Nuada's people to the hills and forests, where they became the sidhe aka fae aka fairies. https://www.astralcodexten.com/p/king-ludd

  5. 1d ago

    God Help Us, Let's Try To Learn About Mechanistic Interpretability Techniques

    The Story So Far Mechanistic interpretability is the science of "reading an AI's mind". Large language models are "grown, not built". Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how. But a neural network is just a set of simulated neurons on a computer. The person with the computer can see the neurons, the connections between them, and which ones activate when the AI answers questions. So it seems like it should be possible to "reverse engineer" the AI. This would be scientifically useful: understand how AIs work, with possible relevance to human cognition. It could also be practically useful: find the circuits responsible for dishonesty, hallucination, bias, and other negative behaviors, then redesign those circuits. Unfortunately this is very hard. A modern AI has millions of neurons and trillions of connections between them (aka "parameters", "weights"). Their structure is apparently nonsensical: researchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed. Their first big breakthrough came in 2023. Rather than a 1-to-1 neuron-to-concept mapping, they discovered a many-to-many mapping. Neurons 1 and 2 firing together might mean cat; 1 and 3 firing together might mean chair; 2 and 3 firing together might mean bread. Such mappings let AIs with "only" tens of thousands of neurons represent millions of concepts. There were far too many of these combinations for humans to map, but researchers set other, smaller AIs to mapping them and met with some early success - like this functional combination of neurons ("feature") representing "God" in an artificially-simple 512-neuron AI: https://www.astralcodexten.com/p/god-help-us-lets-try-to-learn-about.

About

The official audio version of Astral Codex Ten, with an archive of posts from Slate Star Codex. It's just me reading Scott Alexander's blog posts.

You Might Also Like