Into AI Safety

Jacob Haimes

0.0（0）
テクノロジー

The Into AI Safety podcast aims to make it easier for everyone, regardless of background, to get meaningfully involved with the conversations surrounding the rules and regulations which should govern the research, development, deployment, and use of the technologies encompassed by the term "artificial intelligence" or "AI" For better formatted show notes, additional resources, and more, go to https://kairos.fm/intoaisafety/

7月9日

Pretraining Safety w/ Ethan Roland

What if the safest AI models weren't built by adding guardrails after training, but by shaping what gets learned in the first place? Ethan Roland, senior alignment researcher at AE Studio and first author on an ICML 2026 spotlight paper, joins Jacob to talk about gradient routing, a technique that routes dangerous capabilities into isolated parts of a model's architecture where they can be locked or removed entirely. They get into the absorption effect, KYC-style access control frameworks, and what it would actually take for frontier labs to adopt this kind of work before it's needed rather than after. Chapters (00:00) - Introduction (06:39) - Inside AE Studio (15:26) - China & the Alignment vs. Controllability Framing (18:23) - Data Filtering & Gradient Routing (Aside) (30:39) - Mixture of Experts Explained (Aside) (36:25) - Why Pre-Training Interventions Are Rare (42:43) - Ethan's Theory of Change (56:17) - Access Control Governance and KYC (Aside) (01:04:47) - The Researcher's Role in Policy Advocacy (01:11:38) - Speed Round (01:27:55) - Outro LinksBelow are the most important links for this episode. For more, visit the episode page on Kairos.fm.Ethan's websiteThe paper landing pagePreprint - Gradient Routing: Masking Gradients to Localize Computation in Neural NetworksICLR paper and webpage - Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMsWikipedia article - CBRN defenseNTI tutorials on bioweapons and nuclear testing
6月1日

Reclaiming UBI in the AI Age w/ Joe Williams

Today's episode does double duty as an interview and an announcement. Joe Williams, host of the new Kairos.fm show "Beyond the Paycheck: Reclaiming the Case for UBI in the Age of AI," joins Jacob to talk about his background as a freelance translator and how AI quietly dismantled his livelihood in 2025. From there the conversation expands into whether this moment is really different from past waves of automation, who exactly makes up the tech billionaire class, and why you should probably raise an eyebrow when someone like Elon Musk says he supports UBI. Go check out Reclaiming UBI! Chapters (00:00) - Intro (05:01) - Joe's Background (19:41) - What is UBI & Why Does It Matter Now? (27:25) - ASIDE: The Corporation Who Cried " My Technology Will End Work" (32:25) - Wealth Concetration & the Digital Economy (39:06) - The AI Gentry (48:13) - ASIDE: Sam Altman + OpenResearch + Worldcoin (53:38) - Powerful UBI Supporters Should Raise Red Flags (01:02:56) - How to Read AI Gentry Discourse Critically (01:10:00) - Who Is "Reclaiming UBI" For? (01:18:17) - Outro Critical LinksBelow are the most important links for this episode. For more, visit the episode page on Kairos.fm.NEW KAIROS.FM PODCAST - Reclaiming UBI: Work and Values in the Age of AIJosh Steimle blog post - Will AI Destroy Jobs? 500 Years of Predictions Say NoSmithsonian Magazine article - What the Luddites Really Fought AgainstITIF report - Oops: The Predicted 47 Percent of Job Loss From AI Didn't HappenFrontiers in Artificial Intelligence journal article - AI, Universal Basic Income, and Power: Symbolic Violence in the Tech Elite's NarrativeTime article - What to Know About Worldcoin and the Controversy Around ItWikipedia page - World (blockchain)
5月12日

Building Asymmetric Defense w/ Zainab Majid

Zainab Majid, co-founder of Asymmetric Security, joins Jacob for a conversation on the intersection between AI Safety and cybersecurity, as well as the future of digital forensics. Drawing from years of incident response work, she explains how cyber attacks actually unfold, why AI is changing both offense and defense, and how her team is building AI-native tools to investigate breaches faster and more effectively. Other topics explored in this episode include trust in the AI/cybersecurity industries, the realities behind cybersecurity hype, and the challenge of keeping humans meaningfully involved as these systems become more capable. Zainab also gives practical, actionable advice on how you can protect yourself. If you're interested in over 30 minutes of additional content, head on over to the Kairos.fm Patreon where you can become a subscriber for just $2 per month, which helps make this whole podcasting thing a bit more sustainable. Chapters (00:00) - Intro (04:34) - Zainab's Background (08:49) - Jacob & Zainab's History (16:03) - Founding Asymmetric Security (24:49) - How to Know Who You Can Trust (36:31) - The Threats Asymmetric Is Built to Fight (01:05:54) - What's Asymmetric Tackling Next? (01:15:33) - Glasswing, Dual Use, and Power Concentration (01:24:21) - The Relationship Between AI Safety & Cybersecurity (01:37:01) - Outro Critical LinksBelow are the most important links for this episode. For more, visit the episode page on Kairos.fm.Asymmetric Security websiteEvalEval @ NeurIPS workshop paper - Rethinking CyberSecEvalMeta AI report - Purple Llama CyberSecEvalAnthropic press release - Project GlasswingSchneier on Security blogpost - What Anthropic’s Mythos Means for the Future of Cybersecurity
4月6日

Drawing Red Lines w/ Su Cizem

Technology has been moving faster than policy for some time now, and the advent of AI isn't changing that, so what can we do to maintain safety despite uncertainty? Su Cizem has spent the last few years trying to answer that question. As an analyst at the Future Society, she works on global AI governance, specifically on building international consensus around AI red lines: the thresholds we collectively agree must never be crossed. In this conversation, Su walks through her path from philosophy to policy, the evolution of the global AI safety summit series, why voluntary commitments from AI labs aren't enough, and what it would actually take to make international cooperation on AI safety real. Chapters (00:00) - Introduction (03:23) - From Philosophy to Policy (22:25) - What AI Governance Actually Means (26:49) - The Summit Series (43:01) - Drawing The Red Lines (01:10:51) - Can These Companies Govern Themselves? (01:24:01) - Breaking Into The Field (01:27:51) - Closing Thoughts & Outro Critical LinksBelow are the most important links for this episode. For more, visit the episode page on Kairos.fm.Su's LinkedInGlobal Call for AI Red LinesThe Futures Society report - “Facing the Stakes of AI Together”: 2025 Athens Roundtable ReportPolitico article - How the global effort to keep AI safe went off the railsTechPolicy.Press article - A Timeline of the Anthropic-Pentagon DisputeThe Guardian article - AI got the blame for the Iran school bombing. The truth is far more worryingGoogle and OpenAI Employee open letter - We Will Not Be DividedThe Register article - Altman said no to military AI abuses – then signed Pentagon deal anywaySaferAI report - Evaluating AI Providers’ Frontier AI Safety Frameworks
3月10日

Thinking Through "Digital Minds" w/ Jacy Reese-Anthis

Jacy Reese-Anthis, founder of Sentience Institute and researcher at Stanford, began his journey working for animal welfare, but is now finishing up his PhD with research in many different AI subfields at the intersection of neuroscience, philosophy, social science, and machine learning. While this may seem like an odd jump at first, Jacy shares how his work has all been centered around the idea of moral circle expansion. In this episode, we dig into what sentience actually means (or at least how we can begin to think about it), why anthropomorphization is more complicated than it sounds, and how language models may be able to be leveraged as an effective tool for social science research. Jacy also shares his median AGI estimate somewhere in there, so stay tuned if you want to catch it. As part of my effort to make this whole podcasting thing more sustainable, I have created a Kairos.fm Patreon which includes an extended version of this episode. Supporting gets you access to these extended cuts, as well as other perks in development. Chapters (00:00) - Introduction (05:41) - From Animal Welfare to Digital Minds (09:00) - Founding Sentience Institute (22:00) - Defining Sentience (27:13) - The Anthropomorphization Problem (47:51) - Why "Digital Minds" (Not "Artificial Intelligence") (51:05) - LLMs as Social Science Tools (01:07:03) - Jacy’s AGI Timeline & The Singularity (01:09:23) - Final Thoughts & Outro Critical LinksBelow are the most important links for this episode. For more, visit the episode page on Kairos.fm.Jacy's websiteWikipedia article - Jacy Reese AnthisSentience Institute websiteCHI paper - Digital Companionship: Overlapping Uses of AI Companions and AI AssistantsICML paper - LLM Social Simulations Are a Promising Research MethodACL paper - The Impossibility of Fair LLMsWikipedia article - ELIZA effectThe Atlantic article - How a Google Employee Fell for the Eliza Effect
2月2日

Scaling AI Safety Through Mentorship w/ Dr. Ryan Kidd

What does it actually take to build a successful AI safety organization? I'm joined by Dr. Ryan Kidd, who has co-led MATS from a small pilot program to one of the field's premier talent pipelines. In this episode, he reveals the low-hanging fruit in AI safety field-building that most people are missing: the amplifier archetype. I pushed Ryan on some hard questions, from balancing funder priorities and research independence, to building a robust selection process for both mentors and participants. Whether you're considering a career pivot into AI safety or already working in the field, this conversation offers practical advice on how to actually make an impact. Chapters (00:00) - - Intro (08:16) - - Building MATS Post-FTX & Summer of Love (13:09) - - Balancing Funder Priorities and Research Independence (19:44) - - The MATS Selection Process (33:15) - - Talent Archetypes in AI Safety (50:22) - - Comparative Advantage and Career Capital in AI Safety (01:04:35) - - Building the AI Safety Ecosystem (01:15:28) - - What Makes a Great AI Safety Amplifier (01:21:44) - - Lightning Round Questions (01:30:30) - - Final Thoughts & Outro LinksMATSRyan's Writing LessWrong post - Talent needs of technical AI safety teamsLessWrong post - AI safety undervalues foundersLessWrong comment - Comment permalink with 2025 MATS program detailsLessWrong post - Talk: AI Safety Fieldbuilding at MATSLessWrong post - MATS Mentor SelectionLessWrong post - Why I funded PIBBSSEA Forum post - How MATS addresses mass movement building concernsFTX Funding of AI Safety LessWrong blogpost - An Overview of the AI Safety Funding SituationFortune article - Why Sam Bankman-Fried’s FTX debacle is roiling A.I. researchNY Times article - FTX probes $6.5M in payments to AI safety group amid clawback crusadeCointelegraph article - FTX probes $6.5M in payments to AI safety group amid clawback crusadeFTX Future Fund article - Future Fund June 2022 Update (archive)Tracxn page - Anthropic Funding and InvestorsTraining & Support Programs Catalyze ImpactSeldon LabSPARBlueDot ImpactYCombinatorPivotalAthenaAstra FellowshipHorizon FellowshipBASE FellowshipLASR LabsEntrepeneur FirstFunding Organizations Coefficient Giving (previously Open Philanthropy)LTFFLongview PhilanthropyRenaissance PhilanthropyCoworking Spaces LISAMoxLighthavenFAR LabsConstellationColliderNET OfficeBAISHResearch Organizations & Startups Atla AIApollo ResearchTimaeusRAND CASTCHAIOther Sources AXRP website - The AI X-risk Research PodcastLessWrong blogpost - Shard Theory: An Overview
2025/12/29

Sobering Up on AI Progress w/ Dr. Sean McGregor

Sean McGregor and I discuss about why evaluating AI systems has become so difficult; we cover everything from the breakdown of benchmarking, how incentives shape safety work, and what approaches like BenchRisk (his recent paper at NeurIPS) and AI auditing aim to fix as systems move into the real world. We also talk about his history and journey in AI safety, including his PhD on ML for public policy, how he started the AI Incident Database, and what he's working on now: AVERI, a non-profit for frontier model auditing. Chapters (00:00) - Intro (02:36) - What's broken about benchmarking (03:41) - Sean’s wild PhD (14:28) - The phantom internship (19:25) - Sean's journey (22:25) - Market-vs-regulatory modes and AIID (32:13) - Drunk on AI progress (38:34) - BenchRisk (43:20) - Moral hazards and Master Hand (50:34) - Liability, Section 230, and open source (59:20) - AVERI (01:11:30) - Closing thoughts & outro LinksSean McGregor's websiteAVERI websiteBenchRisk BenchRisk websiteNeurIPS paper - Risk Management for Mitigating Benchmark Failure Modes: BenchRiskNeurIPS paper - AI and the Everything in the Whole Wide World BenchmarkAIID AI Incident Database websiteIAAI paper - Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident DatabasePreprint - Lessons for Editors of AI Incidents from the AI Incident DatabaseAIAAIC website (another incident tracker)Hot AI Summer CACM article - A Few Useful Things to Know About Machine LearningCACM article - How the AI Boom Went BustUndergraduate Thesis - Analyzing the Prospect of an Approaching AI WinterTech Genies article - AI History: The First Summer and Winter of AICACM article - There Was No ‘First AI Winter’Measuring Generalization Neural Computation article - The Lack of A Priori Distinctions Between Learning AlgorithmsICLR paper - Understanding deep learning requires rethinking generalizationICML paper - Model-agnostic Measure of Generalization DifficultyRadiology Artificial Intelligence article - Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological PitfallsPreprint - Quantifying Generalization Complexity for Large Language ModelsInsurers Exclude AI Financial Times article - Insurers retreat from AI cover as risk of multibillion-dollar claims mountTom's Hardware article - Major insurers move to avoid liability for AI lawsuits as multi-billion dollar risks emerge — Recent public incidents have lead to costly repercussionsInsurance Newsnet article - Insurers Scale Back AI Coverage Amid Fears of Billion-Dollar ClaimsInsurance Business article - Insurance’s gen AI reckoning has comeSection 230 Section 230 overviewLegal sidebar - Section 230 Immunity and Generative Artificial IntelligenceBad Internet Bills websiteTechDirt article - Section 230 Faces Repeal. Support The Coverage That’s Been Getting It Right All Along.Privacy Guides video - Dissecting Bad Internet Bills with Taylor Lorenz: KOSA, SCREEN Act, Section 230Journal of Technology in Behavioral Health article - Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and PracticeTime article - Lawmakers Unveil New Bills to Curb Big Tech’s Power and ProfitHouse Hearing transcript - Legislative Solutions to Protect Children and Teens OnlineRelevant Kairos.fm Episodes Into AI Safety episode - Growing BlueDot's Impact w/ Li-Lian AngmuckrAIkers episode - NeurIPS 2024 Wrapped 🌯Other Links Encyclopedia of Life websiteIBM Watson AI XPRIZE websiteML Commons websiteWikipedia article
2025/11/24

Against 'The Singularity' w/ Dr. David Thorstad

Philosopher Dr. David Thorstad tears into one of AI safety's most influential arguments: the singularity hypothesis. We discuss why the idea of recursive self-improvement leading to superintelligence doesn't hold up under scrutiny, how these arguments have redirected hundreds of millions in funding away from proven interventions, and why people keep backpedaling to weaker versions when challenged. David walks through the actual structure of singularity arguments, explains why similar patterns show up in other longtermist claims, and makes the case for why we should focus on concrete problems happening right now like poverty, disease, the rise of authoritarianism instead of speculative far-future scenarios. Chapters (00:00) - Intro (02:13) - David's background (08:00) - (Against) The Singularity Hypothesis (29:46) - Beyond the The Singularity (39:56) - What We Should Actually Be Worried About (49:00) - Philanthropic Funding LinksDavid's personal websiteReflective Altruism, David's blogThe Singularity Hypothesis David's Philosophical Studies article - Against the singularity hypothesisTime "AI Dictionary" page - SingularityEA Forum blogpost - Summary: Against the singularity hypothesisJournal of Conciousness Studies article - The Singularity: A Philisophical AnalysisInterim Report from the Panel Chairs: AAAI Presidential Panel on Long-Term AI FuturesEpoch AI blogpost - Do the returns to software R&D point towards a singularity?Epoch AI report - Estimating Idea Production: A Methodological SurveyFunding References LessWrong blogpost - An Overview of the AI Safety Funding SituationAISafety.com funding pageReport - Stanford AI Index 2025, Chapter 4.3Forbes article - AI Spending To Exceed A Quarter Trillion Next YearAI Panic article - The “AI Existential Risk” Industrial ComplexGiveWell webpage - How Much Does It Cost To Save a Life?Wikipedia article - Purchasing power parityPascal's Mugging and the St. Petersburg Paradox Wikipedia article - St. Petersburg ParadoxConjecture Magazine article - Pascal’s Mugging and Bad Explanationsneurabites explainer - Ergodicity: the Most Over-Looked AssumptionWikipedia article - Extraordinary claims require extraordinary evidenceThe Time of Perils Global Priorities Institute working paper - Existential risk pessimism and the time of perilsEthics article - Mistakes in the Moral Mathematics of Existential RiskPhilosophy & Public Affairs article - High Risk, Low Reward: A Challenge to the Astronomical Value of Existential Risk MitigationToby Ord Book - The PrecipiceRethink Priorities blogpost - Charting the precipiceAI Futures Project blogpost - AI 2027Trump's Higher Education Threat Compact Wikipedia article - Compact for Academic Excellence in Higher EducationPen America explainer - What is Trump’s Compact for Higher Education? And More Frequently Asked QuestionsStatement by the Vanderbilt AAUP Executive Committee on the “Compact for Academic Excellence in Higher Education”The Vanderbilt Hustler article - BREAKING: Chancellor Daniel Diermeier fails to reject higher education compact, reaffirms Vanderbilt’s values and openness to discussionThe Vanderbilt Hustler article - Students and faculty organize rally outside Kirkland Hall against Trump administration’s higher education compactFree Speech Center article - Compact for Academic ExcellenceMore of David's Work Global Priorities Institute working paper - What power-seeking theorems do not showBook - Essays on LongtermismVibe Shift Blood in the Machine article - GPT-5 Is a Joke. Will It Matter?Futurism article - Evidence Grows That GPT-5 Is a Bit of a DudGary Marcus substack - GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.Pew Research report - How the U.S. Public and AI Experts View Artificial IntelligenceN...

すべて見る（32）

The Into AI Safety podcast aims to make it easier for everyone, regardless of background, to get meaningfully involved with the conversations surrounding the rules and regulations which should govern the research, development, deployment, and use of the technologies encompassed by the term "artificial intelligence" or "AI" For better formatted show notes, additional resources, and more, go to https://kairos.fm/intoaisafety/

クリエイター

Jacob Haimes
配信期間

23年 - 26年
エピソード

32
制限指定

不適切な内容を含む
Copyright

© Kairos.fm
番組のWebサイト

Into AI Safety