Ep05 - Lost in Translation: Testing the Limits of AI Understanding

In this episode, we talk with Madison Van Doren (AI Research & Strategy Manager at Appen) about the limits of AI translation, what AI safety practices like red-teaming reveal about model guardrails, and why humans in the loop remain essential. Madison explains adversarial prompting, how different large language models reflect the norms and incentives of their builders, and why common translation benchmarks reward surface accuracy while missing cultural nuance. At stake is a larger question: if AI mirrors our words but not our meanings, how can we trust it to serve as a safe partner in education and communication?

We also explore big questions for educators and students:

What skills should humans still master in an AI-saturated world?

What tasks can responsibly be offloaded to machines?

How should we redefine digital literacy for the age of generative AI?

CHAPTERS

0:00-1:30 - Chapter 1 Cold Open: Testing AI’s Limits

1:31-6:54 - Chapter 2: The Trouble with Teaching Machines Language

6:55-15:26  - Chapter 3: Adversarial Prompting: Breaking Models on Purpose

15:27-16:37 - Chapter 4 [Bridge]: Norms in, Norms Out

16:38-28:05 - Chapter 5 - Translation vs Localization: How Benchmarks Can Mislead 

28:06-35:19 - Chapter 6 - Humans in the Loop, New Literacies

35:20-39:04 - Chapter 7 - Closing Reflections: Endless Iteration, Evolving Language, and the Geopolitics of AI

My Robot Teacher is hosted by Sarah Senk and Taiyo Inoue, sponsored by the California Learning Lab, and produced by editaudio. Video Editing by Starline Hodge, Audio Editing by Megan Hayward, and our Production Manager is Kathleen Speckert.

📄 Full transcript: https://calearninglab.substack.com/p/my-robot-teacher-episode-5-transcript

🌐 More about the show: https://www.myrobotteacher.ai

📨 Email us!  We’d love to hear from you!  myrobotteacherpod@gmail.com

🔔 Subscribe for Extras: https://www.youtube.com/@myrobotteacher

🎧 Listen on Apple: https://podcasts.apple.com/us/podcast/my-robot-teacher/id1818032413

Follow us on: 

Instagram: https://www.instagram.com/myrobotteacher/

YouTube: youtube.com/@myrobotteacher

X: x.com/myrobotteacher

Bluesky: https://bsky.app/profile/myrobotteacher.bsky.social

Facebook: facebook.com/myrobotteacher

Tags/ Keywords: Large Language Models (LLMs), AI translation, AI localization, AI safety, Red-teaming AI, Adversarial prompting, Digital Literacy, AI in the classroom, Humans in the loop, AI misbehavior and guardrails, AI and cultural nuance, Teaching with ChatGPT, AI literacy for students and educators, Madison Van Doren, Appen, Linguistics, future of learning, human-centered AI, Sarah Senk, Taiyo Inoue, Cal Poly Maritime Academy, California Education Learning Lab, My Robot Teacher, editaudio