Rand Fishkin’s team ran 2,961 prompts across ChatGPT, Claude, and Google AI. 600 volunteers, 12 different prompts, two months of runs. They wanted to answer one question: how often do you see the same list of brand recommendations twice, even with the exact same prompt? The answer? Less than 1% of the time. The odds of seeing the same list in the same order are closer to one in a thousand. Most conversations about AI inconsistency treat it as a measurement problem: how do I know if my brand is showing up? That’s a legitimate question. But it’s not the only question. And it might not even be the most important one. If AI systems give different recommendations essentially every time, the same inconsistency is already baked into every AI chatbot you’ve deployed — your hotel chat widget, your B2B sales assistant, your customer service tool. Most teams have never measured it. And some of those inconsistent answers are already driving negative reviews for your brand and business. This episode connects three stories Tim has covered over the last three weeks — the AI value gap, the uncertain timeline of agentic commerce, and now AI inconsistency — showing that they all stem from the same underlying condition. It also explains what City of Hope, appearing in 69 of 71 AI responses for "West Coast cancer care hospitals," tells us about how you can fix this problem for your business. Key Insights for Strategic Leaders to Close the Gap In this episode, Tim Peter breaks down: The full SparkToro/Gumshoe.ai research — and what it actually means. Rand Fishkin and Patrick O’Donnell ran nearly 3,000 prompts with 600 volunteers. The list of brands recommended changed more than 99% of the time. Here’s why that reframes everything about how you should be tracking AI visibility. The operational problem most people are missing. AI inconsistency isn’t only a marketing measurement challenge — it’s a liability inside your own deployed tools. Your AI chatbot may be giving materially different answers to different customers right now. And, it’s almost certain that no one on your team is measuring that. City of Hope: what 97% consistency looks like. Why City of Hope appeared in 69 of 71 AI responses for "West Coast cancer care hospitals" and what that reveals about how AI decides which brands it’s willing to commit to — and which ones it isn’t. Why "post more content" is the wrong strategy. How AI actually works: triangulation across independent sources, why your own website is a low-weight signal, and what "digital witnesses" means for building prompt brand equity that holds up. The King, Queen, and Crown Jewels operating model. Content is king, customer experience is queen, and data is the crown jewels, not just as a branding concept, but as the mechanism that drives the AI’s confidence in your brand. Four moves to make this week. Shift from rank to frequency measurement. Audit your deployed AI tools for consistency before worrying about external AI visibility. Build credible witnesses, not content volume. And treat review velocity as a strategic input, not just a reputation metric. Whether you’re in hospitality, retail, or B2B, this episode is for anyone who’s deploying AI in a customer-facing role… or who’s who’s being asked to report on AI visibility and wants a better sense of what they’re actually measuring. The AI Coin Flip: Why AI Gives Every Customer a Different Answer (Digital Reset Episode 488) — Headlines and Show Notes Show Notes and Links NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility – SparkToro Rand Fishkin proved AI recommendations are inconsistent – here’s why and how to fix it Agentic Commerce: ChatGPT Bails on Its Shopping Plans (Ep. 487) The AI Value Gap: Why 82% of Companies are Failing to Gain from AI (Digital Reset Episode 486) SEO vs GEO: How to Show Up When AI is the Concierge Peec.AI — AI brand visibility measurement seoClarity — AI search visibility and GEO tools Buy the Book — Digital Reset: Driving Marketing and Customer Acquisition Beyond Big Tech Tim Peter has written a new book called Digital Reset: Driving Marketing Beyond Big Tech. You can learn more about it here on the site. Or buy your copy on Amazon.com today. Past Appearances Rutgers Business School MSDM Speaker: Series: a Conversation with Tim Peter, Author of "Digital Reset" Free Downloads We have some free downloads for you to help you navigate the current situation, which you can find right here: A Modern Content Marketing Checklist. Want to ensure that each piece of content works for your business? Download our latest checklist to help put your content marketing to work for you. Digital & E-commerce Maturity Matrix. As a bonus, here’s a PDF that can help you assess your company’s digital maturity. You can use this to better understand where your company excels and where its opportunities lie. And, of course, we’re here to help if you need it. The Digital & E-commerce Maturity Matrix rates your company’s effectiveness — Ad Hoc, Aware, Striving, Driving — in 6 key areas in digital today, including: Customer Focus Strategy Technology Operations Culture Data Subscribe to Thinks Out Loud Subscribe in iTunes Subscribe in the Google Play Store Contact information for the podcast: podcast@timpeter.com Past Insights from Tim Peter Thinks Technical Details for Thinks Out Loud Recorded using a Shure SM7B Vocal Dynamic Microphone and a Focusrite Scarlett 4i4 (3rd Gen) USB Audio Interface. Running time: 22m 01s You can subscribe to Thinks Out Loud in iTunes, the Google Play Store, via our dedicated podcast RSS feed (or sign up for our free newsletter). You can also download/listen to the podcast here on Thinks using the player at the top of this page. Transcript: The AI Coin Flip: Why AI Gives Every Customer a Different Answer Welcome back to the show. I’m Tim Peter. I’ve talked about a concept called "Prompt Brand Equity" for a while now, the idea that what matters in AI search isn’t where your brand ranks. It’s whether you show up at all, whether your brand shows up at all. And I mentioned some early research from Rand Fishkin at SparkToro, shows that AI recommendation lists were unpredictable. You could be number one in one chat and number three in the next, even with the exact same prompt, by the exact same person. Well, the full research is out now, and the numbers are much more striking than I expected. Rand’s team ran 2,961 prompts through ChatGPT, Claude and Google AI with 600 volunteers over two months. The question they were trying to answer, how often do you see the same list of brand recommendations twice, even if you run the exact same prompt over and over? The answer? Less than 1% of the time. I want to say that again. Less than 1% of the time do you see the same list twice. In other words, practically never. That has real consequences for how you measure your business’s AI visibility and for how you think about the AI tools that you’ve already deployed in your own business. And for the ROI gap that I covered on episode 486, which if you missed it, was about why 88% of companies are using AI, but only 6% are seeing significant value from it. It turns out that these trends, these traits, these facts are connected. Today I want to get into how. This is episode 488 of Digital Reset with Tim Peter. I’m Tim Peter. Let’s dive in. Okay. Let me start with what the research actually found, because the headline number undersells it a little. Rand Fishkin partnered with Patrick O’Donnell at a company called Gumshoe.ai. They recruited 600 volunteers to run 12 different prompts, things like "recommend headphones under $300," or "what are the best project management tools." They ran these through ChatGPT, Claude, and Google Gemini, Google AI, over and over for two months, nearly 3,000 runs in total. And what they found is this: the list of brands recommended changes more than 99% of the time. The odds of seeing the same list in the same order twice are closer to one in a thousand. That’s nuts, right? So I wanna be fair about what this means and what it doesn’t mean. It doesn’t mean that AI is useless. It doesn’t mean that brand mentions in AI are random and it definitely doesn’t mean you should give up on showing up in AI answers. Far from it. What it means is that where you appear in any given AI response, whether you’re number one or number three, tells you essentially nothing. That position is random. It’s not predictive of anything to you or to your business. The useful metric isn’t rank. It isn’t where you show up. It’s frequency. How often does your brand appear at all, across a large sample of runs on the questions that matter to your customers. That number tells you something real. That number is, of course, prompt brand equity, not position, frequency. I mentioned Rand’s early work on this in episode 485 when I talked about how we’ve moved from a world of card catalogs to a world of concierges. The new data just puts specific numbers into what we already expected. The picture is much clearer now, and if I’m being really honest, a little more dramatic than I expected. Now, here’s what I think is the most under-reported part of the story, and it matters a lot if you are in hospitality or honestly, if you’re in any business that has deployed AI in a customer facing role. When people talk about AI inconsistency, they almost always frame it as a marketing measurement problem. You know, how do I know if my brand is showing up? And it’s a legitimate question. I’ll get back to that in just a moment, but it’s not the only problem here. The second problem is operational, and it’s happening right now in your business. If AI systems give different recommendations essentially every time to cus