Why Your Data Might Be Leading You Astray: Simpson’s Paradox and the Danger of Misleading Results

Here’s a story that should be a wake-up call for B2B SaaS CEOs. Even when you think you’re doing everything right, your data might be leading you toward decisions that could undermine your growth.You’re the CEO of a B2B SaaS company focused on driving growth. Your CMO proposes shifting from your traditional inbound sales motion to a product-led growth (PLG) strategy. Being cautious, you decide to run a test rather than jump straight in.Here’s what happened:Q1 Results:Inbound: 582 opportunities, 183 deals, CVR = 31.44%PLG: 140 opportunities, 45 deals, CVR = 32.14%Encouraged by the results, you expand the PLG test in Q2.Q2 Results:Inbound: 48 opportunities, 12 deals, CVR = 25%PLG: 411 opportunities, 104 deals, CVR = 25.30%After two quarters, it seems like a no-brainer: PLG is outperforming inbound. The higher conversion rates suggest PLG is the future, right?Not so fast.When you combine the results from both quarters, the data tells a different story:Combined Results:Inbound (Q1 + Q2): 630 opportunities, 195 deals, CVR = 31%PLG (Q1 + Q2): 551 opportunities, 149 deals, CVR = 27%Suddenly, your traditional inbound motion is performing 15% better than PLG. How can that be?This is Simpson’s Paradox—a statistical phenomenon where trends that appear in separate data sets reverse when you combine them. Though I used a test of PLG to highlight the challenge, Simpson’s Paradox can occur in many areas:- College admissions- Medical treatments- Income distributions- Sports- A/B testing- And many more...In fact, in this example, the data comes from a real-life instance from baseball. In 1995 and 1996, David Justice had a higher batting average than Derek Jeter. But when you combine the two years, Jeter comes out on top. The data flips when looked at holistically.I’ve been hearing lately about how easy it is to use data to guide marketing decisions, but the truth is, even simple tests can lead you astray if you’re not careful. This example might seem straightforward, but that’s the point—even the simplest decisions can be wrong for reasons most people would overlook.The Lesson: What looks like straightforward data can be deceptively misleading. Before making any decisions, ask yourself:- Are we analyzing the data in the right way?- Are we weighing results properly across different cohorts and time periods?- Are we sure that short-term trends won’t reverse when looked at over time?Making the wrong call based on faulty interpretation of data can be costly. Making the right decision might not be easy. Always dig deeper—your growth depends on it.