The Experimentation Edge

Growthbook
  • 5.0 (10)
  • Business
  • Weekly

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

  1. Episode 1

    How Walmart Connect learned to align cross-sell with shopper intent

    SummaryArv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how months-long ad-free holdouts measure what ads cost in customer trust. Host Ashley Stirrup also hears the story of a cross-sell test everyone expected to win: showing complementary items on searches like banana encouraged exploration, but that engagement did not consistently translate into the conversion outcomes the team had hoped for. That highlighted an opportunity to better align the experience with shopper intent, so the team rebuilt around the customer's mission. The rebuilt version delivered better CTR and better conversion, and the team submitted a white paper on the work to the AAAI 2027 Innovative Applications of AI conference for consideration. Arv closes with how to design experiments for learnings instead of launches and why self-serve experimentation is the next step beyond the analyst. For product, analytics and retail media teams who run experiments on shared pages.     Chapters00:00 Intro01:42 Balancing metrics across search, shopping agents and advertisers03:08 Guardrails, degradation thresholds and long-term holdouts05:58 The banana problem: rethinking a cross-sell idea08:20 Rebuilding around the customer's mission11:16 Designing experiments for learnings, not launches13:48 Trustworthy experiments and where experimentation goes next15:56 Wrap     Takeaways- Every page owner has their own primary metric; ads teams negotiate a degradation threshold per page and stay inside it- Revenue is not the only guardrail: relevance and latency are long-term metrics, and ad-free holdouts running for months measure what ads cost in trust- Customers engaged with complementary recommendations, but that engagement did not always translate into purchases; the fix was reducing diversity and building themes around the customer's mission- Start every experiment by asking how to make the product more usable, not how to launch it; the hypothesis changes and so do the metrics- A written process (test plan, no early peeking, end-of-test checks) catches bugs early and builds trust; self-serve tooling is what lets PMs experiment without an analyst     Connect with the GuestArv Ramchandani LinkedIn: https://www.linkedin.com/in/arv-ramchandaniCompany Website: https://www.walmartconnect.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    How Walmart Connect learned to align cross-sell with shopper intent

  2. Episode 2

    Engagement with safety: Headspace's guardrails for AI tests

    SummaryMary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.     Mary also shares two hard-earned lessons from earlier in her career: an 18 month redesign that taught her to test early and in small pieces, and a payment processor test where the randomization was not what it seemed. She closes with how Headspace is scaling experimentation and why every portfolio needs a learning bucket. For data, product and growth leaders experimenting on AI features and long term outcomes.     Chapters00:00 Cold open01:02 Headspace and Mary's path from Amazon and HBO03:26 Why more engagement is not always better09:25 Predicting long term outcomes with composite metrics13:29 Testing Ebb: inputs, outputs and guardrails19:28 A/B testing LLM features22:35 Office hours and a culture of experimentation28:46 The redesign that waited too long to test32:03 Know your treatment: the payment processor test38:11 Faster tests and the learning bucket     Takeaways- At Headspace the target is health outcomes, not time in the app, so engagement metrics sit alongside self-reported measures of how members feel- Every experiment on Ebb, Headspace's AI companion, is defined by inputs, outputs and guardrails, with clinicians doing clinical QA- Stop rules written before launch let teams self-serve: lift with no guardrail harm ships, guardrail harm rolls back- Test early and in discrete pieces; an 18 month redesign could not show which change was underperforming- Keep a learning bucket of experiments that never have to report a win, alongside small loop tests and big bets     Connect with the GuestCompany Website: https://www.headspace.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    Engagement with safety: Headspace's guardrails for AI tests

  3. Episode 3

    Motive's Shiva Manjunath on CRO as a mindset, not a role

    SummaryShiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how emotional context earns a CRO credibility, how to make an impact without an executive sponsor, and why experimentation needs the kind of marketing AI gets. For CROs, product managers and growth leaders building an experimentation culture.     Chapters00:00 Cold open: gifts under $5001:18 What a true culture of experimentation looks like04:43 CRO as a mindset across every function11:27 Become the sales team's best advocate15:29 Fear, uncertainty and doubt: the window shade story21:51 Customers ask for price when they want value24:17 GTA 6 and why button colors cost CROs trust29:14 Making an impact without an executive sponsor34:27 Making experimentation as exciting as AI     Takeaways- CRO works best as a mindset shared by design, marketing and product, not as one team's job.- Sales and customer service teams hold free qualitative data that makes strong test ideas.- When customers ask for price, test value messaging such as benefits, ROI and time saved.- Understanding the emotional context around a purchase earns a CRO more trust than tactical button tests.- A culture of experimentation needs data-driven decision making first and a safe space to lose.     Connect with the GuestShiva Manjunath LinkedIn: https://www.linkedin.com/in/shiva-manjunath/From A to B podcast: https://open.spotify.com/show/5xDlubE3GZwUDDen64hefzNewsletter: https://fromatob.beehiiv.com/     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    Motive's Shiva Manjunath on CRO as a mindset, not a role

  4. Episode 4

    Building a Culture of Experimentation in Sports Analytics with New York Red Bulls

    SummaryVamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing test is not a failure, why segments can never be static, where good test ideas come from, and how identity resolution will open the door to machine learning and personalized fan experiences. For analytics, marketing and product leaders building experimentation into a business where every event is different.     Chapters00:00 Intro00:20 New York Red Bulls: an original MLS club01:32 Data-driven fan engagement 04:46 Single game buyers outspend season ticket holders07:21 Testing frameworks 09:29 Holistic fan value 11:11 Segments can never be static12:59 A losing test isn't a failure14:51 Loser or lever: judging tests against the North Star17:13 Building a culture with clear end goals20:25 Ideas populate when there is more data23:10 Identity resolution, AI and personalization26:06 Empowering more teams to experiment27:57 Every game is like hosting a wedding     Takeaways- We have seen instances where single game buyers at the New York Red Bulls outspend season ticket holders once concessions and the Hype House zone are counted- The club moved its North Star from average ticket price to gross margin, which changed couponing, fan profiles and retargeting- A losing test is not a failure; judged against the North Star, a negative result shows what not to do and that a lever exists- Segments should be dynamic, and the club is building toward a separate segmentation for every event- Ideas populate when there is more data, and the stakeholders who know the business best should drive the questions     Connect with the GuestVamsi Krishna Kunapuli LinkedIn: https://www.linkedin.com/in/vamsikunapuli/Company Website: https://www.newyorkredbulls.comVamsi is a panelist at DMWF North America 2026, "Connected Data: The Foundation of Intelligent Marketing"     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    Building a Culture of Experimentation in Sports Analytics with New York Red Bulls

  5. Episode 5

    How Preply runs close to 1,000 experiments a year

    SummarySebastien Polis, Senior Experimentation Manager at Preply, joins host Ashley Stirrup to explain how a learning marketplace with 25 to 30 product teams runs close to 1,000 experiments a year. At Preply, no change reaches customers without a test, OKRs can only be claimed through experiments with an approved target metric, and a public Slack channel puts every launch, result and kill decision in front of the exec team.     Sebastien shares why roughly 80% of experiments lose and 30 to 40% are iterations of an earlier idea, how a global control group shows double-digit growth, and what Preply learned when a control group went negative and a simple wording win turned out to be Twyman's law in action. He also covers why marketplaces randomize tutors instead of learners, the clustered analysis that requires, and his plan to build a learning system that gives AI the context it needs. For product, data and growth leaders building an experimentation culture at scale.     Chapters00:00 Preply's learning marketplace and Sebastien's path into experimentation04:52 Tying OKRs to experiments across 25 to 30 product teams06:39 Why 80% of experiments lose and 30 to 40% are iterations08:35 Double-digit growth from a global control group09:46 The control group that went negative and Twyman's law15:03 How a new PM designs an experiment at Preply17:57 A public Slack channel where execs question every test20:48 When clicks go up but learners don't buy23:23 Building a learning system before feeding AI25:20 An experiment run entirely by AI, with a human approver28:13 Randomizing tutors, not learners, in a 2-sided marketplace     Takeaways- Preply ties OKRs to experimentation: teams can only claim results through experiments that use an approved target metric- Losing is the norm; about 80% of Preply's tests fail and 30 to 40% of each year's experiments are iterations of an earlier idea- A surprising win from a simple change deserves a rerun; Preply unscaled a wording test and the lift disappeared- A public Slack channel where execs question PMs about launches and kills keeps rigor and transparency high across 1,000 tests a year- In a marketplace, randomize the tutor and measure learner outcomes with a clustered analysis, and ship bug-free code because sample size is scarce     Connect with the GuestSebastien Polis LinkedIn: https://www.linkedin.com/in/sebastien-polis-b5aa8256/Company Website: https://preply.com     SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    How Preply runs close to 1,000 experiments a year
See All Season 2 (5)

5
out of 5
10 Ratings

About

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Information