The Experimentation Edge

Growthbook

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

  1. The 'wine effect' and other surprises that reshaped how Box runs e-commerce experiments

    9 hr ago

    The 'wine effect' and other surprises that reshaped how Box runs e-commerce experiments

    Summary In this episode of The Experimentation Edge, host Ashley Stirrup talks with Danielle Oleen, Director of E-commerce at Box, about what it really takes to build a culture of experimentation inside a B2B company. Drawing on 15 years across B2C and B2B at Wayfair, Drizly, Zoom, and now Box, Danielle explains why experimentation belongs to every product team and not just e-commerce, walks through a pricing page saga of one win and two losses that exposed the limits of simplification, and shares the "wine effect" test that won for a reason no one predicted. It's a practical, story rich conversation for product managers, growth leaders, and anyone trying to make better decisions with data. Chapters 00:45 Meet Danielle Oleen and Box's reinvention 02:45 Owning the entire customer life cycle 04:45 Why experimentation matters even without a checkout 07:45 The feature that's used but hidden 11:45 Proving ROI with a scrappy manual test 12:45 Building a culture that shares wins and losses 16:45 The pyramid strategy for prioritizing tests 18:45 The simplification tightrope on the pricing page 24:45 When a test wins for the wrong reason 27:45 Where experimentation at Box goes next Takeaways - Experimentation isn't only for e-commerce. Any product with a funnel, even an AI chatbot, can be measured and improved through testing. - Simplification has a limit. Removing too much can strip away the cues and context buyers actually need to decide. - Share losses as openly as wins. Wins build credibility, and losses build the psychological safety a testing culture runs on. - Prioritize like a pyramid. Fix the widest-impact experiences first, then optimize down into smaller cohorts. - Surprising results are the point. A test can win for a reason you never hypothesized, like the "wine effect," and that's where the real learning lives. Connect with the Guest LinkedIn: https://www.linkedin.com/in/dolean1/ Website: https://www.box.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    31 min
  2. Dilligent explains why moving on from an experiment might cost you

    2 days ago

    Dilligent explains why moving on from an experiment might cost you

    Summary Dan Layfield, Director of Product Management at Diligent, joins host Ashley Stirrup on The Experimentation Edge to trace what fifteen years of A/B testing across Codecademy, Uber Eats, and the Fortune 1000 boardroom actually taught him. He breaks down the Codecademy trial-model rebuild that took four months and several rounds to deliver a 35% conversion lift, why moving on from a losing experiment too early is one of a PM's costliest mistakes, how to escape the B2B feature factory with metrics that genuinely ladder up, why retention should ride a product's natural use case instead of fighting it, and where AI is already replacing weeks of research and analysis. It's a practitioner's guide for product managers, growth leaders, data scientists, and engineers bringing experimentation rigor to both B2C and B2B. Chapters 00:45 Meet Dan Layfield and Diligent 01:45 Two worlds of experimentation, Codecademy and Uber 03:45 The trial model that lifted conversion 35% 06:20 What to do with a losing experiment 08:50 Two flavors of experimentation 09:45 Reading forty metrics at Uber Eats 13:10 Escaping the B2B feature factory 16:45 Anchoring the North Star to real usage 19:15 Where AI fits in research and analysis Takeaways A losing experiment is often inconclusive, not negative; treat it as a map of the funnel rather than a verdict, and know when a big problem is worth another round.Persistence paid off at Codecademy: four months and three to four rounds of trial-model testing produced a 35% conversion increase.Separate your two experimentation modes; high-volume CRO chases many small wins, while big, uncertain bets are worth taking multiple shots to de-risk.Most B2B product teams are feature factories; the fix is a top-down OKR system, and planning usually breaks in the connections between layers, not inside them.Anchor retention and engagement to the product's natural use case, and use AI to synthesize research and simple A/B analysis in hours instead of weeks.Connect with the Guest LinkedIn: https://www.linkedin.com/in/layfield/ Website: https://www.diligent.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    22 min
  3. The metric Stitch Fix says every experimenter should chase

    2 Jul

    The metric Stitch Fix says every experimenter should chase

    Summary In this episode of The Experimentation Edge, GrowthBook CMO Ashley Stirrup sits down with Nick Beyler, data science manager at Stitch Fix, where he leads the decision and insights team and owns the company's internal experimentation platform. Nick shares why the metric he most wants is the one he can't measure yet, a North Star that predicts a client's long-term value from their earliest behaviors, and why the most impactful experiment learnings tend to come from adoption friction rather than product bugs. He makes the case that if you're only testing winners you're not taking enough risks, explains how guardrails make that risk safe, and looks ahead to a new in-house platform and the promise of agentic AI. It's a practical, statistician's-eye view of experimentation for product managers, data scientists, and engineers building serious testing programs. Chapters 00:00 Cold open and welcome to the show 01:45 What Stitch Fix actually does 04:15 Balancing AI with the human stylist 05:15 From public policy to the A/B testing adrenaline rush 07:15 Inside the weekly experimentation review group 08:45 The AI style assistant and listening to qualitative feedback 10:45 Why adoption friction beats product bugs 13:45 Testing for losers and building guardrails 15:45 Keep rate, successful fixes, and the holy grail metric 18:15 The new platform and the promise of agentic AI Takeaways The most impactful experiment learnings usually come from adoption friction, not product bugs. By the time a big feature reaches A/B testing, it's often already a winner, so the open question is how and where to introduce it.A losing test is a finding, not a failure. If every experiment wins, you're not taking enough risk to learn anything new.Guardrails and stopping criteria are what make risk-taking safe, especially when the experience is as personal as shopping.The most valuable North Star metric is the one you can't measure yet, long-term client value, and causal-inference modeling helps predict it from short-term behavior.Quantitative results are only half the story. Direct, qualitative client feedback inside an experiment often reshapes the rollout more than the numbers do.Connect with the Guest LinkedIn: https://www.linkedin.com/in/nick-beyler-381864119/ Website: https://www.stitchfix.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to http://growthbook.io

    21 min
  4. What the Expedia Group cannot measure, it cannot ship

    1 Jul

    What the Expedia Group cannot measure, it cannot ship

    Summary Amir Moghaddam, Director of Software Engineering at Expedia Group, joins host Ashley Stirrup on The Experimentation Edge to make the case that measurement is not a reporting step but a gate: what you cannot measure, you cannot ship. Drawing on nearly four years at DoorDash and his current work leading Expedia's air booking platform, Amir explains why he refuses to label experiments winners or losers, how a "failed" pricing test pushed his team toward full personalization, and why a three sided marketplace forces hard trade-offs between competing metrics. The conversation closes on how the same experimentation discipline now applies to shipping and measuring AI. Built for product managers, engineers, data scientists, and growth leaders who care about rigor over opinion. Chapters 00:00 Cold open00:50 Meet Amir and the air booking platform at Expedia03:10 DoorDash, growth, and a 70 experiment year04:20 Three kinds of experimentation at Expedia06:30 AI velocity and the new frontier model pace08:30 What you cannot measure, you cannot ship10:45 The DoorDash carousel and the price experiment12:45 The three sided marketplace and competing metrics16:55 There are no losing experiments20:45 Predictability, LLMs, and Expedia's road ahead Takeaways "What you cannot measure, you cannot ship" — if you can't measure an outcome, you can't decide whether it's better, so you're just debating opinions.Measurement spans three live dimensions: spend (more with less), speed (sprints instead of quarters), and quality, with guardrail "do no harm" metrics on top.There are no losing experiments. A flat result is a signal to either refine the hypothesis or step back and look from a completely different angle.DoorDash's price experiment proved price by itself doesn't predict orders. Different customers want different things at different times, which pushed the team toward personalization.A three sided marketplace (buyers, merchants, Dashers) makes metrics compete. Running the test is easy; deciding what to optimize when goals conflict is the real work.Connect with the Guest LinkedIn: https://www.linkedin.com/in/amirmoghaddamWebsite: https://www.expediagroup.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io

    29 min
  5. How Fin went from weeks to hours of analysis using AI

    30 Jun

    How Fin went from weeks to hours of analysis using AI

    Summary In this episode of The Experimentation Edge, host Ashley Stirrup sits down with Raunak Kumar, senior manager of GTM analytics at Fin (formerly Intercom), to unpack how experimentation actually works when the data is messy and the traffic is thin. Drawing on nearly 12 years in marketing analytics across Atlassian, Stripe, and Fin, Raunak explains how AI tools like Claude Code have collapsed analysis from weeks to hours and freed his team to clear its experiment backlog, why declining organic search traffic and a 5x jump in untagged ChatGPT referrals are forcing teams to rethink attribution, and how the most valuable experiments are often the ones that "lose." From a Jira Service Desk bundling test that won on trials but had to be rolled back, to a Stripe contact form that was quietly blocking real buyers, this conversation is a practical guide for product managers, engineers, data scientists, and growth marketers who want to learn more from every test they run. Chapters 0:45 Welcome and what the show is about1:45 Raunak's role and 12 years in marketing analytics2:45 How AI and Claude Code changed the analyst's day4:15 LLMs, declining organic traffic, and the 5x ChatGPT jump5:15 Two kinds of experiments at Fin: on page and off page7:15 The Jira Service Desk bundling experiment10:45 Why the trial winner became a rollback11:45 Contextual onboarding turns the loser into a winner14:45 Reading an experiment that loses18:45 What's next: incrementality, connected TV, and testing creative Takeaways AI has collapsed marketing analysis from weeks to hours, and the real payoff is a cleared experiment backlog plus analysts who compete on the questions they ask, not the speed they query.Organic search traffic is declining as ChatGPT, Gemini's AI mode, and Claude answer buyers in place; Fin saw a 5x rise in ChatGPT referrals, but LLMs don't tag that traffic, so attribution has to be proven through experiments.A guardrail metric saved Atlassian from a costly mistake: bundling Jira Service Desk lifted trials more than 50 percent but tanked activation and paid conversion, forcing a rollback.A failed test can hold the real winner; contextual onboarding matched to user intent roughly doubled activation and became the default variant after the bundling experiment was rolled back.In low-volume B2B, read losing experiments for sub-segment signal; a "failed" Stripe form simplification revealed the form was blocking legitimate small-business buyers using Gmail.Connect with the Guest LinkedIn: http://linkedin.com/in/raunakkumar1991Website: https://fin.ai SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

    23 min
  6. Inside The Home Depot's experimentation at a $25B scale

    29 Jun

    Inside The Home Depot's experimentation at a $25B scale

    SummaryWhat does experimentation look like inside a $150 billion retailer? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Kim Ting Li, Senior Manager of Experimentation at The Home Depot, where one centralized team tests every major change to a $25 billion online business. Kim explains how 40 people serve 40–50 business teams, why executives join test readouts and ping analysts directly, how every result since 2020 lives in a searchable library, and why scaling beyond hundreds of experiments per year depends on server-side testing capabilities more than AI. For product, data, and engineering leaders building or scaling experimentation programs. Chapters 00:00 Intro 00:45 From neuroscience research to Home Depot 01:45 A $150B enterprise, a $25B online business 02:45 The centralized experimentation model 03:45 Inside the 40-person team 04:30 Readouts, blast emails, and the experiment library 05:40 Executive visibility and the golden rule 06:15 "If you won't act on a bad result, don't run the test" 11:15 Learning from losing tests 12:30 Scaling up: AI, server-side testing, and what's next Takeaways One centralized team of about 40 people tests every major change to Home Depot's $25B online business, serving 40–50 business teams with consistent hypothesis and analysis standards.Executive engagement is real at Home Depot: leaders join 30-minute readouts, search the experiment library, and ping analysts directly because they treat A/B testing as the golden rule for measuring incrementality.Institutional memory is infrastructure — every test result since 2020 lives in a centralized, searchable archive so no one re-runs a question the company already answered.Kim's stakeholder filter: if you wouldn't do anything differently after a bad result, don't run the test.Scaling past low hundreds of experiments per year is a capabilities problem before it's an AI problem — Home Depot is moving from client-side to server-side testing so winners release quickly, end to end. Connect with the Guest LinkedIn: https://www.linkedin.com/in/kimtingli Website: https://www.homedepot.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

    12 min
  7. How Disney picks which experiments to run

    29 Jun

    How Disney picks which experiments to run

    Summary What does it look like to kill a multimillion dollar feature before anyone builds it? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Crystal Ammari, a digital product optimization and experimentation strategy leader whose career spans Nike and The Walt Disney Company. Crystal shares the "dry test" that used a single fake button to measure demand for video chat (4 million users, 106 clicks), why she reframes experimentation as savings and gains rather than wins and losses, how a misconfigured tool, not bad methodology, made tests take six months, and how a stuck Disney team went from "we don't know where to start" to 110 scored and prioritized test ideas. For product, data, and engineering leaders building or scaling experimentation programs. Chapters 00:00 Intro 00:45 The mindset shift from shipping to results 02:00 Why testing took six months, a tooling problem 03:15 The dev team that laughed, and the vendor who agreed 04:50 An executive demand for video chat 05:35 Dry testing with a fake button 06:30 106 clicks and a multimillion dollar save 07:30 Savings and gains, not wins and losses 08:45 The Disney team that didn't know where to start 10:30 From low engagement to 110 prioritized ideas 12:45 Just get something live, and where AI fits next Takeaways A "dry test", a fake "Click here to video chat" button that grayed out on click — measured real demand without building the feature. Of roughly 4 million users, only 106 clicked, killing a multimillion dollar build.Reframe experiment outcomes as savings and gains, not wins and losses. A "losing" test saves you from a costly mistake, which keeps teams focused on learning instead of fearing failure.Slow experimentation is often a tooling problem, not a methodology problem. One program's six month test cycle came from rebuilding every page instead of overlaying changes the way the tool intended.Getting a stuck team unstuck starts with data and a workshop. A Disney team went from "we don't know where to start" to 110 scored, prioritized test ideas, using Contentsquare heatmaps to diagnose low engagement first.The biggest thing that gets a team testing is to just do it. Stop designing the perfect experiment and get something simple live to take away the mystery. Connect with the Guest LinkedIn: https://www.linkedin.com/in/crystal-ammari/ Website: https://thewaltdisneycompany.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

    33 min
  8. Ship faster, measure better: experimentation tips from JPMorgan Chase

    25 Jun

    Ship faster, measure better: experimentation tips from JPMorgan Chase

    Summary How do you know if the thing you just shipped actually worked? On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Kevin Yang, Executive Director and Head of Experimentation at JPMorgan Chase, who has spent six years building experimentation across Chase's digital platforms. Kevin shares how his team turned experimentation into more than a billion dollars of estimated value, why the losing experiments matter more than the winners, and the simple chart exercise he uses to prove that a million-dollar change is invisible without a control group. He and Ashley also dig into measuring engagement without chasing vanity metrics, planning for failure to defeat confirmation bias, and why AI is pushing experimentation into a golden era. It's a practical look for product managers, data scientists, and engineers at how a bank operating at massive scale makes better decisions. Chapters 00:00 Welcome to the experimentation edge 01:45 Kevin's role leading experimentation at chase 04:15 Why chase invested in experimentation 06:45 A billion dollars and the value of losers 12:45 Plan for failure to beat confirmation bias 14:30 The million dollar change you can't see 18:45 Sharing learnings and experimentation wrapped 20:45 Engagement without vanity metrics 22:00 Experimentation's golden era with AI 23:30 Why AI needs more experimentation, not less Takeaways Chase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.A control group is non-negotiable: at scale, a change worth millions is invisible under noise and seasonality, and no one can spot it by eye.Treat engagement carefully. For a bank, more time in the app isn't a win; trust, fast task completion, and healthy repeat engagement are.Plan for failure before you run a test. A pre-built playbook for a loss prevents confirmation bias and keeps teams from gaming the metrics.AI is ushering in a golden era for experimentation, because shipping faster only compounds mistakes unless you measure what you ship.Connect with the Guest LinkedIn: https://www.linkedin.com/in/kevintyang Website: https://www.jpmorganchase.com SponsorGrowthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts. Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse. With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction. See a demo at https://www.growthbook.io/

    26 min
See All (26)

About

How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale. Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams. Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.

Information

  • Creator
    Growthbook
  • Years Active
    2k
  • Episodes
    26
  • Rating
    Clean
  • Copyright
    © Ashley Stirrup
  • Show Website
    The Experimentation Edge
  • Provider

You Might Also Like