Data Breakthroughs · Season 2, Episode 2 Runtime: [[57]] minutes · Category: Data Analysis & Reporting A note before you start: this is one of the last episodes in this format — and I’m asking for your view on it below. Why this one is worth an hour There’s a version of this problem that’s about dashboards, and it’s the boring version. Somebody built the wrong report; build a better one. The version Timo found in the first five minutes of his brainstorm is different, and I didn’t see it coming. Nothing here is technically broken. The tracking works. The data exists. Three competent people produce exactly what was asked for, every two weeks, on time. And the output is a decision made on instinct anyway. What’s actually broken, in his words, is that “mentally they’re in a bad place”— and that no fix laid on top of that frustration will survive it. That reframe changed the whole session. What follows is where we got to. The problem, as submitted Category: Data Analysis & Reporting. Submitted by: Anonymous. Context: B2C wellness app, Series A, ~250,000 monthly users. Three-person data team. Issue — Every two weeks, before sprint planning, the PM asks the data team what to prioritize. She receives a 40+ chart report from Mixpanel: session duration, feature adoption, retention cohorts, everything. She still can’t make a decision. “The data shows everything but recommends nothing.” She makes gut calls anyway, which feels wrong given how much data is on the table. The cost, per cycle: three people, one full day, to build it. Two to three days for her to read it. Then the deadline hits and the decision goes to whatever the CEO mentioned last week or whatever’s loudest in the support tickets. Trigger — Every sprint cycle. It came to a head last quarter: they redefined what the dashboard needed to show, the data team took seven weeks to rebuild the report, and she still couldn’t decide fast enough. Result: a wasted A/B run and a feature release that caused churn. (Additional context I happened to have, knowing the PM: the Mixpanel API connection was fine, but the aggregation kept collapsing under the event volume — a lot of manual correction every cycle, and eventually a need for different tooling.) Boundaries — Two-week sprint cycle, velocity must hold. Mixpanel stays. And the fix has to start working next sprint, not after a months-long transformation. Tension — Two sentences, both worth sitting with: “The data team is undervalued.” “We are not data informed, we are data paralyzed.” Clarity statement (where we landed live): Reverse-engineer from the decision backwards. Define what success of the product actually looks like, then design the smallest set of metrics that tells the PM where her biggest lever is before a sprint — rather than redesigning the report again. Our guest timo dechau 🕹🛠 — solo consultant, product and growth analytics. Based in Aalborg, Denmark (not Copenhagen, as he’d like on the record). Timo started in product and never really left it — most of what he does in data is still built on product principles. His path ran through classic tracking implementation, then an equally long stretch on the data warehouse side, and in the last two or three years into strategy, which he says he’d written off as “something for old people” until he found out what it’s actually for: setting expectations early enough to prevent the problems that show up later. He works with companies trying to get their warehouse stack into a shape that supports real product analytics, growth prediction, and marketing attribution, and he’s building a product that sits on top of the usual suspects — Amplitude and Mixpanel — to add a more strategic layer to product analytics. Connect with Timo: * Website: timodechau.com * LinkedIn: He posts two or three times a week, and describes LinkedIn as his first sounding board for ideas before they become blog posts or videos Where the two approaches met Timo’s first move: clear the room before you fix anything He got stuck on this for the longest part of his twenty minutes, and it’s the most useful thing in the episode. “I think the biggest problem is — they’re in a bad place mentally. Not business-wise, not from a data perspective. They’re collecting data, stuff is there. But mentally they’re in a bad place.” Frustration like this doesn’t start two months ago. It accumulates, on both sides, and by the time somebody writes “we are data paralyzed” into a problem submission, it has an audience — the product team has told other teams, management has heard about it, and the question in the room has quietly become why hasn’t this been solved already, when everyone else has solved it? (Very few companies have. Plenty claim to. That gap makes the environment harder, not easier.) So his quick win is a stop, not a start: Produce no sprint reporting at all for three sprints. Use the reclaimed capacity — a full day per cycle, per person — to build the replacement. And this cannot be agreed between the product and data teams alone. It goes up to management, it gets stated openly, and something gets delivered at the end. Otherwise the same trap closes again. Then: borrow the product strategy to narrow the scope Product is measurable in a thousand directions, which is why forty charts happened in the first place. Timo’s scope-narrowing device is the strategy nobody in data usually reads. There’s a business strategy. Product derives its strategy from it. Spend time with management and product understanding what they actually want to move in the next twelve months — then translate that movement into metrics. The payoff is political as much as analytical: “When we can come up with some metrics that show strategic progress, everyone is happy — even when we haven’t solved the core problem yet.” It takes pressure out of the room, it tells management whether their strategy is working, and it buys the credit needed to fix the rest properly. Then: measure outcomes, not interactions “One of the big issues of almost all product analytics setups is that it’s focusing on interactions and it’s not focusing on outcomes. Interaction is easier to track — there’s a button, we can click it, we can track it. But the real value comes when you take a step back and say: what are the outcomes of our wellness app? What do we want people to achieve?” Practically, that means event storming sessions to map the journey and identify value moments — even if it’s been done before — and then a shift in how results get presented: * Away from retention curves and cohort reports. Beautiful assets, genuinely useful, and readable by professional analysts. Not by a PM at 9am before sprint planning. * Towards metrics: a one-month, three-month, six-month retention rate. Put them on a time series. Cohort them later if you want. A PM can look at three of those and answer “did the last sprint move anything?” in about ten seconds. * Or user-state measurement, growth-model style: new → activated → active → at risk → dormant, and measure the movement between states. Activation rate, at-risk rate, and so on. Two constraints he names honestly. First, Mixpanel is a poor fit for this: “Mixpanel is not a metric-based tool. It’s an event data exploration tool” — metrics arrive late and aren’t first-class. Second, when the conversation stalls on “our tracking is bad,” the way out is often to stop tracking and instead derive from the product database. That’s source data. “Tracking cannot be good, because it happens in a browser.” Lior’s move: three questions every KPI has to survive I came at it from the other end — the architecture, not the tooling. Start from one to three KPIs that measure business value. Then put each one through three questions before it goes anywhere near a dashboard: * What action will this require of us? If a number moves and nobody does anything differently, it isn’t a KPI. It’s decoration. * Why do I need it? The purpose, stated. Day-7 retention exists so I can tell whether a feature is sticky enough to bring people back. Once that’s written down, question one has an answer. * Who owns the data, and who owns the KPI? Two different jobs. One is accuracy and lineage. The other is the definition, the calculation when it changes, and the communication. Then, in order: design the visualization and the filters → map back to the data sources so lineage is documented → communicate to management and get their buy-in, so nobody is surprised when the number moves → and test the whole thing with a simple CSV before asking anyone to build it. That last step matters more here than usual. The data team is already frustrated. Validating the concept by hand, before commissioning work, is the difference between one build and four. On ownership — my honest answer to Timo’s question about whether anyone ever agrees to own a KPI is a bad, mostly. But the principle holds: the requester is the domain expert, and the domain expert owns it. Handing it to the data team makes no sense, because as Timo put it, “they cannot make the smell test.” They’ll present a number, someone in the domain will say “that can’t be right,” and they’ll have no way to know who’s correct. And the decision book — an idea I took from Philip back in episode three and have since used myself. Alongside the metrics, write down what you do when each one moves. High level, not exhaustive. Each time a new scenario shows up, add it. It’s what turns “the number went down” into a sprint conversation instead of an argument. Three insights 1. Minimal, strategy-linked metrics get you halfway on their own. Both of us arrived at a small number — Timo from the strategy end, me from the KPI end. The convergence matters less than the discipline: don’t have more metrics than you
