Data Science in the Wild

Peter Liu

How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact. Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It's data science in the wild: not textbook theory, but the models and decisions real companies run in production. New episodes several times a week, about 15–20 minutes each, so they fit a commute. Hosted by Peter Liu, a data scientist with 10+ years in consumer tech (Meta, Webflow, Adobe, eBay). Episodes are produced with AI-generated voices from curated source articles and papers, which are linked in each episode's notes.

  1. 13h ago

    Huawei — Does the Latest Search Actually Drive Recommendations?

    Do recommenders use your latest search? Huawei's FocalSwap audit of 66,139 rows finds the credit flips with the control: masking credits search, matched swaps don't. Key takeaways: The control decides the credit. On the same 66,139 rows across three UniSAR checkpoints, swapping all search inputs for a neutral ID gives a positive contribution (+2.0 to +5.4 ×10⁻³ reciprocal rank). Category/profile-matched real searches give −1.18 to −0.35 ×10⁻³, below zero for one checkpoint. The paired gap (+2.8 to +6.6 ×10⁻³) is above zero every time and holds across five alternative donor assignments.Latest-search credit isn't established. Changing only the latest search slot gives +0.138 ×10⁻³ for SESRec and −0.213 to +0.089 ×10⁻³ for UniSAR. Every interval crosses zero, under matched donors too. That's “not established”, not proof of zero.Moving ranks isn't the same as helping. Neutral masks change 59.7–61.7% of target ranks and matched swaps 44.1–44.8%, but only masking yields a positive mean.The audit does catch real signals. On GenSAR's current-query task (8,958 rows), masking the query cuts RR by 0.035 and a matched donor query by 0.106. With Qilin's text, BM25 shows a focal-query edge (+0.084 over placebo) only on a 6.1% high-overlap subset.Our take. Pair masks with matched replacements before crediting search. The paper doesn't say whether masks look better because donors keep useful coarse info or because a neutral ID disrupts the model; results cover only the audited checkpoints.Chapters: 00:00 Cold open: did your search drive that pick? 00:53 Welcome: which search input earns the lift? 02:28 Why attribution matters for platforms 03:31 Headline: the control changes the answer 04:25 FocalSwap: fixed rows, two scopes, two controls 07:41 Reciprocal rank, KuaiSAR, Qilin, checkpoints 09:35 Calibration: planted signal and GenSAR 10:19 All-source results on UniSAR 12:57 Latest-slot results and Qilin text evidence 14:28 Why masking can mislead; related work 17:26 Our take, caveats and recap Sources: Do Search-Augmented Recommenders Use the Latest Search Episode? (Jiandong Ding (Huawei); arXiv, Nov 4, 2026; WSDM '27)FocalSwap code (GitHub)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Huawei — Does the Latest Search Actually Drive Recommendations?

  2. 1d ago

    University of Granada — Do LLM Recommenders Pick by Position?

    Do LLM recommenders pick by position? A University of Granada study ran 40,000 shuffled prompts through 8 open models, and every one failed both statistical tests for position independence. Key takeaways: Order matters for every model. On a RAG testbed (50,000 Amazon products, 2,000 queries × 20 orderings of the top-5 candidates), all 8 models (LLaMA 3.1 8B/70B, LLaMA 4 Scout, GPT-OSS 20B/120B, three DeepSeek R1 distills) rejected position independence under both tests, POB-ICM (chi-square on a contingency matrix) and POB-UNBP (a uniformity test).Reordering flips picks often. Permutation sensitivity ran from 15.5% (GPT-OSS 120B) to 34.0% (DeepSeek R1 7B); positional bias, the distance from uniform, from 2.4% to 5.9%.Soft vs. strong bias. Strong positional bias, when position pushes the pick outside the comparably relevant items, ranged from 1.0% to 9.8%. The paper treats it as an upper bound: for DeepSeek R1 7B, decoding noise alone gives 9.1%.Shape depends on the model, and grows with list size. GPT-OSS and LLaMA 3.1 70B favor early slots, LLaMA 4 Scout under-picks slot 1. With 20 candidates, Scout's positional bias rises from 2.9% to 33.8%; raising temperature doesn't remove it.Our caveats. One catalog, one fixed (untuned) prompt template, single-turn top-1 recommendation. The authors suggest routine audits and order randomization.Chapters: 00:00 Cold open: would an AI pick by position? 00:41 Welcome: why prompt order matters 02:01 Setup: LLM recommenders on top of RAG 04:27 Testbed: Amazon catalog, 8 LLMs, 40K runs 06:22 Method: POB-ICM vs. POB-UNBP 09:36 Results: sensitivity, bias, strong bias 13:11 Which positions each model favors 14:26 Robustness: list size and temperature 15:52 Novelty and related work 18:26 Our take, caveats and recap Sources: Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence (Peláez-González, Herrera, Herrera-Poyatos (University of Granada); arXiv, Oct 27, 2026)Code (GitHub)Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) (arXiv)Large Language Models are Zero-Shot Rankers for Recommender Systems (Hou et al., 2023) (arXiv)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    University of Granada — Do LLM Recommenders Pick by Position?

  3. 2d ago

    Independent research — Should You Recalibrate CTR Models on Fresh Data?

    Should you recalibrate CTR models on fresh data? Rui Liu's Avazu study: a 6-hour Platt/isotonic refit made calibration worse in about a third of applications. Key takeaways: Random splits flatter calibration. On Avazu (40.4M impressions; LR, DeepFM, DCN-v2), a random split reported LR's ECE at 55–62% of its value on the nearest future day in every seed. For the deep models the ratio was seed-unstable, and no model's ECE grew steadily with distance from training.Recalibration can backfire. Platt or isotonic refits on the preceding 6 hours removed up to 61% of ECE in badly miscalibrated cells, but increased it in 14–17 of 48 applications. Worst case: LR under Platt, 0.0054 to 0.0112 (+107%).Span beats volume. A 12-hour slice cut harm from 13–17 to 3–5 of 36 applications; 24 hours raised it again. A 93K-impression subsample spread over those 12 hours (5% of the volume, 20–25× faster to fit) recovered 88% of the gain. Why span helps (hour mix, age, distinct hours) isn't separated yet.No easy gate, small profit effects. Gating on the slice's ECE showed no benefit out-of-origin. In a simulated second-price auction, recalibration moved profit by −0.21% to +0.26%.Our caveats. One 2014 dataset, two overlapping origins, and a fixed value per click of 1.0.Chapters: 00:00 Cold open: bidding on miscalibrated probabilities 01:42 Why calibration is what ad systems consume 04:38 Setup: Avazu, three models, random vs. temporal 08:11 RQ1: random splits flatter calibration 09:46 RQ2: no steady drift with distance 10:26 RQ3: recalibration harms a third of the time 11:49 Slice-length sweep: span beats volume 13:17 Can a pre-serving gate prevent harm? 14:07 RQ4: profit in a simulated auction 15:26 Novelty and related work 17:40 Our take and caveats 19:19 Recap and closing question Sources: How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration (Rui Liu (independent researcher); arXiv, Oct 23, 2026)Code and per-cell results (GitHub)Field-aware Calibration (Pan et al., WWW 2020) (arXiv)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Independent research — Should You Recalibrate CTR Models on Fresh Data?

  4. 3d ago

    Spotify — Can You Ship an LLM Change on Evals Alone?

    Can you ship an LLM change on evals alone? Spotify's Mårten Schultzberg: only if eval effects predict A/B results. On 183 Upworthy tests, his gate ships none. Key takeaways: The question. Judge–human agreement doesn't tell you which changes can skip an A/B test. That needs trial-level surrogacy: eval treatment effects that predict online effects across a class of interventions. It's logically independent of unit-level (Prentice) surrogacy.The gate. Subtracting each intervention's reported noise variances from the covariance undoes attenuation (a random-effects correction dating to DerSimonian and Laird). A bivariate-normal model then gives a closed-form ship/test/kill rule. In the running example (R² 0.6, α 0.1) it ships above +2.94 pp and kills below −3.02 pp: 14% ship, 6% kill.Judges and audits. A judge 3 pp less sensitive to treated outputs turns a +2.0 pp gain into −0.25 pp. In a simulated reward-hacking break, the unaudited gate's false-ship rate hit 0.31 vs. a promised 0.10; a 0.4 audit rate caught it in 96% of runs.Upworthy. A GPT-4o-mini judge rated headline informativeness; the outcome was click-through. Measurement was precise (reliability 0.93 and 0.74), but corrected pooled R² was 0.023 and the ship threshold 134 pp, above the 100 pp maximum, so all 183 went to A/B tests.Our take. Fix intervention classes before estimating: two classes with zero surrogacy can pool to 0.81.Chapters: 00:00 Cold open: can an LLM judge replace A/B tests? 00:46 The paper, and why judge agreement isn't enough 03:25 Interventions as the unit; noise and attenuation 05:38 Moment correction and the ship/test/kill gate 06:56 Upworthy: 183 headline tests 09:29 Fail-closed, and when the gate pays off 10:52 Arm-dependent judge error 12:09 The closed loop and randomized audits 13:54 Trial-level vs. Prentice surrogacy 15:19 Related work 16:36 Our take: don't pool intervention classes 18:27 Recap and closing question Sources: When Can You Ship on Evals Alone? Trial-Level Surrogacy for Offline Evaluation of LLM Systems (Mårten Schultzberg (Spotify); arXiv, Oct 7, 2026)Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference (Persson, Schultzberg, Ankargren; arXiv 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Spotify — Can You Ship an LLM Change on Evals Alone?

  5. 6d ago

    Brown University — Is Difference-in-Differences Actually Broken?

    Is difference-in-differences broken? Brown economist Peter Hull used AI agents to replicate 1,346 staggered-adoption results from 108 papers: TWFE and modern DiD estimates correlate about 0.9, and significant sign flips are rare. Key takeaways: The worry. In staggered rollouts, two-way fixed effects (TWFE) can put negative weights on some treated observations, so a policy that helps everyone could in theory look harmful.The test. AI agents (OpenAI Codex) screened 16,094 records from ten economics journals, with the author overseeing. TWFE was compared with DiD-averaging (Callaway and Sant'Anna) and imputation (Borusyak, Jaravel and Spiess).Results. Correlations are 0.88 and 0.93. Signs flip in 14.5% and 10.8% of pairs, but only 3.5% and 2.2% flip with a significant difference. Negative weights carry a median 1.4% of TWFE's weight; modern standard errors are 47% and 14% larger.Noise removed. A hierarchical model gives 0.91–0.98 estimand correlations and a 2.1% and 0.6% chance of a significant wrong-signed TWFE estimate. More always-treated units mean more flips.Our caveats. The paper doesn't say these rates justify skipping modern estimators, and parallel-trends failures aren't tested.Chapters: 00:00 Cold open: can DiD flip the sign? 00:55 The paper: Peter Hull's meta-analysis 02:03 Staggered adoption and negative weights 02:49 Analogy: a staggered software rollout 07:01 AI agents build the replication sample 09:44 Results: correlations and sign flips 11:51 Why flips are rare: tiny negative weights 13:02 The precision tradeoff 14:18 What predicts a flip 16:04 Hierarchical model: estimates vs. estimands 18:05 What's new and related work 21:59 Recap and closing question Sources: How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis (Peter Hull (Brown University); arXiv, Oct 29, 2026)Difference-in-Differences with Multiple Time Periods (Callaway and Sant'Anna; J. Econometrics 2021)Revisiting Event Study Designs: Robust and Efficient Estimation (Borusyak, Jaravel, Spiess; REStud 2024)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Brown University — Is Difference-in-Differences Actually Broken?

  6. Oct 28

    iFood — Predicting the Discount That Tips a Customer

    LCM-Promo (Prosus and iFood) turns a 4B-parameter LLM into an uplift model for discounts. In a one-month A/B test on churned users it lifted conversion 1% and cut promotion cost 6% vs. the old policy. Key takeaways: Discounts as tokens. Each prompt holds session context (day, lunch or dinner shift), an LCM-generated user summary, and discount options mapped to letters. The model predicts one letter, or a no-purchase token, read as the smallest discount that would make the user order.Monotonic by construction. Summing those probabilities up the discount ladder gives a response curve that never falls as the discount grows, and the no-purchase mass keeps it below 100%.Online result. 90% control on Segment-EMA, 10% on LCM-Promo, same budget and same allocator: +1% relative conversion and 6% lower promotion cost. Allocation grew from about 50 segments to over 1,000 clusters and from 10 wallet configurations to over a million combinations; it now serves all churn users.Offline check. Against an uplift random forest (causalml) on RCT data, LCM-Promo was comparable on AUUC (0.0608 vs 0.0578) and Qini (0.0490 vs 0.0466) and lower on Uplift@30 (0.1626 vs 0.1708). The waterfall-label variant did much worse (AUUC 0.0278).Our caveats. Results cover one lifecycle stage on one platform. Our take: uplift margins may be thinner for high-intent daily users, and precomputed user summaries can't react to mid-session changes.Chapters: 00:00 Cold open: the pizza and the 10% coupon 01:00 The paper: LCM-Promo from Prosus and iFood 02:29 Headline: one-month churn-reactivation A/B test 03:21 Large Commerce Models and the 4B base 04:49 Discounts as letters, one-token prediction 05:50 Conversion thresholds and monotonic curves 07:21 Offline test vs. uplift random forest 08:46 The waterfall labels that failed 10:08 Online A/B test design and results 11:16 From 50 segments to 1,000+ clusters 12:30 Serving: vLLM, BESH, 8 H200s 13:31 What's genuinely new 14:29 Our caveats 15:56 Recap and closing question Sources: LCM-Promo: An LLM-Based Uplift Model for Promotion Allocation (Midavaine, Fok, Mendes, Silva, Akaçık, Lima, Genç (Prosus, iFood); arXiv, Oct 27, 2026; ECML PKDD 2026)CausalML: Python package for causal machine learning (Chen et al.; arXiv 2020)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    iFood — Predicting the Discount That Tips a Customer

  7. Oct 27

    Recommender Systems — How User Activity Creates Phantom Item Similarity

    Phantom correlation (Favier et al., University of Antwerp): user activity alone makes items look similar. On MovieLens-1M, an activity-only model removes 81% of the independence model's co-occurrence mismatch. Key takeaways: The toy example. 100 equally liked films; half the users watch each film with probability 0.01, half with 0.20, all independently. Lift still says every pair co-occurs about 1.82 times more often than chance.The model. Three assumptions (independent consumption, more activity raises every item, activity is the only difference) yield exactly one model, the Rasch model: activity multiplies every item's odds by the same factor, so every pair is positively associated.Results. Over all pairs, activity removes 81% (MovieLens-1M), 73% (MovieLens-20M) and 50% (Amazon Books) of the independence model's systematic mismatch, and 96–99% for lift, cosine and Pearson over popularity blocks on MovieLens. Last.fm, capped at each user's top 50 artists, shows none.Measure choice matters. Cosine inherits a baseline that grows with popularity, a head start for popular items; lift and PMI inherit one that is largest for rare items.Limits and our take. The authors don't call collaborative filtering fake: taste lives in which pairs stand out above the baseline. On Amazon Books, heavy readers drift to niche books, a second dimension the model leaves out. Our take: check item-item scores against an activity baseline before reading them as taste.Chapters: 00:00 Cold open: footprints that look like company 01:35 The paper: Favier et al., University of Antwerp 02:29 Item-item similarity and implicit feedback 03:59 Toy example: 100 films, lift 1.82, no taste 05:18 Four similarity measures, EASE, independence 06:21 The activity model: three conditions, Rasch 08:36 Four datasets and Poisson deviance 09:11 Results: MovieLens-1M and 20M 10:32 Amazon Books and capped Last.fm 12:16 What's new and related work 14:54 Our take, and the authors' caveats 17:05 Recap and closing question Sources: Do Similarity Measures Measure Similarity? How User Activity Creates Phantom Correlation in Recommender Systems (Favier, Daniëls, Leysen, Marinus, Goethals; arXiv, Oct 26, 2026)Embarrassingly Shallow Autoencoders for Sparse Data (EASE) (Steck; WWW 2019)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Recommender Systems — How User Activity Creates Phantom Item Similarity

  8. Oct 26

    Time-Series Foundation Models — Keeping Forecast Intervals Honest When Volatility Shifts

    Volatility-Corrected Prediction Intervals (Cho & Matteson, Cornell): time-series foundation models can hit 80% coverage on average yet miss when volatility rises. A GARCH correction on their residuals steadies local coverage without widening intervals. Key takeaways: The failure mode. Marginal coverage can sit at 80% while local coverage swings between over- and undercoverage. The paper calls this local scale miscalibration and measures it with RMSCE over rolling 100-forecast windows.How VCPI works. It standardizes the residuals by the forecaster's own predictive scale, fits a GARCH-family model to them (variant and error distribution chosen on a validation segment), and applies a multiplicative correction to future intervals.Results (80% intervals, horizons 1–4). Chronos on Capital Bikeshare: RMSCE 0.051 vs 0.100 (ACI), 0.096 (QT) and 0.079 (native). TimesFM on Colorado River streamflow: 0.062 vs 0.187 (QT) and 0.192 (ACI), with a median width of 761 vs 1,411 (QT) and 872 (native).Why it reacts faster. ACI and QT update on whether the last interval missed; GARCH uses the size of the error. The paper proves that after a large volatility jump, the first GARCH update lands closer to the oracle interval than quantile tracking.Limits and our take. Misspecified models carry bias, multi-step GARCH forecasts mean-revert beyond 4–8 steps, and 95% tails are more sensitive. Slight overcoverage is a fair price for stable local coverage.Chapters: 00:00 Cold open: a forecast blind to its own uncertainty 00:59 The paper: Cho and Matteson, Cornell 02:07 Local scale miscalibration vs marginal coverage 04:21 Foundation models: Chronos, Moirai, TimesFM 05:08 RMSCE: measuring local coverage 06:40 How VCPI works in four steps 07:42 Which GARCH model, chosen on validation data 09:02 Baselines and four datasets 10:09 Results: Chronos on bike rentals 11:32 Results: TimesFM on Colorado River 13:20 Error size vs a binary miss 15:08 Econometrics meets conformal prediction 16:30 Limitations, our take and recap Sources: Volatility-Corrected Prediction Intervals (Cho, Matteson, Cornell; arXiv, Oct 22, 2026)Conformal PID Control for Time Series Prediction (Angelopoulos et al.; NeurIPS 2023)Adaptive Conformal Inference Under Distribution Shift (Gibbs, Candès; 2021)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Time-Series Foundation Models — Keeping Forecast Intervals Honest When Volatility Shifts
See All (53)

About

How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact. Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It's data science in the wild: not textbook theory, but the models and decisions real companies run in production. New episodes several times a week, about 15–20 minutes each, so they fit a commute. Hosted by Peter Liu, a data scientist with 10+ years in consumer tech (Meta, Webflow, Adobe, eBay). Episodes are produced with AI-generated voices from curated source articles and papers, which are linked in each episode's notes.

Information