Data Science in the Wild

Peter Liu
  • 0.0 (0)
  • Technology

How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact. Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It's data science in the wild: not textbook theory, but the models and decisions real companies run in production. New episodes several times a week, about 15–20 minutes each, so they fit a commute. Hosted by Peter Liu, a data scientist with 10+ years in consumer tech (Meta, Webflow, Adobe, eBay). Episodes are produced with AI-generated voices from curated source articles and papers, which are linked in each episode's notes.

  1. 8h ago

    Time-Series Foundation Models — Keeping Forecast Intervals Honest When Volatility Shifts

    Volatility-Corrected Prediction Intervals (Cho & Matteson, Cornell): time-series foundation models can hit 80% coverage on average yet miss when volatility rises. A GARCH correction on their residuals steadies local coverage without widening intervals. Key takeaways: The failure mode. Marginal coverage can sit at 80% while local coverage swings between over- and undercoverage. The paper calls this local scale miscalibration and measures it with RMSCE over rolling 100-forecast windows.How VCPI works. It standardizes the residuals by the forecaster's own predictive scale, fits a GARCH-family model to them (variant and error distribution chosen on a validation segment), and applies a multiplicative correction to future intervals.Results (80% intervals, horizons 1–4). Chronos on Capital Bikeshare: RMSCE 0.051 vs 0.100 (ACI), 0.096 (QT) and 0.079 (native). TimesFM on Colorado River streamflow: 0.062 vs 0.187 (QT) and 0.192 (ACI), with a median width of 761 vs 1,411 (QT) and 872 (native).Why it reacts faster. ACI and QT update on whether the last interval missed; GARCH uses the size of the error. The paper proves that after a large volatility jump, the first GARCH update lands closer to the oracle interval than quantile tracking.Limits and our take. Misspecified models carry bias, multi-step GARCH forecasts mean-revert beyond 4–8 steps, and 95% tails are more sensitive. Slight overcoverage is a fair price for stable local coverage.Chapters: 00:00 Cold open: a forecast blind to its own uncertainty 00:59 The paper: Cho and Matteson, Cornell 02:07 Local scale miscalibration vs marginal coverage 04:21 Foundation models: Chronos, Moirai, TimesFM 05:08 RMSCE: measuring local coverage 06:40 How VCPI works in four steps 07:42 Which GARCH model, chosen on validation data 09:02 Baselines and four datasets 10:09 Results: Chronos on bike rentals 11:32 Results: TimesFM on Colorado River 13:20 Error size vs a binary miss 15:08 Econometrics meets conformal prediction 16:30 Limitations, our take and recap Sources: Volatility-Corrected Prediction Intervals (Cho, Matteson, Cornell; arXiv, Oct 22, 2026)Conformal PID Control for Time Series Prediction (Angelopoulos et al.; NeurIPS 2023)Adaptive Conformal Inference Under Distribution Shift (Gibbs, Candès; 2021)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Time-Series Foundation Models — Keeping Forecast Intervals Honest When Volatility Shifts

  2. 3d ago

    AI Analysts — Keeping Type I Error Under Control as They Evolve

    Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng & Wang, Purdue): every prompt edit or model swap gives an AI analyst another chance at a false positive. Continual validation tests each update on simulated null data before it ships. Key takeaways: The risk accumulates. Over 20 runs of 10 updates proposed by Claude Sonnet 5.5, the worst deployed version without validation reached a Type I error of 0.098 (bioequivalence) and 0.183 (A/B tests), against a 0.05 target.Continual validation. Each frozen update is tested on fresh data simulated under the null before deployment. With α = 0.05, δ = 0.005 and α′ = 0.0452, the chance of ever deploying a version above α′ in a tested null scenario is at most δ. Worst deployed error fell to at most 0.045 and 0.044.Off-grid check. Bioequivalence versions stayed at or below 0.05 in at least 666 of 690 off-grid scenarios (max 0.058); A/B versions stayed at or below 0.044 in all 116.Modest cost. With fixed, code-computed analysis paths, validation needs no LLM calls (Prop. 2). Median one-GPU time: 1,101 s for ~447M simulated studies, 329 s for ~29M A/B experiments.Power. Validated analysts kept 82% and 96% of an oracle's power. Absolute power was 0.55 vs 0.67 (bioequivalence) and 0.20 vs 0.21 for A/B features with 1–3% relative lifts. Some power is given up.Our take. The guarantee covers only the simulated null scenarios, so the work shifts to keeping those simulations realistic.Chapters: 00:00 Cold open: an AI analyst that rubber-stamps 01:28 Why an evolving analyst accumulates Type I error 02:52 Without validation: 0.098 and 0.183 04:17 Continual validation and null-simulated data 05:56 Alpha, delta and alpha prime 06:40 Results with validation 07:13 Off-grid scenarios 08:21 Cost: no LLM calls, minutes on one GPU 11:14 Power vs an oracle 13:41 Classical tests vs lifetime guarantees 14:35 Our take: simulations must match reality 15:44 Recap and closing question Sources: Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng, Wang, Purdue; arXiv, Oct 22, 2026)Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control (Angelopoulos et al.; arXiv, 2021)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    AI Analysts — Keeping Type I Error Under Control as They Evolve

  3. 4d ago

    Bayesian LLM Agents — Reasoning Does the Math, Code Handles the Rest

    From Prompts to Posteriors (Filbry, Melo & Gal, Oxford): LLMs break Bayes' rule without reasoning, but reasoning models compute simple posteriors almost exactly, and with code even intractable ones. Key takeaways: Three tests. Dutch-book coherence (Gemini-2.5-Pro, Gemma-4-31B-it, Qwen-2.5-32B-Instruct), then posterior fidelity (KL to a Bayesian oracle) and experimental design (posterior-mass gap) on 20 Questions and the new ProteinWorld, with five open models.Coherence depends on the chat. In separate chats Gemini gave 50% to a Democrat winning in 2028 and 55% to a non-Democrat; violations top 50% of scenarios for almost all model and axiom pairs. Within one dialogue they fall to 10% for Gemini and 1 of 30 for Gemma.No reasoning, no Bayes. Without reasoning, the mass gap on the true answer exceeds 30% for every model in both environments; strong models often pick the right answer but with the wrong probability.Reasoning, then code. With reasoning alone, KL is near 0 on closed-form ProteinWorld and Gemma-4-31B-it matches the oracle, doing Bayes in its reasoning, not Python. On intractable ProteinWorld-hard, Gemini-2.5-Pro faked simulation output; with Python it wrote a correct nested Monte Carlo estimator and nearly matched the oracle.Our take. A plain reasoning prompt matched the BED-LLM harness on 20Q, so strict harnesses may no longer be needed. Caveats: close, not perfect; 20Q still falls short of the oracle; 40 runs per environment.Chapters: 00:00 Cold open: the paper and the question 01:20 Bayesian inference, detective-style 02:16 The old assumption and its harnesses 02:58 Coherence: 50% vs 55% in separate chats 06:00 Updating without reasoning 07:38 Reasoning, no tools: KL near 0 10:06 ProteinWorld-hard: intractable posteriors 11:47 Code execution: nested Monte Carlo 13:21 Related work and auditing 15:10 Caveats: close, not perfect 15:58 Recap and closing question Sources: From Prompts to Posteriors: Can LLM Agents Perform Bayesian Inference in Context? (Filbry, Melo, Gal, Oxford; arXiv, Oct 21, 2026)BED-LLM (Choudhury et al.; ICLR 2026)BayesBench (Samanta et al.; arXiv, 2026)LLMs are not (consistently) Bayesian (Chen et al.; arXiv, 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Bayesian LLM Agents — Reasoning Does the Math, Code Handles the Rest

  4. 5d ago

    Instagram Feed — Keeping the Baseline in Ranking

    Instagram Feed's ranker stops deleting "bias": Meta keeps the raw engagement prediction and adds its debiased, incremental part as a second, separately weighted term. Key takeaways: Baseline vs incremental. Each value-model prediction mixes a baseline (a serial liker, a popular creator, a strong batch) with the candidate's own excess. Most prior work removes the baseline; Meta keeps the raw term and weights the incremental term separately (Eq. 1–2). No causal claim.One operator, three axes. Viewer, item and request debiasing are one operator (Eq. 3; per-term forms in Table 4). The request baseline is the p75 or mean of predictions over the request's own candidates, not time of day or device.Six four-arm live ablations (none/raw/incremental/both, 3% of traffic per arm, one week). Both beats each single term on point estimates in 11 of 13 pre-assigned (term, metric) pairs, 7 with both contrasts significant; 17 of 26 contrasts significantly for, 1 against (impressions).Beyond Feed. Separate Reels launches (Table 6): sessions +0.029% to +0.083%, DAU +0.026% to +0.042%.Our caveats. A Meta-authored case series of six terms; production weights unpublished; single-term arms not re-tuned; small session effects. Offline diagnostics should screen signals, not rank them.Chapters: 00:00 Cold open: why your feed shows what it does 00:58 The paper: Baseline and Incremental 01:56 The value model's two hidden parts 04:10 Removing the baseline vs keeping it 06:30 Two terms and one operator (Eq. 1–3) 07:58 Request context = the live candidate pool 10:26 The four-arm live ablation 11:58 Reels launches: sessions and DAU 13:15 Judged pairs and multiplicity 15:00 Results: 11 of 13, 17 of 26, one loss 16:11 Novelty and our caveats 17:54 Offline diagnostics: screen, don't rank 18:53 Related work: testing in the wild 20:16 Recap and closing question Sources: Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking (Chen, Wang, Zhou et al., Meta; arXiv, Oct 20, 2026)Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations (Tannenbaum; arXiv, 2026; our S2E9)Don't Rewrite the SID Map: Topology Preservation for Generative Retrieval in Dynamic E-Commerce Search (Zhu et al., Alibaba; arXiv, 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Instagram Feed — Keeping the Baseline in Ranking

  5. 6d ago

    Selling in the Wild — A Randomized Trial of an AI Sales Agent

    Selling in the Wild: in a 14.8M-visitor field trial, the Javo AI sales agent lifted revenue per visitor 10% and held its own against human chat staff at 1.8% of the cost. Key takeaways: Four live arms. 100 sites (e-commerce, travel, high-consideration services incl. B2B logistics), May–Sep 2026: no agent 2.6M, Javo 5.6M, frontier agent 5.7M, human chat 0.9M. Both bots disclosed they were AI. Javo (Qwen3-32B tuned on 4.6M consented sales chats; 2,240 H100 GPU-hours) raised revenue per visitor +10.0% (95% CI +5.1 to +14.9) on 85 transactional sites.Javo vs GPT-5.2, same widget. Engaged conversion +21.6% (+16.9 to +26.3); revenue per visitor +16.9%. Identical launcher, greeting and AI line, and the visitor writes first, so the model can't change who engages: 8.07% vs 8.06% (p = 0.536).Javo vs typical human associates (20 staffed sites): engaged conversion 11.26% vs 11.20% (+0.5%, −5.0 to +6.0), non-inferior (margin −7%; −10% in the design), not superior. $0.06 vs $3.40 per engaged conversation. Widgets differed (Javo proactive; 8.07% engaged overall vs 5.80% for humans); trimming bounds are planned but not reported.Guardrails held over five months: unsupported claims 0.15% of turns (frontier 0.56%), no unauthorized offers; returns, complaints in margin.Our caveats. The sole author is Founder/CEO of AISO Boost Ltd., which operates Javo; no independent review, registration or ethics approval is claimed. Sites came partly from his client network; one frozen GPT-5.2 snapshot.Chapters: 00:00 Cold open: wind tunnel vs rush hour 01:48 The paper: Selling in the Wild 02:51 Javo: 4.6M sales chats, 2,240 GPU-hours 04:19 14.8M visitors and four studies 06:28 Revenue lift and intention-to-treat 08:23 Skeptic check: who chooses to chat? 10:14 Javo vs humans: conversion and cost 11:48 Human widgets and trimming bounds 13:21 Guardrails and novelty 14:53 Related work 17:00 Caveats: conflict of interest 19:04 Recap and closing question Sources: Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations (Benjamin Tannenbaum; arXiv, Oct 18, 2026)Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches (Alcain et al.; arXiv, 2026)Measuring Gift Card Program Incrementality via Causal Data Fusion (Whitehouse et al.; arXiv, 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Selling in the Wild — A Randomized Trial of an AI Sales Agent

  6. Oct 19

    Conformal Policy Selection — Safety Gates for Individual AI Decisions

    Conformal Policy Selection (Cao, Gao, Saar-Tsechansky & Yin) gates AI decisions person by person—zero harmful EHR treatments deployed for q ≤ 0.85. Key takeaways: Per unit, not on average. Each candidate is a test: will the outcome under the AI policy clear a threshold? Weighted conformal p-values correct for the behavior-to-target policy shift; a Benjamini–Hochberg-style rule plus randomized pruning bounds the FDR (the expected share of false selections, not a per-person guarantee).Doubly robust, within limits. Correct propensity weights give finite-sample FDR ≤ q; a correct outcome model with wrong weights gives asymptotic control. In linear synthetic tests FDR held when either model was right; with both wrong it reached 0.29 at a 0.15 target.EHR safety gate. On real covariates (5,993 patient-years, age 60+, BMI 30+) with simulated outcomes, treatment harms nearly everyone, yet a standard risk rule treats the top-risk decile. The gate deployed none of those treatments for q ≤ 0.85 (FDR 0.046 at q = 0.75). Semi-synthetic: no claims about real patients.Our caveat: power. Safety can mean few selections. The paper reports ~0.80 power at q = 0.95 (synthetic) but 0.012 at q = 0.15 on LaLonde (≈1.9 of ~185 selected).Chapters: 00:00 Cold open: the 80/20 drug 01:25 Who should get the AI decision? 03:17 A personalized guess vs a guarantee 04:28 Weighted conformal p-values 06:50 BH-style selection, pruning and FDR 08:44 Doubly robust: two models, two chances 09:55 Synthetic tests: where control breaks 11:50 EHR study: the deployment safety gate 14:02 Novelty and related work 16:13 Our caveat: statistical power 18:21 Recap and closing question Sources: Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention (Cao, Gao, Saar-Tsechansky & Yin; arXiv, Oct 16, 2026)Conformal Prediction Under Covariate Shift (Tibshirani, Barber, Candès & Ramdas; NeurIPS 2019)Covariance-Calibrated Prediction-Powered Inference (Chen, Bai & Zhu; arXiv, 2026)Probabilistic Forecasting of Electric Vehicle Charging in a Time-varying Hierarchical Infrastructure (Principato et al.; arXiv, 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Conformal Policy Selection — Safety Gates for Individual AI Decisions

  7. Oct 16

    Causal Forests — Treatment Effects on Zero-Heavy Outcomes

    ZIC-BCF-Smear (Souto, Wang & Shuryak) pairs two Bayesian causal forests for zero-heavy outcomes—finding panitumumab adds ~10 severe-toxicity days per patient. Key takeaways: Two forests, one effect. A Probit BCF on everyone models clearing zero; a Gaussian BCF on log intensity uses active units only. Duan's smearing (mean of exponentiated residuals) replaces the fragile exp(σ²/2) back-transform.SPA, precisely. It re-estimates the treatment propensity within the active subset (not the odds of being active) and feeds it to the prognostic forest to mitigate regularization-induced confounding.Simulations. Across log-normal, Gamma and Tweedie data (3–85% zeros, 100 seeds) it matches well-specified parametric hurdles; on nonlinear surfaces it has the lowest CATE RMSE and no divergences, while Gamma models diverge and DPglm over-covers. Mild under-coverage (89.7–91.7%) remains.SPECTRUM trial. In 520 head-and-neck cancer patients (23.7% zeros), panitumumab + chemo adds 10.0 distinct severe-toxicity days per patient (95% CrI 3.6–16.6), mostly via how many days, not whether any; that margin is uncertain.Our caveats. Prep was abstract-level. Treatment time differed (83.7 vs 130.3 days); at a common day 84–120 the gap is ~3–4 days. The Etsy paper we contrast is from Oct 2024 (CLeaR 2026).Chapters: 00:00 Cold open: weighing a ghost 01:37 Semicontinuous outcomes: a spike at zero 02:49 Why today's options fail 05:33 ZIC-BCF-Smear: two decoupled forests 07:35 Selection bias and SPA 10:38 Duan's smearing: back to days and dollars 13:06 Simulations: stable where others diverge 14:13 SPECTRUM trial: ~10 more severe-toxicity days 16:41 Lessons for product experiments 17:16 Novelty (3/5) and related work 20:52 Caveats: abstract-only, treatment duration 23:16 Recap and closing question Sources: Bayesian Causal Forests for Zero-Inflated Continuous Outcomes (Souto, Wang & Shuryak; arXiv, Oct 8, 2026)Bayesian regression tree models for causal inference (Hahn, Murray & Carvalho; original BCF, 2017)Variance reduction combining pre-experiment and in-experiment data (Lin & Crespo, Etsy; 2024)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    Causal Forests — Treatment Effects on Zero-Heavy Outcomes

  8. Oct 15

    WeChat — Generative Retrieval for Industrial Search

    WeChat's WeGR turns e-commerce search into generative retrieval: CoCo-VAE builds collaborative, collapse-free Semantic IDs, and DHPO trains with hierarchical partial credit—lifting orders +2.539% while cutting CPU/GPU ~25%. Key takeaways: Generative retrieval, not a full stack teardown. WeGR generates Semantic IDs for candidates, then still feeds the industrial cascade (candidate generation → pre-ranking → ranking). Deployed V1.2 removes 80% of other-channel candidates—not the channels or pre-ranking themselves.CoCo-VAE + DHPO. CoCo-VAE injects Q2I/I2I collaborative signals and fights codebook collapse via refresh/regularization. DHPO expands reward and optimization horizons over SID prefixes, giving partial credit without auxiliary reward models.Online wins with less compute. Two-month A/B (June 22–Aug 4, 2026) vs the incumbent: orders +2.539%, CTR +1.508%, CVR +1.381%; CPU −25.5%, GPU −26.1%; ranking load −38.10%; latency −21.7 ms (Channels) / −15.4 ms (Search).Novelty (hosts' 4/5). SID generative retrieval (e.g. TIGER) is established; the hosts flag DHPO's sparse-reward fix at WeChat scale as the sharper advance versus tearing down the cascade (contrast: OneTrans-V2-style unification).Our caveats. Round "2.5%" once spoken as orders is exactly +2.539%. A two-month window supports near-term gains, not permanent habit change. Stratified Tables 5–7 and daily HR@200 live in the paper body; the abstract alone is thin on breakdowns/CIs.Chapters: 00:00 Cold open: ask the haystack for the needle 01:12 Mission: WeChat WeGR for e-commerce search 01:32 Why multi-channel retrieval is expensive 03:39 Two bottlenecks: SIDs and sparse RL 05:05 Codebook collapse, explained 06:28 CoCo-VAE: collaborative, collapse-free IDs 11:04 DHPO: partial credit without reward models 13:43 V1.2: keep the cascade, cut 80% of candidates 16:31 A/B results: +2.539% orders, −25% compute 17:44 Novelty vs TIGER and OneTrans-V2 19:12 Caveats: stability, drift, abstract vs body 21:47 Recap and generative-fulfillment teaser Sources: WeGR: Unified Generative Retrieval for Industrial Search Systems in WeChat (Han, Ma, Felix, Xu, Yu, Guo, Wang, Ye, Hua, Xie, Yang & Wu; arXiv, Oct 12, 2026)Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify. Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups. Made with AI-generated voices; curated by Peter Liu from the sources above.

    WeChat — Generative Retrieval for Industrial Search
See All (46)

About

How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact. Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It's data science in the wild: not textbook theory, but the models and decisions real companies run in production. New episodes several times a week, about 15–20 minutes each, so they fit a commute. Hosted by Peter Liu, a data scientist with 10+ years in consumer tech (Meta, Webflow, Adobe, eBay). Episodes are produced with AI-generated voices from curated source articles and papers, which are linked in each episode's notes.

Information