Ops I did it again by Out of Pocket

Alex Dou
  • 5.0 (5)
  • Management
  • Updated Biweekly

A limited series podcast by Out-of-Pocket, hosted by Alex Dou. Each episode is a post-mortem on a product or feature that didn't work the way it was supposed to. The goal: an encyclopedia of known failures so the next generation of builders can skip the mistakes we've already made and fail at something new.

  1. Provider Directory: I will destroy your roadmap with Misha Nasrollahzadeh + Grant Veldhuis

    May 15

    Provider Directory: I will destroy your roadmap with Misha Nasrollahzadeh + Grant Veldhuis

    Grant Veldhuis (software engineer at Thatch, former Ribbon Health) and Misha Nasrollahzadeh (co-founder and CEO of Joyce Health, formerly Castlight Health and Ribbon Health) have both spent significant stretches of their careers trying to solve this. They walk through what they've learned, where they've hit walls, and what they'd do differently. What we cover - A working definition of Provider Directory: provider identity, location, specialty, phone number, and network status. Everything else: subspecialty inference, cost data, scheduling, is a layer on top. Don’t get it twisted - Then you layer things on top of that working definition: NPPES (National Plan and Provider Enumeration System) gives you specialty. Claims data lets you infer subspecialty. CPT codes have to be translated into human language before care navigation is actually possible. Scheduling (Zocdoc's territory) is hideously complex and you don’t want that smoke - Ways to constrain the problem (and *maybe* justify building in-house): Castlight maintained separate employer-by-employer directories built from claims data. Grant built a localized directory for the University of Michigan's 60,000-person campus. Both found success through scope restriction. Nationwide coverage is usually the enemy of accuracy. - Be suspicious: Someone is going to tell you their data is 99% accurate. Maybe a vendor. Maybe an internal team lead. You need to dig into what that number actually means. One payer's head of provider directory once defended that figure but his definition was *whether or not the fields were populated*. Not whether the phone number actually worked or whether the address was still a functioning practice (The address turned out to be a car wash) - Sharing is NOT caring: why CAQH (Council for Affordable Quality Healthcare) exists but payers don't have a great incentive to improve the underlying data, even when it benefits everyone. - Oblique, non-obvious signals for keeping data fresh: referral coordinators at a value-based primary care group were the best signal Ribbon ever found, because they were calling offices every day and had direct incentive to log corrections. Published research papers are another: "they just published with this institution, so they must still be affiliated." - Network effects and why they're hard: small care navigation companies can be required to contribute edits in exchange for data access. National payers paying orders of magnitude more cannot be asked the same thing. - Build vs. buy, the age-old question: default to buying unless your use case is genuinely outside what vendors have mapped. ACA (Affordable Care Act) individual market plans are one example where standard data quality may not hold. University of Michigan campus resources are another. Brought to you by Basata AI - Basata builds AI agents for specialty practices. They answer phones 24/7, schedule appointments, process referrals and faxes, and take repetitive work off healthcare teams so they can focus on patients. For inquiries about sponsoring the podcast, email sales@outofpocket.health

    1h 9m
  2. Predicting the Future (risk stratification) with Shay Sayed

    Apr 29

    Predicting the Future (risk stratification) with Shay Sayed

    What we cover Risk stratification is ranking patients by probability of an adverse outcome. Traditional indices like the Charlson Comorbidity Index use clinician-designed scoring systems; ML-based approaches automate feature generation and let the model surface correlations that a heuristic would miss. The tradeoff is interpretability: with tens of thousands of computations per prediction, explaining a ranking to a clinician of why it ranked Patient A ahead of Patient B is ... surprisingly hard to do. It's kind of like asking a human being "why did you do this thing X weeks ago"... memory fades, and us humans have this "delightful" post-hoc rationalization feature Actually getting the data in a good shape is harder than training the model. Schema differences between organizations are structural: different table names, different column types, different ways of representing the same event. ML tolerates directional imperfection in a way that population analytics does not, but the cleanup is still slow and dependent on tribal knowledge that data owners often can't fully explain. "Feature engineering" in this context means building hypotheses the model can test. An example we discussed was “if I’m trying to risk stratify kidney stones, what hypotheses might I try? Maybe soda intake. Maybe dehydration. Maybe SDOH." Those three things are all “features” in this context. The platform ClosedLoop built could generate complex clinical features in about ten minutes, which was most of the competitive advantage. Failure modes tend to be around operations, not accuracy of the algorithm. Buyers without a clear care management strategy can't actually impact patients on the list. ROI attribution takes years, by the which case people might revert to the mean. And without tracking what the clinical program is actually doing, you can't separate a model problem from a workflow problem ETHOS is Epic's transformer trained on serialized clinical event histories from 300 million patients. The way I think about this is if LLMs “predict the next word most likely to occur”, then ostensibly you could get a training set of healthcare events and “predict the next {event} most likely to occur” where {event} is NICU stay Brought to you by Toboggan Labs: A consultancy for healthcare builders. If you have a health product that needs engineers, product people, or experienced operators to help you build or fix something, go talk to them at https://bit.ly/oop-readmission For inquiries about sponsoring the podcast, email sales@outofpocket.health Find Shay https://www.linkedin.com/in/shaayaan-sayed-8097b1100/ Timestamps [02:07] Shay's background: training models from scratch at Closed Loop [04:22] How Shay got into ML in high school by cold-emailing every professor in Houston. By contrast, Alex really got into Dynasty Warriors in high school [10:43] The CMS (Centers for Medicare & Medicaid Services) AI Health Outcomes Challenge. ClosedLoop won $1 million against some big names: Mayo Clinic, Geisinger, and Mathematica. The two components: predictive performance across 13 to 15 adverse outcomes, and interpretability for clinical teams [16:00] A layperson’s definition of risk stratification: a ranked patient list by probability of an adverse outcome. The Charlson Comorbidity Index as a standard example, and why ML outperforms it once you need more than one outcome. [29:27] The data layer you need. Claims, EHR (Electronic Health Record) dumps, SDOH (Social Determinants of Health) feeds, ADT (Admission, Discharge, Transfer) data. This is hard because everybody has different schema: payer one's data looks nothing like payer two's, and the data “owner” often can't explain their own tables. [41:50] Feature engineering: building hypotheses the model can test. The difference between "feature" as a PM uses the word and "feature" as a data scientist uses it. [47:52] Interpretability: being able to tell a human being why a patient ranked where they did. Two structural issues: incomplete data and unknown causal frameworks [54:14] Failure modes: Buyers without a care management strategy. Reversion to the mean within two years and you don’t know whether you made a difference. Not knowing where to cut the list (Patient number 50 vs 51?). And a related issue: missing data on what the clinical program is actually doing, which makes it impossible to separate a bad model from a bad workflow [01:09:39] Whether anyone should still learn traditional ML, or just LLMs. Shay's answer: gradient boosted trees and transformers are on a spectrum so it’s kind of a false dichotomy. Then: the ETHOS paper from Epic, a transformer trained on 300 million patient records that enables one model for many outcomes and counterfactual inference. And what Shay is watching next: robotics and the last-mile problem. AI can identify a list of people with fall risk but something or someone still has to act on it

    1h 28m
  3. $400 CAC, $100 MSRP: why it’s so hard to build hardware, with Erynn Petersen

    Apr 15

    $400 CAC, $100 MSRP: why it’s so hard to build hardware, with Erynn Petersen

    What we cover Your hardware margin is grocery-store thin. Founders coming from software expect 80% margins. Hardware is more like 5%. Grocery stores run at 3%. Getting to even nominally profitable meant Emme flipping its model entirely: give the case as a loss leader, sell the app. CAC went from ~$400 to ~$25 Civil engineers take an oath, why not software engineers? Civil engineers are personally liable if a bridge falls. In healthcare/healthtech, we often make ethical choices without even realizing it. There are actually a lot of duties that we have to our patients, our local community, our world, that aren’t always obvious. Selling data is a tantalizing revenue driver to boost your razor-thin margins: how do you tie yourself to the mast to avoid that call? How are we thinking about LLMs and the non-zero resources that each prompt takes?  Ooo: more pink taxes. Women's health is a restricted category on most major ad platforms. Reproductive health is doubly so. Shadow bans are common, policies change without notice, and the cost of reaching your customer is structurally higher. It is unfair, yes, but if you are building in this space, you should know it exists If you get software investors for your hardware company, you're gonna have a bad time. Look for experienced hardware investors. They understand the margin profile and the multi-year payback curve. They're also one of the better early validators for whether an idea is worth building at all. Taking checks from people expecting software returns, then managing that conversation at year three, is an all-too-common failure mode Firmware versioning was a decade-long problem. Now it isn't. Managing versions across device serial numbers, OS versions, and phone generations used to require a support matrix that could kill a small team. AI has quietly made this tractable: feed your codebase and version history to your AI engine of choice, and it maps the dependencies Brought to you by Toboggan Labs A consultancy for healthcare builders. If you have a health product that needs engineers, product people, or experienced operators to help you build or fix something, go talk to them at https://bit.ly/oop-readmission For inquiries about sponsoring the podcast, email sales@outofpocket.health Find Erynn Emme: emme.com LinkedIn Timestamps 0:00 – Intro 1:20 – Pleasantries (Friday the 13th, alligator moons, and a rectangular February) 6:04 – Resilience and ethics in healthtech: the biggest system failures come from within, civil engineers take an oath, and Emme's answer to the data-selling temptation 15:08 – The economics of healthcare hardware: 5% margins if you're lucky, the path through payers is getting harder, and why consumer-grade industrial design is creeping into med tech 18:20 – Why Rhode Island is an underrated place to build: defense, healthcare, research, and a manufacturing base small enough to navigate quickly. Also: return to office (said it) 25:05 – The hard economics and margins of hardware: 5% if you're lucky, before marketing. Grocery stores run at 3%. 29:04 – Ad: Toboggan Labs – if you're selling a physical health product DTC and a third of your kits never come back completed, go find them at tobogganlabs.com 31:34 – Advertising in women's reproductive health is hard on hard mode: shadow bans, shifting platform policies, and the SEO-to-GEO transition hitting just as Emme had figured out content strategy. Find hardware-aligned investors before you take a check from someone who expects software returns. 39:34 – Ad: Data Camp – June 25-26 in Boston. Tactical sessions, hands-on practitioners, great swag, and a secret plan for a data wreck room involving a paper-mache EDI file 41:39 – Firmware versioning: a decade-long nightmare, why AI has made it tractable, and whether it’s worth draining a lake in South America faster to answer which states had the most sex during the Super Bowl

    1h 4m
  4. Ignore brokers at your own risk. Antagonize them at your ruin. Build a health plan with Nick Soman

    Apr 1

    Ignore brokers at your own risk. Antagonize them at your ruin. Build a health plan with Nick Soman

    What we cover Start with brokers, not against them. The typical "tech disruption" approach frames brokers as rent-seeking middlemen.  Nick's counterargument:  Brokers fill a trust role that no website can replicate They touch roughly 70% of small-to-mid-size business health plan sales They will pick up the phone (how many millennials and Gen Zers can say that? Certainly not I) Brokers will also tell your product team what's missing. All of those "oH, MemBeRS woN’T bOOk USer IntErvIeWs wITh mE" Product Managers should probably be talking to brokers If you don't have rate relief, you have nothing. Rate relief is the one question brokers are being asked by their clients, day in and day out. If you can't show that your plan is cheaper for at least 40% of groups, the conversation doesn't start. Nick's heuristic: if the broker doesn't have price, they have nothing to sell, no matter how elegant the technology. Therefore: focus on steerage. The difference between the cheapest and most expensive MRI in Austin, TX is roughly 8x. If you can build a plan that guides people to the lower-cost option for non-emergency shoppable services, you've found the actual lever. Nick's starter heuristic: for all non-emergency hospital claims, do you have an alternative good spot to send members to next time?  Then simplify until it hurts. Decent spent four years before it could credibly sell the Zero Plan: “if you do what the plan wants you to do, it costs you $0 outside your monthly contribution.” That is the level of simplicity you need to actually break through to the decision maker at your prospects. The HR person who has to make this decision once a year and really doesn't want to be in this conversation needs a message that lands in one sentence. This also helps brokers sell (remember rule number one). Brought to you by Stedi - Stedi is the modern healthcare clearinghouse. They’re the new way to verify insurance, submit claims, and track claim payments. http://stedi.com/demo For inquiries about sponsoring the podcast, email sales@outofpocket.health Find Nick nsoman@nice.healthcare nick@decent.com Nice Healthcare: https://www.nice.healthcare/ Decent: decent.com Exciting upcoming events in the OOP world We have a Hackathon in SF from Apr 17-19. We’re full on Hackers, but you should definitely come to Demo Day and see what people built. I built an app for people to lend Oura Sleep/Readiness Points to each other… at 10% Daily Compounding. I’m calling it “Oura Farming”. I guarantee you there will be 10x better ideas than that at Demo Day Timestamps 00:00 Intro: Why Tech People Fail in Healthcare (With Nick Soman, Former CEO of Decent) 09:47 Tip #1 — If You Can't Sell to Brokers, You Have Nothing 19:17 Reframe Brokers as Channel Partners, Not Middlemen 29:06 Make It Simpler. Then Simpler Again. (The Story of the Zero Plan) 34:53 The Secret Weapon: Steerage (And Why an 8x Price Spread Changes Everything) 43:56 Narrow Networks, Broker Trust, and Building Something You'd Put Your Own Family On

    54 min
  5. PROs, UX/UI Design for trials, and the nocebo effect | Paul Wicks, PhD

    Mar 18

    PROs, UX/UI Design for trials, and the nocebo effect | Paul Wicks, PhD

    The episode moves through three connected ideas. First: clinical trials have a UX problem. One in five trials recruits zero patients. Half under-recruit. Paul's early work was understanding why – and fixing it. The answer was almost never the science. It was that patients had no parking, caregivers had no wifi (or babysitting!), and the protocol assumed everyone could get to Mass General by 9 AM on a Friday via public transit. Solving that required showing trial designs to actual patients before locking them, and listening to their recorded reactions rather than just summarizing their survey scores. Second: measuring patient experience is genuinely hard. There are no inches of insomnia. No pounds of pain. Patient Reported Outcomes – PROs – exist because the most burdensome chronic conditions don't have actual units that can be measured. There are thousands of PROs, many are outdated (the fibromyalgia questionnaire from the early 2000s asks how well you can vacuum and cook for your family), and most were designed for clinical trials, not for weekly check-ins on a mobile screen. So for you, Product Manager/Engineer/Builder: how are you going to measure whether your Care Model actually improves patient health? Third: building PROs into a product creates specific traps. Paul runs through the ones he has seen firsthand: copyright violations (most scales are licensed and litigation can cost millions), engagement drop-off (5–10% of users is a good day for consistent tracking), incentive fraud (cash rewards attract bot farms), the nocebo effect (asking about pain can worsen it: hey, 1-10 scale, how itchy are you right now?), and the regulatory ceiling – go too far with your tracker and you've crossed into medical device territory, with all the compliance that comes with it. The Five Things to Know Before You Build 1. You probably don't have permission to use that questionnaire. Most validated scales are owned by universities and licensed for a fee. Saying you found it published online doesn't mean it's free. The database PROQOLID lists rights holders, and some of them are quite litigious  2. Only about 5–10% of patients will track at the cadence you're planning. The ones who do are not representative of your broader user base. Obsessive trackers skew your data. Users with executive dysfunction may not be able to log in, let alone complete a questionnaire. 3. Cash incentives destroy your data. If you offer a financial reward for completion, expect an avalanche of bots and fraudulent responses. Survey researchers report 80–90% fraud rates in incentivized studies. Things that seem to work better: setting group goals, an AMA with the scientists behind the study, and giving users a summary of what their data revealed. 4. Asking about symptoms can worsen them. The nocebo effect is real (for example: scale of 1-10, how itchy do you feel right now?). Paul's team designed positively-framed questions ("how well are you sleeping?") rather than deficit-focused ones ("how bad is your insomnia?") specifically to reduce iatrogenic harm. 5. Interpreting a score triggers regulation. Tracking is fine. Saying that a score of 5/5 means "severe" COULD make your app a medical device and suddenly make you beholden to a lot more regulation Brought to you by Nirvana - Nirvana delivers real-time eligibility checks and cost estimates and plugs right into your existing workflows For inquiries about sponsoring the podcast, email sales@outofpocket.health Resources Paul Wicks on LinkedIn — linkedin.com/in/paulwicks Paul’s newsletter, ProofPoints Paul’s podcast, Prove It! Paul’s website, ProofStack Health RAAPID Inc course on Risk Adjustment: V28, AI, and multi-million dollar settlements. Apr 7-9 Sponsor the Hardware Hackathon! Apr 17-19 in SF! PROQOLID — eprovide.mapi-trust.org — find the rights holder for any validated PRO scale PHQ-9: free, validated, widely used depression screener GAD-7: free, validated anxiety scale Timestamps 00:00 — Paul's origin story: ALS research, patient communities, and clinical trial ethics 03:00 — PatientsLikeMe and showing trial protocols to patients before locking them can actually INCREASE recruitment 11:00 — What PROs (Patient Reported Outcomes) are, why there are so many, and why none of them agree 18:00 — Pharma, payers, providers, patients – and who actually cares about PRO data 26:00 — The fifth stakeholder: scale developers. Beware for license fees and lawsuit risk 30:00 — Which patients actually track data, and what to do about everyone else 33:00 — False starts: psoriasis body maps, crab-to-clam scales, and positive framing 38:00 — The regulatory ceiling and pharmacovigilance

    45 min
  6. How Photon Used Automations to Change E-Prescriptions | Michael Rado, CPO, Co-Founder, Photon Health

    03/28/2024

    How Photon Used Automations to Change E-Prescriptions | Michael Rado, CPO, Co-Founder, Photon Health

    In this episode of Ops I Did It Again, Michael Rado (aka Rado), Co-Founder and CPO at Photon Health joins the thinksquad aka Danielle and Nikhil to break down how they built and scaled products to support over 20,000 e-prescriptions a month. Rado shows a live demo of their build (on YouTube version) and candidly shares his learnings on how to scale with optimization and automation in mind.  This episode is sponsored by Out of Pocket, because no one is prouder than us than us: https://www.outofpocket.health/ To register for the upcoming Healthcare Call Center 101 crash course course visit: https://www.outofpocket.health/courses/how-to-build-a-healthcare-call-center; Use code: ANSWERS for $100 off; Next cohort starts 4/16- 5/2 To register for the upcoming Healthcare 101 crash course course visit: https://www.outofpocket.health/courses/healthcare-101-crash-course; Use code: IBELIEVEINME for $100 off; Next cohort starts 4/23-5/9 Hosts: Danielle Poreh (https://www.linkedin.com/in/danielleporeh/) Nikhil Krishnan (https://twitter.com/nikillinit) Guests: Michael Rado (https://www.linkedin.com/in/michaelrado)    Timestamps: (00:00) Intro (01:45) Michael Rado and his journey to Photon (03:14) How Photon Health works (07:36) Courses by Out Of Pocket! (16:49) What photon automates and measures (24:23) The approach towards product development at Photon Health  (29:25) Practical experiments for orgs to try

    34 min
  7. Digital Health Ops: A Playbook for Every Growth Stage | Rahul Agarwal, COO Medplum

    03/18/2024

    Digital Health Ops: A Playbook for Every Growth Stage | Rahul Agarwal, COO Medplum

    On this episode of Ops I Did it Again, Rahul Agarwal (COO of Medplum) joins Danielle to share his digital health operations playbook. The “hero’s journey” playbook breaks down pro tips and gotchas throughout every stage of scale: from pre-seed all the way to scaling to 50 states. If you are building a clinical workforce, scaling your digital health operations or looking into the future on how you should be building, this episode will give you several tactical tips to implement. This episode is sponsored by Out of Pocket, because no one is prouder than us than us: https://www.outofpocket.health/ To register for the upcoming Healthcare Call Center 101 crash course course visit: https://www.outofpocket.health/courses/how-to-build-a-healthcare-call-center; Use code: ANSWERS for $100 off; Next cohort starts 4/16- 5/2 To register for the upcoming Healthcare 101 crash course course visit: https://www.outofpocket.health/courses/healthcare-101-crash-course; Use code: IBELIEVEINME for $100 off; Next cohort starts 4/23-5/9 Hosts: Danielle Poreh (https://www.linkedin.com/in/danielleporeh/) Nikhil Krishnan (twitter: https://twitter.com/nikillinit) Guests: Rahul Agarwal (https://www.linkedin.com/in/rahul-agarwal-330a979/)  Timestamps: (00:00) Intro (02:23) Rahul explains his playbook (02:58)Tips for early stage operators (06:32) Leaving the jungle: finding product-market fit (08:55) Operational playbook for scaling digital health services (13:45)  Courses by Out Of Pocket! (15:20) How to define  encounters (24:10) How to develop care plans/clinical pathways (30:59) Optimizing provider recruiting and workload with EHR design (33:30) Standardize your metrics and avoid metric soup (35:50) Dos and don’ts when scaling to 50 states (39:43) Practical experiments for orgs to try

    43 min
  8. Building Differentiated Patient Experiences | Kerem Ozkay, COO and Ayo Omojola CPO, Carbon Health

    02/29/2024

    Building Differentiated Patient Experiences | Kerem Ozkay, COO and Ayo Omojola CPO, Carbon Health

    In this episode of Ops I Did It Again, Kerem Ozkay (Chief Operating Officer) and Ayo Omojola (Chief Product Officer) from Carbon Health join the thinksquad aka Danielle and Nikhil, to discuss how they designed and deployed product solutions. They breakdown real examples of AI tools to enhance operations, strategies for patient care, and unique top of funnel marketing approaches.  Later in the episode, Ayo shares a behind the scenes view at Carbon’s home grown EHR - it’s best viewed on YouTube so you can all the magic. This episode is sponsored by Out of Pocket, because no one is prouder than us than us: https://www.outofpocket.health/ To register for the upcoming Healthcare Call Center 101 crash course course visit: https://www.outofpocket.health/courses/how-to-build-a-healthcare-call-center; Use code: ANSWERS for $100 off; Next cohort starts 4/16- 5/2 To register for the upcoming Healthcare 101 crash course course visit: https://www.outofpocket.health/courses/healthcare-101-crash-course; Use code: IBELIEVEINME for $100 off; Next cohort starts 4/23-5/9 Hosts: Nikhil Krishnan (twitter: https://twitter.com/nikillinit) Danielle Poreh (https://www.linkedin.com/in/danielleporeh/) Guests: Kerem Ozkay (https://www.linkedin.com/in/keremozkay/) Ayo Omojola (https://www.linkedin.com/in/omojola/)   TIMESTAMPS (00:00) Introduction (01:43) 4 levers for tech-enabled healthcare (03:03) Running patient acquisition, operations, and marketing (09:34) SEO strategy and AI integration (15:36) Courses by Out Of Pocket! (17:11) The experience of scheduling (28:04) Performance metrics and feedback loops (29:54) Being 2x better at one thing vs 10x better at everything (33:53) Understanding patient acquisition and call content (36:04) The best tech for clinic reports (37:49) Convincing doctors to adopt new tools (41:02) Importance of localized patient acquisition (41:31) Moving off of slack for field based team (43:32) Automating Revenue Cycle Management (RCM) (50:44) Charting and patient care with AI (live demo) (55:05) Future of AI in healthcare (01:03:26) Kerem and Ayo’s team dynamics (01:05:38) Closing thoughts

    1h 7m
See All (15)

Trailer

5
out of 5
5 Ratings

About

A limited series podcast by Out-of-Pocket, hosted by Alex Dou. Each episode is a post-mortem on a product or feature that didn't work the way it was supposed to. The goal: an encyclopedia of known failures so the next generation of builders can skip the mistakes we've already made and fail at something new.

Information

You Might Also Like