This episode puts Jeremy Pickens, Maura Grossman, and Robert Keeling in the same room to dig into a new study comparing generative AI review against continuous active learning, better known as TAR 2.0. Robert walks through the numbers his team at Redgrave published: 88% recall for generative AI against 64% for CAL, along with the precision tradeoffs that come with it. Maura pushes back on the framing and reworks the comparison herself, arguing the study raises more questions than it answers once you account for what happens when documents still need human eyes before they go out the door. Jeremy layers in his own read of the data and lands on a simple truth that keeps showing up throughout the conversation: proper sampling matters more than which tool you pick. We also get into whether prompts count as protected work product, how much the skill of the person writing the prompt actually shapes the results, and what it would take for the industry to adopt a real evaluation standard. And yes, rescue dog number four makes his debut. Key Takeaways: • The headline numbers, 88% recall for generative AI versus 64% for TAR, don't tell the whole story on their own. Once Maura standardized the methods to compare apples to apples, generative AI performed competitively rather than as a clear winner, and the right tool depended heavily on whether the goal was producing without human review or reviewing every document before it went out the door. • Sampling methodology is the real foundation everything else rests on. Jeremy made the point that if you sample correctly, the tool itself almost stops mattering. Robert's study did the sampling right, but both Jeremy and Maura agreed that's still far from the industry norm. • Prompt writing is turning into its own skill set. Robert's expert prompt writer, who is also a software engineer, produced results a typical generalist attorney couldn't match. Maura connected this directly to her research on whether sophisticated prompts start crossing into protected work product territory. • The reviewer's underlying job shapes the outcome more than people expect. Jeremy's small study found that an attorney doing internal investigations optimized for recall, while an attorney focused on document production optimized for precision, even without anyone planning the comparison that way. • Robert flagged a real limitation in his own data: the recall calculation for generative AI didn't include the human review or privilege review hours that a full production workflow requires, which means actual cost comparisons need more than the raw numbers in the study. Chapter Markers: 00:00:00 - Introduction and guest panel: Jeremy Pickens, Maura Grossman, and Robert Keeling 00:03:48 - Robert introduces the generative AI review study and Redgrave's methodology 00:08:56 - The monkeys and cows analogy: why sampling correctly matters more than the tool 00:12:37 - The headline numbers: 88% recall for generative AI versus 64% for TAR 00:15:17 - Maura challenges the framing and reworks the comparison 00:19:14 - Standardizing the methods: apples-to-apples results 00:22:09 - Jeremy breaks down the cost tradeoffs of senior review versus contract review teams 00:28:08 - Robert responds and explains why his clients often skip human review at scale 00:32:46 - Does the prompt writer matter? The role of expertise in generative AI results 00:35:28 - Jeremy's study on how an attorney's day job shapes their prompting outcomes 00:39:35 - Are AI prompts protected work product? 00:45:31 - Sharing prompts with opposing counsel and the meet and confer process 00:48:12 - Closing thoughts: what still needs to be studied 00:52:09 - Lucky the rescue dog and closing remarks generative AI review, TAR 2.0, continuous active learning, technology assisted review, eDiscovery, document review, recall and precision, predictive coding, generative AI prompts, prompt engineering, work product privilege, legal technology, AI in litigation, contract attorney review, sampling methodology, review workflow, machine learning classifier, Redgrave study, Jeremy Pickens, Maura Grossman, Robert Keeling, ESI Bytes podcast, legal AI research, discovery cost efficiency, production review, complex litigation review, AI validation methods
