Ellen Brandenberger @ Stack Overflow

Guest:

Ellen Brandenburger – Product leader and coach; former head of product at Chegg Skills and Stack Overflow’s data licensing team.

What we cover in this episode:

• How Ellen joined Stack Overflow just two weeks before ChatGPT launched, reshaping the company’s future overnight
• The creation of Overflow AI: a team tasked with exploring “what’s just now possible” for developers
• Four iterations of conversational search:
• V1: a chat UI on top of keyword search
• V2: semantic search to handle natural questions
• V3: fallback to GPT-4 for gaps in Stack Overflow’s corpus
• V4: adding RAG for attribution and transparency
• Why attribution and transparency were critical for developer trust
• How the team used simple spreadsheets and subject-matter experts to evaluate answer accuracy, relevance, and completeness
• Why Stack decided to sunset conversational search despite heavy investment—what they learned and why it wasn’t wasted
• The pivot to data licensing: how Stack Overflow leveraged its 14M+ Q&A corpus to power LLM training and benchmarks
• Building industry benchmarks with subject-matter experts to prove Stack data improved LLM accuracy and relevance

Key lessons:

• Take one bite of the apple at a time—prototype, learn, iterate
• Product in the AI era means managing probabilities, not certainties

Links & References:

• Ellen Brandenburger on LinkedIn
• The Changing State of the Internet and Related Business Models
• ProLLM: LLM benchmarks for real-world use-cases