Jonathan Evens is a product lead at Google DeepMind, where he has spent more than a decade applying machine learning and AI across industries — from the smart grid at AutoGrid, to detecting roads and buildings from satellite imagery at Planet, to recommender systems, Google Search's AI Overviews and AI Mode, and now live avatars. He is also an advisor to the Evens Foundation, where he is building a "digital citizenry": a democracy sandbox that uses synthetic citizens to pre-test how the public might react to a policy before it is written. Jonathan returns to The Product Experience, where hosts Lily Smith and Randy Silver pick up the conversation they started at MTPcon London, to dig further into what actually separates an AI product manager from a product manager who simply uses AI tools, why product principles have to come before evaluations, and how synthetic users can help — and mislead — at very different scales of product. We discuss: 1. Why "AI product manager" has become a near-meaningless label, and the two distinct roles hiding underneath it: the modelling product manager working on core model capabilities, and the AI feature product manager building AI-powered products 2. Why using an LLM as a thinking partner or a coding assistant does not make someone an AI product manager — it makes them a product manager using AI tools, full stop 3. How Google Search's North Star metrics have stayed constant even as the proxy metrics beneath them — side-by-side win rates, user ratings, RLHF signals — have had to be rebuilt from scratch 4. Why product principles, not evaluations, are the real starting point for any AI feature, and how Google Search resolved the problem of trustworthy sources disagreeing on basic facts 5. How Google Search builds trust into its AI Overviews through sourcing, citation placement and UX cues such as highlighting, so users can judge at a glance what to verify 6. Where synthetic users genuinely help — cold-start problems, privacy-sensitive research, automated regression testing — and where they fall short 7. Building the Evens Foundation's "digital citizenry", and the core technical problem behind it: AI-generated personas that are less diverse and more extreme than real people 8. How team size and structure differ between a fully resourced lab like Google DeepMind and a resource-constrained non-profit team, and why Jonathan resists a single answer for the "right" team size 9. How the product manager's job is shifting as engineers absorb more of the evaluation work themselves through prompting and iteration 10. Jonathan's advice for product managers building AI features, and his case for following the Makers Manifesto Key takeaways "AI product manager" covers two distinct jobs. The modelling product manager defines and measures a model's core capabilities — factuality, reasoning, long context — and that role is concentrated almost entirely inside frontier labs. The AI feature product manager builds a product or feature on top of an existing model, and needs domain expertise and user empathy far more than technical depth. Conflating the two is why the title has become so diluted.Using an LLM to think faster or write code faster does not make someone an AI product manager. It makes them a product manager using AI as part of their toolkit — the same as any other knowledge worker. The distinction matters because it clarifies what skills are actually being tested.Product principles have to come before evaluations, not after. Before Jonathan starts building an eval set for a new product, he first asks what the product is meant to feel like and what values it should encode. Google Search's response to sources disagreeing on a monument's construction date, or to large language models hallucinating at scale, came from principles about trustworthiness established before any metric was built.Trust in an AI feature is built through sourcing and interface design as much as through the model itself. Google Search's AI Overviews are constrained to draw only from ranked, trustworthy documents rather than the model's own memory, and users are given UX signals — citation placement, highlighting — that let them judge at a glance how much to verify.Synthetic users add genuine value in cold-start scenarios, privacy-sensitive research and automated regression testing. Where they fall short is diversity: AI-generated personas tend to be less varied and more extreme than real people, which is the central technical problem behind the Evens Foundation's digital citizenry project.There is no fixed answer to the right team size. Jonathan sees a gradient, from a senior developer working entirely alone, up to the Evens Foundation's single product manager with AI-assisted development skills, up to a fully staffed Google team — with the deciding factor being how unsolved the underlying problem is, not company size.As engineers absorb more evaluation work themselves through prompting and iteration, roles are blending. What still sits with product management is the judgement calls that follow from product principles — deciding, for example, which technical trade-offs actually matter to the use case, rather than which are easiest to measure.Features links - Evens Foundation — https://evensfoundation.eu - Makers Manifesto — https://makersmanifesto.org - Google DeepMind — https://deepmind.google - AutoGrid — smart grid AI company where Jonathan began applying machine learning to industry - Planet — satellite imagery company where Jonathan worked on automated road and building detection We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! Our Hosts Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.
