Why are experienced PMs getting stuck in PM interviews they would have cleared two years ago? Because the interviews have changed. The old rounds are still there. Product sense, metrics, estimation and behavioural. But many PM interviews now add an AI layer on top. That layer does not ask you what RAG is or what an agent is. It asks what you would do when one breaks. That is the gap I talk about in the video above. Many PMs have learned AI. Far fewer have built with it. The new questions are written to tell the two apart. Below are the four types of AI questions we see most. For each one, I show what the question looks like, what it takes to pass, and why building helps. 1. Take-home assignments Some companies now send a take-home assignment. Here is a real one for a Product Manager, Platform Experience role. The problem was customer questions at scale. * You get 4 to 6 hours. If you reach hour seven, you stop and send what you have. * You build a small working prototype of the first thing you would build. Any tool is allowed, including AI tools. A rough script over the sample data, a clickable page or a working demo all count. * You write a PDF of up to 4 pages or 8 slides. It covers how you framed the problem, what you chose to build and not build, and how you would know it is working. * You have five days to send it. * Then there is a 45-minute call where they push back on your choices. The brief says they judge what the prototype shows, not how it is engineered. It also says a prototype that answers one question well and correctly refuses one it shouldn’t beats a polished mock of everything. To pass, you need to do five things. * Pick one narrow problem and make it work on the sample data. * Make it refuse questions it cannot answer, instead of making something up. * Write down what you left out, and why. * Say how you would measure it, with a few test questions and one or two numbers. * Defend every choice in the follow-up call. This is where building helps. If you have built AI products before, you know how much fits in 6 hours, so you keep the scope small. You know an AI tool will answer anything with confidence, so you add the refusal on purpose. Someone who has only learned AI often spends the 6 hours on a polished mock. The brief tells you that it loses. 2. RAG questions RAG is when a chatbot first fetches the right pieces of your documents and then answers from them. It is the AI topic we see asked the most. It rarely comes as a question about what RAG is. It comes like this. Chunking Our chatbot answers from a 200-page policy document. The answers are often half right. What would you change? To pass, talk about how the document is cut into pieces, called chunks. Small chunks lose context. Big chunks bring in text that has nothing to do with the question. Split by headings and sections, not by a fixed length. Keep tables in one piece. Let chunks overlap a little, so a sentence cut at the edge still appears whole in one of them. Reranking The right chunk is found, but it comes back eighth, and the model only sees the top three. What do you do? To pass, explain reranking. First, fetch more chunks, say 30, with fast search. Then a second model scores each chunk against the question again, and only the best 3 to 5 go to the model. Also mention the cost. Every answer gets a little slower. Root cause Our support chatbot started giving wrong answers last week. Find out why. To pass, split the problem into two. Take 20 wrong answers and check if the right chunk was fetched. If it was not, the problem is retrieval, so look at chunking, search and reranking. If it was fetched and the answer is still wrong, the problem is generation, so look at the prompt and the model. Then ask what changed last week. It is usually the documents, the prompt or the model. Coverage You have a Harry Potter book, and Snape is mentioned 400 to 500 times. Sometimes by name, sometimes just as he, sometimes as the Half-Blood Prince. How does your chatbot use every mention when it answers a question about him? To pass, first spot the problem. Normal search fetches only the top few chunks, and it misses the chunks where Snape is never named. Then fix the index. Work out which character every pronoun and nickname points to, and tag each chunk with the characters in it. Add a short line of context to each chunk before storing it. For questions about the whole book, answer from chapter summaries instead of hundreds of chunks. This is where building helps. When you build RAG, you watch a table get cut in half, and the answer go wrong. You see the right chunk come back eighth. You change the chunk size and see the answers change. In the interview, you talk about what you saw, not a definition. 3. AI agent questions An AI agent is a model that decides its own next step and uses tools, like search or email, to finish a task. Agent questions look like this. Agent or workflow Would you build this as an agent or as a fixed workflow? To pass, say that a fixed workflow is better when the steps are known in advance. It is cheaper, faster and easier to test. Use an agent only when the next step depends on what the user says or what a tool returns. Good answers start with the simpler option. Handoffs between agents Your system passes a task from one agent to the next. How do you know the handoffs are working? To pass, name the metrics. What share of tasks finish end to end. At which step tasks fail the most. Whether the next agent got everything it needed. How many tasks end up with a human. Time and cost per task. Stuck agents Your agent keeps calling the same tool again and again. What do you do? To pass, start with the trace, which is the step-by-step log of what the agent did. Then set a step limit and a clear stop condition. Write better tool descriptions, so the agent knows when each tool is the right one. Human approval What should the agent never do without asking a person? To pass, point to actions that cannot be undone. Refunds, payments, emails to customers and deleting data. Those wait for a human to approve. This is where building helps. Once you build an agent, you see it loop, pick the wrong tool, or pass half the information to the next step. You learn to read traces. That is exactly what these questions ask about. 4. AI product sense and metrics questions The classic product sense round now has an AI version. Should this be AI at all? Design an AI feature for a food delivery app. To pass, start with the user problem, not the model. Say when a simple rule would do the job. Then cover what AI adds, what each answer costs to run, and what the user sees when the AI is wrong. Launch decisions Your model is right 92% of the time. Do you launch? To pass, ask what the other 8% looks like. A bad song suggestion is fine. A wrong refund amount is not. Group the failures by type, decide what a wrong answer costs the user, and set the bar from that. After launch How would you know it is working? To pass, use two kinds of checks. Before launch, build a test set of real questions with the right answers marked, including questions it should refuse. After launch, watch how users react. How often they accept the answer, give a thumbs down, or ask for a human. This is where building helps. When you build, you write those test questions yourself. You watch a bot you thought was good fail more of them than you expected. After that, you stop treating one accuracy number as the answer. How to prepare for the new PM interviews Look at what every question above has in common. None of them asks for a definition. Each one asks what you would do when something breaks, and why. You can only answer that well if you have seen it break. That is why we built the AI Product Builder Cohort at Technomanagers around one objective. To make you job-ready to build AI products. * 12 weeks of real projects across RAG, AI agents, evals, prototyping and AI product strategy * 10+ projects you build yourself, with mentor feedback * One portfolio-ready AI product as your capstone * 20 hours of AI PM interview p… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.technomanagers.com
