Andrea Malagodi is CTO at Sonar, which builds AI code verification and governance tools. When finance asked about the size of his team's AI bills, he calculated cost per PR. At one extreme was an engineer with more than 500 PRs in a short period and a low cost per PR. At the other was an engineer with very high spend and very few PRs. Full transcript and chapters: https://chainofthought.show/podcast/76-your-best-ai-engineer-might-have-the-worst-metrics-sonar-cto-andrea-malagodi/ Judged on cost alone, the first engineer wins and the second looks like a problem. Malagodi looked closer. The first had built a personal agent factory, with specification, design review, documentation records, coding, verification and functional testing, and about two-thirds of the output was tests and validations. The second was working on a hard problem that needed the AI to reason through long, multi-turn sessions and didn't reduce to a line count. Numbers alone tell you something, but not which engineer was doing the more valuable work. We cover: Why cost per PR is becoming the industry's standard AI spend metric, and how Malagodi's 500-PR engineer and high-spend engineer broke itThe history of bad proxies, from lines of code to story points to token leaderboards, and why Malagodi says the people who removed code were often the most importantSonar's guide, verify, solve loop: Vortex for codebase context, algorithmic plus LLM-based PR analysis, a remediation agent for legacy debt, and the Hunter agent for security vulnerabilitiesWhy Malagodi breaks agent work into small pieces, and why a five-day session that produces 400,000 lines of code leaves you with no idea what is in your codebaseSetting up disagreement between agents with an orchestrator and personas for engineering, product management and quality, plus Conor Bronsdon's cross-model-family review setupRolling AI tools out at enterprise scale: one cost view for finance, up to 30% savings from cutting repeated context reads, and why budgets as small as $50 a week are hard to work withRotating a triage duty to protect deep work, including how Sonar's teams built skills to sort through hundreds of reported CVEs, a number of them likely false positivesNew episodes, the ideas behind them and Conor's essays land in the Chain of Thought newsletter first. Subscribe: https://newsletter.chainofthought.show/ Chapters:(0:00) Human reviewers rubber-stamp big AI changes(2:43) Learning the craft alongside AI tools(9:09) From lines of code to PR counts(12:17) Spreading top engineers' setups org-wide(16:50) The guide, verify, solve agent loop(22:51) Specialized agent lanes and multitasking limits(26:51) Clear asks and guardrails for agent coordination(30:29) Building disagreement between agents(34:59) Mixing models instead of picking one(39:46) Rolling out AI tooling at enterprise scale(46:19) Defense in depth for AI-written code(50:52) Protecting time for incident investigations(54:08) Personalized software for teams and individuals Links from the episode: Sonar Vortex research: https://www.sonarsource.com/blog/cut-your-coding-agents-cost-with-sonar-semantic-code-navigation/SonarQube Hunter Agent: https://www.sonarsource.com/products/sonarqube/hunter-agent/Connect with Andrea Malagodi: LinkedIn: https://www.linkedin.com/in/malagodia/Sonar: https://www.sonar.com/Connect with Chain of Thought host Conor Bronsdon: Newsletter: https://newsletter.chainofthought.show/Twitter/X: https://x.com/ConorBronsdonLinkedIn: https://www.linkedin.com/in/conorbronsdon/YouTube: https://www.youtube.com/@ConorBronsdonMore episodes: https://chainofthought.show Thanks to Svix, presenting sponsor of season four of Chain of Thought. Svix delivers billions of reliable webhooks for startups and the Fortune 500. Get started at https://link.svix.com/cot. Qualified startups get $12,000 in credits, and YC companies get $50,000. Thanks to Walrus, presenting sponsor of season four of Chain of Thought. Walrus Memory gives AI agents portable, verifiable memory that carries context across apps, sessions, and other agents. Get started at https://walrus.xyz/cot Thanks to Inngest, presenting sponsor of season four of Chain of Thought. Agents in production run long - they call models and wait on APIs and people. But the longer agents run, the more they break. Inngest handles that with durable execution. You build your agent as steps in TypeScript, Python, or Go. When a step fails, Inngest retries it with exponential backoff, and completed steps are saved and skipped. Try it out: https://inngest.link/cot-pod Thanks to G2i for sponsoring this episode - for over a decade, they vetted and placed engineers at other companies, from startups to FAANG. Two years ago, they turned that same judgment inward, building their own bench to review RL environments, evals, and training data that models are trained on. Get access: https://fandf.co/3SFxVm6
