Training Data Sequoia Capital

Archimedes said that with a large enough lever, you can move the world. For decades, software engineering has been that lever. And now, AI is compounding that lever. How will we use AI to apply 100 or 1000x leverage to the greatest lever to move the world?

Matan Grinberg and Eno Reyes, co-founders of Factory, have chosen to do things differently than many of their peers in this white-hot space. They sell a fleet of “Droids,” purpose-built dev agents which accomplish different tasks in the software development lifecycle (like code review, testing, pull requests or writing code). Rather than training their own foundation model, their approach is to build something useful for engineering orgs today on top of the rapidly improving models, aligning with the developer and evolving with them. 

Matan and Eno are optimistic about the effects of autonomy in software development and on building a company in the application layer. Their advice to founders, “The only way you can win is by executing faster and being more obsessed.”

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned: 


Juan Maldacena, Institute for Advanced Study, string theorist that Matan cold called as an undergrad


SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, small-model open-source software engineering agent


SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, an evaluation framework for GitHub issues


Monte Carlo tree search, a 2006 algorithm for solving decision making in games (and used in AlphaGo)


Language agent tree search, a framework for LLM planning, acting and reasoning


The Bitter Lesson, Rich Sutton’s essay on scaling in search and learning 


Code churn, time to merge, cycle time, metrics Factory thinks are important to eng orgs


Transcript: https://www.sequoiacap.com/podcast/training-data-factory/

00:00 Introduction
01:36 Personal backgrounds
10:54 The compound lever
12:41 What is Factory? 
16:29 Cognitive architectures 
21:13 800 engineers at OpenAI are working on my margins 
24:00 Jeff Dean doesn't understand your code base
25:40 Individual dev productivity vs system-wide optimization 
30:04 Results: Factory in action 
32:54 Learnings along the way 
35:36 Fully autonomous Jeff Deans
37:56 Beacons of the upcoming age
40:04 How far are we? 
43:02 Competition 
45:32 Lightning round
49:34 Bonus round: Factory's SWE-bench results

Last year, AutoGPT and Baby AGI captured our imaginations—agents quickly became the buzzword of the day…and then things went quiet. AutoGPT and Baby AGI may have marked a peak in the hype cycle, but this year has seen a wave of agentic breakouts on the product side, from Klarna’s customer support AI to Cognition’s Devin, etc.

Harrison Chase of LangChain is focused on enabling the orchestration layer for agents. In this conversation, he explains what’s changed that’s allowing agents to improve performance and find traction. 

Harrison shares what he’s optimistic about, where he sees promise for agents vs. what he thinks will be trained into models themselves, and discusses novel kinds of UX that he imagines might transform how we experience agents in the future.     

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital

Mentioned: 


ReAct: Synergizing Reasoning and Acting in Language Models, the first cognitive architecture for agents


SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, small-model open-source software engineering agent from researchers at Princeton


Devin, autonomous software engineering from Cognition


V0: Generative UI agent from Vercel


GPT Researcher, a research agent 


Language Model Cascades: 2022 paper by Google Brain and now OpenAI researcher David Dohan that was influential for Harrison in developing LangChain


Transcript: https://www.sequoiacap.com/podcast/training-data-harrison-chase/

00:00 Introduction
01:21 What are agents? 
05:00 What is LangChain’s role in the agent ecosystem?
11:13 What is a cognitive architecture? 
13:20 Is bespoke and hard coded the way the world is going, or a stop gap?
18:48 Focus on what makes your beer taste better
20:37 So what? 
22:20 Where are agents getting traction?
25:35 Reflection, chain of thought, other techniques?
30:42 UX can influence the effectiveness of the architecture
35:30 What’s out of scope?
38:04 Fine tuning vs prompting?
42:17 Existing observability tools for LLMs vs needing a new architecture/approach
45:38 Lightning round

Join us as we train our neural nets on the theme of the century: AI. Sequoia Capital partners Sonya Huang and Pat Grady host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies and their implications for technology, business and society.

The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.