Training Data

Sequoia Capital
  • 4,7 (24)
  • TECHNOLOGIES
  • CHAQUE SEMAINE
Training Data

Join us as we train our neural nets on the theme of the century: AI. Sonya Huang, Pat Grady and more Sequoia Capital partners host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies—and their implications for technology, business and society. The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.

  1. OpenAI Researcher Dan Roberts on What Physics Can Teach Us About AI

    -4 J

    OpenAI Researcher Dan Roberts on What Physics Can Teach Us About AI

    In recent years there’s been an influx of theoretical physicists into the leading AI labs. Do they have unique capabilities suited to studying large models or is it just herd behavior? To find out, we talked to our former AI Fellow (and now OpenAI researcher) Dan Roberts. Roberts, co-author of The Principles of Deep Learning Theory, is at the forefront of research that applies the tools of theoretical physics to another type of large complex system, deep neural networks. Dan believes that DLLs, and eventually LLMs, are interpretable in the same way a large collection of atoms is—at the system level. He also thinks that emphasis on scaling laws will balance with new ideas and architectures over time as scaling asymptotes economically. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital  Mentioned in this episode: The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks, by Daniel A. Roberts, Sho Yaida, Boris Hanin Black Holes and the Intelligence Explosion: Extreme scenarios of AI focus on what is logically possible rather than what is physically possible. What does physics have to say about AI risk? Yang-Mills & The Mass Gap: An unsolved Millennium Prize problem AI Math Olympiad: Dan is on the prize committee

    42 min
  2. Google NotebookLM’s Raiza Martin and Jason Spielman on Creating Delightful AI Podcast Hosts and the Potential for Source-Grounded AI

    15 OCT.

    Google NotebookLM’s Raiza Martin and Jason Spielman on Creating Delightful AI Podcast Hosts and the Potential for Source-Grounded AI

    NotebookLM from Google Labs has become the breakout viral AI product of the year. The feature that catapulted it to viral fame is Audio Overview, which generates eerily realistic two-host podcast audio from any input you upload—written doc, audio or video file, or even a PDF. But to describe NotebookLM as a “podcast generator” is to vastly undersell it. The real magic of the product is in offering multi-modal dimensions to explore your own content in new ways—with context that’s surprisingly additive. 200-page training manuals become synthesized into digestible chapters, turned into a 10-minute podcast—or both—and shared with the sales team, just to cite one example. Raiza Martin and Jason Speilman join us to discuss how the magic happens, and what’s next for source-grounded AI. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital

    32 min
  3. Snowflake CEO Sridhar Ramaswamy on Using Data to Create Simple, Reliable AI for Businesses

    8 OCT.

    Snowflake CEO Sridhar Ramaswamy on Using Data to Create Simple, Reliable AI for Businesses

    All of us as consumers have felt the magic of ChatGPT—but also the occasional errors and hallucinations that make off-the-shelf language models problematic for business use cases with no tolerance for errors. Case in point: A model deployed to help create a summary for this episode stated that Sridhar Ramaswamy previously led PyTorch at Meta. He did not. He spent years running Google’s ads business and now serves as CEO of Snowflake, which he describes as the data cloud for the AI era. Ramaswamy discusses how smart systems design helped Snowflake create reliable "talk-to-your-data" applications with over 90% accuracy, compared to around 45% for out-of-the-box solutions using off the shelf LLMs. He describes Snowflake's commitment to making reliable AI simple for their customers, turning complex software engineering projects into straightforward tasks.  Finally, he stresses that even as frontier models progress, there is significant value to be unlocked from current models by applying them more effectively across various domains. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital Mentioned in this episode:  Cortex Analyst: Snowflake’s talk-to-your-data API Document AI: Snowflake feature that extracts in structured information from documents

    59 min
  4. OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

    2 OCT.

    OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

    Combining LLMs with AlphaGo-style deep reinforcement learning has been a holy grail for many leading AI labs, and with o1 (aka Strawberry) we are seeing the most general merging of the two modes to date. o1 is admittedly better at math than essay writing, but it has already achieved SOTA on a number of math, coding and reasoning benchmarks. Deep RL legend and now OpenAI researcher Noam Brown and teammates Ilge Akkaya and Hunter Lightman discuss the ah-ha moments on the way to the release of o1, how it uses chains of thought and backtracking to think through problems, the discovery of strong test-time compute scaling laws and what to expect as the model gets better.  Hosted by: Sonya Huang and Pat Grady, Sequoia Capital  Mentioned in this episode: Learning to Reason with LLMs: Technical report accompanying the launch of OpenAI o1. Generator verifier gap: Concept Noam explains in terms of what kinds of problems benefit from more inference-time compute. Agent57: Outperforming the human Atari benchmark, 2020 paper where DeepMind demonstrated “the first deep reinforcement learning agent to obtain a score that is above the human baseline on all 57 Atari 2600 games.” Move 37: Pivotal move in AlphaGo’s second game against Lee Sedol where it made a move so surprising that Sedol thought it must be a mistake, and only later discovered he had lost the game to a superhuman move. IOI competition: OpenAI entered o1 into the International Olympiad in Informatics and received a Silver Medal. System 1, System 2: The thesis if Danial Khaneman’s pivotal book of behavioral economics, Thinking, Fast and Slow, that positied two distinct modes of thought, with System 1 being fast and instinctive and System 2 being slow and rational. AlphaZero: The predecessor to AlphaGo which learned a variety of games completely from scratch through self-play. Interestingly, self-play doesn’t seem to have a role in o1. Solving Rubik’s Cube with a robot hand: Early OpenAI robotics paper that Ilge Akkaya worked on. The Last Question: Science fiction story by Isaac Asimov with interesting parallels to scaling inference-time compute. Strawberry: Why? O1-mini: A smaller, more efficient version of 1 for applications that require reasoning without broad world knowledge. 00:00 - Introduction 01:33 - Conviction in o1 04:24 - How o1 works 05:04 - What is reasoning? 07:02 - Lessons from gameplay 09:14 - Generation vs verification 10:31 - What is surprising about o1 so far 11:37 - The trough of disillusionment 14:03 - Applying deep RL 14:45 - o1’s AlphaGo moment? 17:38 - A-ha moments 21:10 - Why is o1 good at STEM? 24:10 - Capabilities vs usefulness 25:29 - Defining AGI 26:13 - The importance of reasoning 28:39 - Chain of thought 30:41 - Implication of inference-time scaling laws 35:10 - Bottlenecks to scaling test-time compute 38:46 - Biggest misunderstanding about o1? 41:13 - o1-mini 42:15 - How should founders think about o1?

    45 min
  5. Why Vlad Tenev and Tudor Achim of Harmonic Think AI Is About to Change Math—and Why It Matters

    24 SEPT.

    Why Vlad Tenev and Tudor Achim of Harmonic Think AI Is About to Change Math—and Why It Matters

    Adding code to LLM training data is a known method of improving a model’s reasoning skills. But wouldn’t math, the basis of all reasoning, be even better? Up until recently, there just wasn’t enough usable data that describes mathematics to make this feasible. A few years ago, Vlad Tenev (also founder of Robinhood) and Tudor Achim noticed the rise of the community around an esoteric programming language called Lean that was gaining traction among mathematicians. The combination of that and the past decade’s rise of autoregressive models capable of fast, flexible learning made them think the time was now and they founded Harmonic. Their mission is both lofty—mathematical superintelligence—and imminently practical, verifying all safety-critical software. Hosted by: Sonya Huang and Pat Grady, Sequoia Capital  Mentioned in this episode: IMO and the Millennium Prize: Two significant global competitions Harmonic hopes to win (soon) Riemann hypothesis: One of the most difficult unsolved math conjectures (and a Millenium Prize problem) most recently in the sights of MIT mathematician Larry Guth Terry Tao: perhaps the greatest living mathematician and Vlad’s professor at UCLA Lean: an open source functional language for code verification launched by Leonardo de Moura when at Microsoft Research in 2013 that powers the Lean Theorem Prover mathlib: the largest math textbook in the world, all written in Lean Metaculus: online prediction platform that tracks and scores thousands of forecasters Minecraft Beaten in 20 Seconds: The video Vlad references as an analogy to AI math Navier-Stokes equations: another important Millenium Prize math problem. Vlad considers this more tractable that Riemann John von Neumann: Hungarian mathematician and polymath that made foundational contributions to computing, the Manhattan Project and game theory Gottfried Wilhelm Leibniz: co-inventor of calculus and (remarkably) creator of the “universal characteristic,” a system for reasoning through a language of symbols and calculations—anticipating Lean and Harmonic by 350 years! 00:00 - Introduction 01:42 - Math is reasoning 06:16 - Studying with the world's greatest living mathematician 10:18 - What does the math community think of AI math? 15:11 - Recursive self-improvement 18:31 - What is Lean? 21:05 - Why now? 22:46 - Synthetic data is the fuel for the model 27:29 - How fast will your model get better? 29:45 - Exploring the frontiers of human knowledge 34:11 - Lightning round

    40 min
  6. Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

    17 SEPT.

    Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

    AI researcher Jim Fan has had a charmed career. He was OpenAI’s first intern before he did his PhD at Stanford with “godmother of AI,” Fei-Fei Li. He graduated into a research scientist position at Nvidia and now leads its Embodied AI “GEAR” group. The lab’s current work spans foundation models for humanoid robots to agents for virtual worlds. Jim describes a three-pronged data strategy for robotics, combining internet-scale data, simulation data and real world robot data. He believes that in the next few years it will be possible to create a “foundation agent” that can generalize across skills, embodiments and realities—both physical and virtual. He also supports Jensen Huang’s idea that “Everything that moves will eventually be autonomous.” Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode: World of Bits: Early OpenAI project Jim worked on as an intern with Andrej Karpathy. Part of a bigger initiative called Universe Fei-Fei Li: Jim’s PhD advisor at Stanford who founded the ImageNet project in 2010 that revolutionized the field of visual recognition, led the Stanford Vision Lab and just launched her own AI startup, World Labs Project GR00T: Nvidia’s “moonshot effort” at a robotic foundation model, premiered at this year’s GTC Thinking Fast and Slow: Influential book by Daniel Kahneman that popularized some of his teaching from behavioral economics Jetson Orin chip: The dedicated series of edge computing chips Nvidia is developing to power Project GR00T Eureka: Project by Jim’s team that trained a five finger robot hand to do pen spinning MineDojo: A project Jim did when he first got to Nvidia that developed a platform for general purpose agents in the game of Minecraft. Won NeurIPS 2022 Outstanding Paper Award ADI: artificial dog intelligence Mamba: Selective State Space Models, an alternative architecture to Transformers that Jim is interested in (original paper here) 00:00 Introduction 01:35 Jim’s journey to embodied intelligence 04:53 The GEAR Group 07:32 Three kinds of data for robotics 10:32 A GPT-3 moment for robotics 16:05 Choosing the humanoid robot form factor 19:37 Specialized generalists 21:59 GR00T gets its own chip 23:35 Eureka and Issac Sim 25:23 Why now for robotics? 28:53 Exploring virtual worlds 36:28 Implications for games 39:13 Is the virtual world in service of the physical world? 42:10 Alternative architectures to Transformers 44:15 Lightning round

    49 min
  7. Founder Eric Steinberger on Magic’s Counterintuitive Approach to Pursuing AGI

    10 SEPT.

    Founder Eric Steinberger on Magic’s Counterintuitive Approach to Pursuing AGI

    There’s a new archetype in Silicon Valley, the AI researcher turned founder. Instead of tinkering in a garage they write papers that earn them the right to collaborate with cutting-edge labs until they break out and start their own. This is the story of wunderkind Eric Steinberger, the founder and CEO of Magic.dev. Eric came to programming through his obsession with AI and caught the attention of DeepMind researchers as a high school student. In 2022 he realized that AGI was closer than he had previously thought and started Magic to automate the software engineering necessary to get there. Among his counterintuitive ideas are the need to train proprietary large models, that value will not accrue in the application layer and that the best agents will manage themselves. Eric also talks about Magic’s recent 100M token context window model and the HashHop eval they’re open sourcing. Hosted by: Sonya Huang, Sequoia Capital Mentioned in this episode: David Silver: DeepMind researcher that led the AlphaGo team Johannes Heinrich: a PhD student of Silver’s and DeepMind researcher who mentored Eric as a highschooler Reinforcement Learning from Self-Play in Imperfect-Information Games: Johannes’s dissertation that inspired Eric  Noam Brown: DeepMind, Meta and now OpenAI reinforcement learning researcher who eventually collaborated with Eric and brought him to FAIR ClimateScience: NGO that Eric co-founded in 2019 while a university student  Noam Shazeer: One of the original Transformers researchers at Google and founder of Charater.ai  DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit Poker: the first AI paper Eric ever tried to deeply understand LTM-2-mini: Magic’s first 100M token context model, build using the HashHop eval (now available open source) 00:00 - Introduction 01:39 - Vienna-born wunderkind 04:56 - Working with Noam Brown 8:00 - “I can do two things. I cannot do three.” 10:37 - AGI to-do list 13:27 - Advice for young researchers 20:35 - Reading every paper voraciously 23:06 - The army of Noams 26:46 - The leaps still needed in research 29:59 - What is Magic? 36:12 - Competing against the 800-pound gorillas 38:21 - Ideal team size for researchers 40:10 - AI that feels like a colleague 44:30 - Lightning round 47:50 - Bonus round: 200M token context announcement

    51 min

  8. 3 SEPT.

    Crucible Moments Returns for S2: The ServiceNow Story ft. CEO Frank Slootman & Founder Fred Luddy

    On Training Data, we learn from innovators pushing forward the frontier of AI’s capabilities. Today we’re bringing you something different. It’s the story of a company currently implementing AI at scale in the enterprise, and how it was built from a bootstrapped idea in the pre-AI era to a 150 billion dollar market cap giant.  It’s the Season 2 premiere of Sequoia’s other podcast, Crucible Moments, where we hear from the founders and leaders of some legendary companies about the crossroads and inflection points that shaped their journeys. In this episode, you’ll hear from Fred Luddy and Frank Slootman about building and scaling ServiceNow. Listen to Crucible Moments wherever you get your podcasts or go to: Spotify: https://open.spotify.com/show/40bWCUSan0boCn0GZJNpPn Apple: https://podcasts.apple.com/us/podcast/crucible-moments/id1705282398 Hosted by: Roelof Botha, Sequoia Capital Transcript: https://www.sequoiacap.com/podcast/crucible-moments-servicenow/

    43 min
Tout afficher (20)

4,7
sur 5
24 notes

À propos

Join us as we train our neural nets on the theme of the century: AI. Sonya Huang, Pat Grady and more Sequoia Capital partners host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies—and their implications for technology, business and society. The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.

Informations

  • Chaîne
    Sequoia Capital
  • Création
    Sequoia Capital
  • Années d’activité
    2 k
  • Épisodes
    20
  • Classification
    Tous publics
  • Site web de l’émission
    Training Data

Plus de contenus par Sequoia Capital

Vous aimeriez peut‑être aussi

Pour écouter des épisodes au contenu explicite, connectez‑vous.

Recevez les dernières actualités sur cette émission

Connectez‑vous ou inscrivez‑vous pour suivre des émissions, enregistrer des épisodes et recevoir les dernières actualités.
Choisissez un pays ou une région

Afrique, Moyen‑Orient et Inde

Asie‑Pacifique

Europe

Amérique latine et Caraïbes

États‑Unis et Canada