AI Journey OpenBook Podcast

Bowen
  • 0.0 (0)
  • TECHNOLOGY
  • UPDATED WEEKLY

Implementation for the different ML tasks on the limit computing resources. aisuko.substack.com

Episodes

  1. 02/14/2025

    Training AI Assistants: From Pre-Training to Human-Guided Conversations

    Pretraining and post-training chapter of Deep Dive into LLMs like ChatGPT by Andrej Karpathy. Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    20 min

  2. 02/14/2025

    Understanding Base Model Inference: How AI Models Generate Text

    In this section, the speaker explores the concept of base model inference, explaining how large AI models are trained, released, and function as token simulators rather than full assistants. The key points discussed include: Base Models and Their Availability * Training large AI models is extremely costly, but big tech companies often release “base models“ after training. * A base model is a token simulator that predicts text sequences but is not yet an AI assistant. Examples of Base Models * GPT-2 (1.5B parameters, trained on 100B tokens) was one of the first widely released base models. * LLaMA3 (405B parameters, trained on 15T tokens by Meta) is a modern, larger base model. Components of a Model Release * Requires two main parts: * Python Code * Model Parameters Base Model Behavior * It functions as an advanced autocomplete system, generating text based on statistical patterns from training data. * It does not inherently provide factual or structured responses like an assistant. Characteristics of Base Models * Stochastic Nature - Given the same input, different competitions may be generated. * Knowledge Compression - Acts like a lossy “zip file” of internet text, storing probabilistic patterns rather than explicit facts. * Memorization & Regurgitation - Can recall high-frequency training data, sometimes verbatim (e.g, Wikipedia entries). Limitations of Base Models * Cannot provide factual updates beyond their training data cutoff. * Tends to “hallucinate” (generate plausible but false information) Practical Uses of Base Models * In-context Learning - Few-shot prompting enables them to recognise and follow simple patterns * Simulated AI assistants - Carefully structured prompts can trick a base model into behaving like an assistant by mimicking a conversation format Acknowledgment Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    17 min

  3. 02/13/2025

    The Internal Working and Training of Neural Networks

    The Neural Network Internals and Training of Angrej Karpathy's video Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    6 min

  4. 02/12/2025

    The Neural Network I⧸O

    The video clip of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    6 min

  5. 02/12/2025

    Speech Recognition via Whisper

    This is chapter3 from Andrej Karpathy video Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    7 min

  6. 02/10/2025

    Understanding Reasoning Models

    Understanding Reasoning Model Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    1 min

  7. 02/08/2025

    Understanding Reasoning in Large Language Models

    The original article is https://www.linkedin.com/pulse/understanding-reasoning-large-language-models-bowen-li-kgadc/?trackingId=WtLIWBUiTFmVwCPL4HCwUw%3D%3D The audio generated by Amazon Polly. It is not better than the OpenAI ChatGPT audio service. Get full access to AI Journey OpenBook at aisuko.substack.com/subscribe

    2 min
  • 7 Episodes

About

Implementation for the different ML tasks on the limit computing resources. aisuko.substack.com

Information