Learn AI in Bits

Dan W
  • 5.0 (1)
  • Technology
  • Updated Semiweekly

AI explained in bits. Each episode takes one concept, like tokens, embeddings, hallucinations, or prompt injection, and explains it in about five minutes. No jargon, no filler. Just the idea, why it matters, and what to remember. If you're curious about AI or already building with it, you'll come away understanding how these systems work. One concept. Five minutes. That's the whole show.

  1. 2d ago

    086 - What Is Recursive Self-Improvement?

    Recursive self-improvement is the idea that an AI can improve the very process it uses to improve itself, and this episode separates what's been demonstrated so far from the dramatic predictions often attached to the term. It draws a clear line between self-correction (revising one answer), automated optimization (keeping the best of several tested variants), and true recursive improvement, where the system changes the process that generates future improvements, so the next round starts with a better improver. The central example is AIDE², a September 2026 research system with a nested loop: an inner agent performs AI research and coding tasks, while an outer agent rewrites the inner agent's search policy, context management, and workflow, with each candidate version evaluated against held-out tests so the system can't just game the acceptance criteria. In one autonomous eight-day run, the researchers reported seven successive improvements, including a new search policy and context-compression mechanisms, with the resulting agent performing competitively against a strong human-engineered research agent across four external benchmarks, including an out-of-domain weather-forecasting task. The episode is explicit about the limits of that result: it shows improvement at the level of agent code and workflow, not an AI redesigning its own neural architecture or becoming unboundedly more intelligent. It covers why evaluation is the hardest part of any improvement loop, including the risk of reward hacking, where a score improves without the underlying goal improving, and notes that the AIDE² researchers reported reward hacking declining during their run while flagging genuine uncertainty about whether the resulting system could keep improving as the outer-loop improver itself. It also covers the practical limits of compute, time, and cost that any such loop still runs into. The episode closes on October 2026 remarks from Google cofounder Sergey Brin, reported by Business Insider, describing Google's researchers and models working together on successor models and the company getting onto what he called the "recursive self-improvement treadmill," alongside Google's statement that AI agents now write much of its production code under human review — framed as growing AI assistance in development, not evidence that fully autonomous recursive self-improvement has arrived. Sources & References Srikanth, Zhao, Xu, Wu, Jiang: Recursive self-improvement of AI research agents (arXiv, September 2026) — https://arxiv.org/abs/2609.26457 Business Insider: Sergey Brin says Google's AI models are reaching a critical moment of self-improvement — https://www.businessinsider.com/sergey-brin-google-gemini-ai-argon-recursive-self-improvement-rsi-2026-10 Wang et al.: Recursive Self-Improvement in AI: A Survey (Preprints.org, September 2026) — https://www.preprints.org/manuscript/202609.2680 Voice narration is AI-generated.

  2. 4d ago

    AI News Sunday Wrap-Up (10/12–10/18)

    This week's AI News Sunday Wrap-Up covers a week where the industry moved on several fronts at once: new flagship and open models, enterprise agent governance, a serious safety warning, legal pressure, and the physical infrastructure limits behind the AI race. Google launched Gemini 4 Argon on October 13, built for complex multi-step work across coding, research, business tasks, and cybersecurity, with new task-oriented features in the Gemini app for connecting services and carrying out cross-app work; Google also described its own safety testing, including a security partner finding a critical vulnerability in hospital software. Meta had its busiest week yet: introducing Muse Wonder and Wonder Full on October 12 around Active Memory and Active Focus, then releasing the open-weight Muse Spark Open on October 15, an 816-billion-parameter multimodal model with about 42 billion active per token, a 1-million-token context window, and the ability to run across two Mac Studio systems with M5 Ultra chips. NVIDIA followed with Nemotron 3.5 Super VL, a 120-billion-parameter open model for specialized agents, paired with smaller Nemotron Bolt models built for local and edge deployment. On the enterprise side, ServiceNow announced AI Passport, a runtime control layer for checking what AI agents and MCP servers are authorized to do. The most serious story was Anthropic's Frontier Red Team research warning that small models may soon be capable of spreading across inexpensive internet-connected devices: a 9-billion-parameter model ran inference on 8 of 9 tested hardware types, and stronger models completed a simplified end-to-end replication exercise in controlled conditions, a forward-looking risk assessment rather than a report of spread happening in the wild. Legal and economic pressure also mounted: eight U.S. states sued OpenAI on October 14 over alleged harms to young users, and Anthropic announced on October 18 that it would commit 5% of its shares, subject to stockholder approval, to a new Public Benefit Fund aimed at nonprofits and people affected by labor-market disruption. The episode closes on infrastructure limits, with Reuters reporting Microsoft's withdrawal from a UK data-center project over power constraints, and the World Bank's push for "small AI" suited to limited connectivity and local languages in developing economies. Sources & References Google: Gemini 4 Argon comes to the Gemini app — https://blog.google/innovation-and-ai/products/gemini-app/gemini-4-argon-availability/ Google: Developing Gemini 4 Argon responsibly and safely — https://blog.google/innovation-and-ai/technology/safety-security/gemini-4-model-safety-security/ Meta AI Research: Muse Wonder: A Frontier Model For Personal & Professional AI Agents — https://research.meta.ai/blog/muse-wonder Meta AI Research: Introducing Muse Spark Open, The Most Capable US Open-Weight Model to Date — https://research.meta.ai/blog/muse-spark-open NVIDIA: Open From Data Center to Desktop: NVIDIA Expands Nemotron Family With Highest Efficiency Super VL and Bolt Models — https://blogs.nvidia.com/blog/nemotron-3-5-super-vl/ ServiceNow: ServiceNow advances its operational harness for enterprise AI with AI Passport and expanded governance capabilities — https://newsroom.servicenow.com/press-releases/details/2026/ServiceNow-advances-its-operational-harness-for-enterprise-AI-with-AI-Passport-and-expanded-governance-capabilities/default.aspx Voice narration is AI-generated.

  3. 6d ago

    085 - All About Meta's Muse Spark

    Meta's Muse story is moving fast, and this episode takes a closer look at Muse Spark, the multimodal reasoning model family behind Meta's push into AI agents, including yesterday's release of an open-weight version for developers. Muse Spark launched in April as the first model from Meta Superintelligence Labs, designed to handle text and visual inputs, reason through difficult questions, call tools, and coordinate multiple agents, with Meta's own examples including analyzing a photo, identifying appliance parts, and turning a visual prompt into an interactive experience. Muse Spark 1.3, released in September, focused on more dependable long-running agent work: holding onto detailed instructions across a long thread, finding gaps in its own plan, asking clarifying questions when a request is ambiguous, and confirming before consequential actions. Meta reports that, in its own coding comparisons, 1.3 used about 20% fewer tool calls and 25% fewer tokens than Spark 1.2, though that's Meta's evaluation setup rather than a guarantee for every project. The headline news is Muse Spark Open, released October 15 under the Muse Model License with limited commercial terms. The model has 816 billion total parameters with roughly 42 billion active per token in a sparse mixture-of-experts design, a 1-million-token context window, and support for text, image, video, and audio input. It's available through hosting platforms and APIs, downloadable from services like Hugging Face, and runs on inference engines including vLLM, SGLang, TensorRT-LLM, Transformers, and Apple's MLX framework. Notably, Meta and Apple optimized it to run locally across two Mac Studio systems with M5 Ultra chips, each offering up to 512 gigabytes of unified memory, together holding the full model — a serious desktop setup, not a typical laptop. Meta demonstrated this configuration using Muse Code with Spark Open to ship changes to an open-source iOS app and operate DaVinci Resolve end-to-end to review footage and create a vertical crop. The episode is careful about the trade-offs: open weights don't automatically make a model cheap to run, a long context window doesn't guarantee correct retrieval or use of every detail, and a model that can operate a terminal or applications needs clear permission boundaries, sandboxing, and human approval for risky actions, on top of the additional safety post-training Meta says Spark Open received as a standalone release. Sources & References Meta AI Research: Introducing Muse Spark: Scaling Towards Personal Superintelligence — https://ai.meta.com/blog/introducing-muse-spark-msl/ Meta AI Research: Introducing Muse Spark 1.3 — https://research.meta.ai/blog/introducing-muse-spark-1-3 Meta AI Research: Introducing Muse Spark Open, The Most Capable US Open-Weight Model to Date — https://research.meta.ai/blog/muse-spark-open Meta Developer: Build with Muse Spark Open — https://dev.meta.ai/resources/blog/build-with-muse-spark-open Voice narration is AI-generated.

  4. Oct 15

    084 - Speculative Decoding: Making LLMs Generate Faster

    When a large language model generates text one token at a time, even a fast GPU hits a sequential bottleneck. This episode explains speculative decoding, an inference technique that prepares several tokens at once and checks them together, and when the extra computation pays off. Ordinary generation makes the target model run a full forward pass for every single token, since each prediction depends on the one before it. Speculative decoding adds a cheap draft step: a smaller model, a lightweight prediction head, or a pattern-matching method proposes a short run of likely next tokens, and the target model verifies the whole batch in one pass. Using the illustrative example "The capital of France is" followed by candidate tokens "Paris," "and," "it," "is," the episode walks through how the target model accepts a correct prefix, rejects a wrong candidate, and continues generating from that rejection point rather than discarding everything. Done with the right verification and sampling procedure, this preserves the target model's exact output distribution — the speedup doesn't require swapping in a weaker model. The original 2023 ICML paper introducing the technique reported a 2x-3x acceleration in its T5 experiments, a result specific to that setup rather than a universal guarantee. The episode covers the main ways to build a draft: a separate smaller model (which needs a compatible tokenizer with the target), prediction heads or internal-feature-based methods like EAGLE, multi-token prediction, and N-gram or prompt-lookup methods that reuse matching text already in the prompt without a second model at all. It explains why the benefit depends on acceptance rate, draft cost, and available spare compute, and why speculative decoding can even reduce throughput once a GPU is already busy serving many concurrent requests. The episode closes with concrete evaluation guidance: measure time to first token, inter-token latency, accepted-draft rate, memory overhead, and total throughput across varied workloads and concurrency levels, since a setup that feels faster for one user can perform worse once the server gets busy. Sources & References Leviathan, Kalman, and Matias: Fast Inference from Transformers via Speculative Decoding (ICML 2023) — https://proceedings.mlr.press/v202/leviathan23a.html vLLM Documentation: Speculative Decoding — https://docs.vllm.ai/en/latest/features/speculative_decoding/index.html NVIDIA TensorRT-LLM Documentation: Speculative Decoding — https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/speculative-decoding.md NVIDIA Triton Inference Server Documentation: Speculative Decoding with TensorRT-LLM — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Feature_Guide/Speculative_Decoding/TRT-LLM/README.html Voice narration is AI-generated.

  5. Oct 15

    083 - Why LLMs Are Slow: Inside GPU Inference

    When an AI model answers slowly, the problem usually isn't its intelligence — it's how the serving system moves data through a GPU. This episode goes one level below the basic inference loop covered in an earlier episode, into GPU memory, the key-value cache, attention kernels, and the scheduling techniques used by systems like vLLM and NVIDIA TensorRT-LLM. It starts by separating prefill, which processes prompt tokens in parallel and benefits from raw compute throughput, from decode, which is autoregressive and becomes memory-bound because each step needs the model weights and the growing attention state for every prior token. The episode works through a concrete KV-cache size estimate: with 32 transformer layers, 8 key-value heads, a head dimension of 128, and 2 bytes per cached value, each token costs 131,072 bytes, or 128 kibibytes, of cache; at 8,192 tokens, one sequence's cache runs to roughly a gibibyte before overhead, showing why long context and concurrent users turn KV-cache capacity into a scheduling problem. From there, it covers the main optimizations serving engines use: grouped-query attention (GQA), which shares key-value heads across multiple query heads to shrink the cache; FlashAttention, which computes exact attention while cutting memory traffic through tiling and on-chip memory; continuous batching, which replaces finished requests in an active batch instead of waiting for a fixed batch to clear; chunked prefill, which splits long prompts into pieces interleaved with decode work so one long prompt doesn't stall other users; PagedAttention, which allocates KV-cache memory in non-contiguous blocks to reduce fragmentation; prefix caching, which reuses cached state across requests sharing an identical prompt prefix; and quantization, which trades numerical precision for memory and speed. The episode closes with the metrics that tie these optimizations to specific symptoms: time to first token for queueing and prefill issues, inter-token latency and throughput for decode and batching issues, and memory use and tail latency under concurrency for capacity issues, arguing that LLM inference is as much a hardware-and-scheduling problem as a model problem. Sources & References NVIDIA Technical Blog: Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference — https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference NVIDIA Technical Blog: Mastering LLM Techniques: Inference Optimization — https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization vLLM Documentation: Automatic Prefix Caching — https://docs.vllm.cc/en/latest/design/automatic_prefix_caching.html vLLM Documentation — https://docs.vllm.cc/en/latest/ NVIDIA TensorRT-LLM Documentation: Paged Attention, IFB, and Request Scheduling — https://nvidia.github.io/TensorRT-LLM/1.2.0rc3/features/paged-attention-ifb-scheduler.html NVIDIA Technical Blog: Streamlining AI Inference Performance and Deployment with TensorRT-LLM Chunked Prefill — https://developer.nvidia.com/blog/?p=92052 Voice narration is AI-generated.

  6. Oct 14

    082 - The OpenAI Bugging Incident: AI Helps Investigate a Hidden Recording Device

    A hidden recording device was found behind a television in an OpenAI executive conference room, concealed in toilet paper and plugged into a USB port, leading to an FBI arrest and federal charges. This episode follows the publicly reported timeline and examines OpenAI's statement that its own AI models helped analyze the device and identify a suspect, while being careful to separate established facts from allegations in an ongoing case. According to the criminal complaint from the U.S. Department of Justice, OpenAI security staff found the device on September 30, 2026, installed around September 28, in a San Francisco conference room used for meetings involving confidential technical, commercial, and strategic information. Office video allegedly showed Abdulrahman Alfozan, a 35-year-old OpenAI engineer and Saudi national, near the room on multiple dates despite lacking authorization, and footage from October 5 allegedly showed him looking behind the television where the device was found; a second device turned up the same day in a bag containing his wallet and identification. He was arrested October 9 and charged October 13 with using or attempting to use a device to intercept oral communications. The episode is precise about the AI angle: OpenAI says its internal models helped it understand the device's hardware and trace it to an employee, though the company hasn't detailed exactly which models or how much of that process was automated. The complaint separately alleges that a chatbot account linked to Alfozan's spouse asked about the device's custom header and network identifier shortly before its discovery, and that this account shared identifiers connecting it to Alfozan himself, alongside his own productivity-tool searches about office logistics. The episode is explicit that a model's output is a lead investigators still have to verify, not proof on its own, and that AI assisted both the investigation and, allegedly, the planning. OpenAI has said it found no indication the employee improperly accessed model weights, source code, or customer data. The episode closes on a measured note: an arrest isn't a conviction, the public record doesn't establish motive, and the organizational lesson worth taking is pairing AI-assisted analysis with access controls, monitoring, and a process for verifying what AI-generated leads show. Sources & References U.S. Department of Justice: Saudi National Arrested for Installing Surreptitious Audio Recording Device at OpenAI — https://www.justice.gov/opa/pr/saudi-national-arrested-installing-surreptitious-audio-recording-device-openai U.S. Attorney's Office, Northern District of California: Saudi National Arrested for Installing Surreptitious Audio Recording Device at OpenAI in San Francisco — https://www.justice.gov/usao-ndca/pr/saudi-national-arrested-installing-surreptitious-audio-recording-device-openai-san TechCrunch: FBI arrests OpenAI employee accused of planting hidden recording device in conference room — https://techcrunch.com/2026/10/13/fbi-arrests-openai-employee-accused-of-planting-hidden-recording-device-in-conference-room/ WIRED: An OpenAI Employee Allegedly Used a Chatbot to Help Plant a Recording Device at the Office — https://www.wired.com/story/an-openai-employee-allegedly-used-a-chatbot-to-help-plant-a-recording-device-at-the-office/ Voice narration is AI-generated.

  7. Oct 13

    081 - How Does AI Inference Work?

    When you type a question into an AI assistant and watch the answer appear word by word, what's happening underneath is inference: the process of running a trained model to produce that answer. This episode follows one request through the full inference path, from tokenization to the KV cache to production serving. A prompt is first assembled with any system instructions, conversation history, and context, then converted by a tokenizer into token IDs. The model processes that token sequence through its transformer layers, including attention, producing logits, which are scores for every possible next token. A decoding step turns those scores into a choice, whether through greedy decoding, which picks the top score, or sampling methods shaped by settings like top-k, top-p, and temperature. The model generates output one token at a time this way, rather than retrieving a complete answer from a database. The episode explains the prefill and decode split: prefill processes the input prompt and builds attention state, while decode generates output tokens one at a time, reusing that state through the key-value cache, or KV cache, so the model doesn't have to recompute previous context on every step. It distinguishes time to first token from inter-token latency, and explains how the KV cache becomes a major memory consideration as conversations and concurrent requests grow, which is why serving systems manage cache allocation, batching, and sometimes prefix reuse across requests. Zooming out to production, the episode covers how inference engines schedule and batch requests across accelerators, how quantization trades precision for memory and speed, and why practical performance is measured in latency, throughput, memory use, and cost per token rather than benchmark scores alone. It also distinguishes the model's inference step from the rest of an application's pipeline, including retrieval, safety filters, and multimodal input encoding, and closes with a key clarification: running inference on a deployed model doesn't mean it's learning from your conversation as it happens, since that depends entirely on the service's own data and training policies. Sources & References NVIDIA TensorRT Model Connect: Inference Fundamentals — https://github.com/NVIDIA/TensorRT-Model-Connect/blob/main/website/docs/getting-started/inference-fundamentals.md vLLM: Inside vLLM: Anatomy of a High-Throughput LLM Inference System — https://vllm.ai/blog/2025-09-05-anatomy-of-vllm vLLM Documentation: Disaggregated Prefilling (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/ NVIDIA Technical Blog: Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache — https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache Voice narration is AI-generated.

  8. Oct 12

    080 - [Extended] AI at Work: From Chatbots to Projects, Skills, and Agents

    If you already know your job well, AI raises a new question: which parts of your work could you do better, faster, or at a larger scale with an assistant that reads, writes, analyzes, searches, and sometimes acts? This episode maps what's available from OpenAI, Anthropic, and Google beyond basic chat, and how an experienced worker can put it to use. The episode frames the toolset in layers: the model itself, the chat interface, Projects or knowledge files for ongoing context, reusable Skills, Connectors for approved tools and data, and agents that combine all of it into a multi-step loop. In ChatGPT, Projects keep related chats, files, and instructions together for work that spans more than one conversation, while connected apps and, for eligible users, scheduled tasks and workspace agents extend what it can search and do. Claude's progression runs through Claude.ai projects and connectors to Claude Code, which can inspect a repository, edit files, and run commands, supported by Anthropic's Model Context Protocol, MCP, for tool connectivity and Skills for reusable workflow knowledge. Google is adding Markdown-native skills to Gemini and Workspace, built on the open SKILL.md standard, with Gems transitioning toward that format over time. The episode explains retrieval-augmented generation, or RAG, as a way to ground answers in a company's own documents by retrieving relevant passages before generating a response, while cautioning that poor retrieval or outdated documents can still produce a wrong answer that needs checking. It walks through a practical example for an operations manager, then covers company deployments: OpenAI's published stories on Asana, Sophos, Oracle, and Lowe's, and Anthropic's account of Barclays scaling Claude across operations and client service, all framed explicitly as company-reported examples rather than independent verification. The episode closes with concrete guidance on agent permissions and prompt injection risk, and a method for comparing vendors using three of your own everyday tasks instead of benchmark charts. The throughline: start with one specific problem, add capability only as the task demands it, and let years of professional judgment guide and verify what the AI produces. Sources & References OpenAI Help Center: ChatGPT release notes — https://help.openai.com/en/articles/6825453-chatgpt-release-notes OpenAI Help Center: Connected apps in ChatGPT — https://help.openai.com/en/articles/11487775-connected-apps-in-chatgpt OpenAI Help Center: ChatGPT Workspace Agents for Enterprise and Business — https://help.openai.com/en/articles/20001143/ OpenAI: Customer stories — https://openai.com/business/customer-stories/ Anthropic: Model Context Protocol (MCP) documentation — https://docs.anthropic.com/en/docs/mcp Anthropic: Fundamentals: The complete guide to building Skills for Claude — https://claude.com/resources/guides/the-complete-guide-to-building-skills-for-claude/fundamentals Anthropic: Barclays scales Claude to upgrade operations and improve client experience — https://www.anthropic.com/news Google Workspace Updates: Introducing skills in the Gemini app and Workspace, plus what's next for Gems — https://workspaceupdates.googleblog.com/2026/09/skills-gemini-app-workspace.html Google Workspace: Less switching, more flow: 5 new agentic capabilities across Google Workspace apps — https://workspace.google.com/blog/product-announcements/less-switching-more-flow-5-new-agentic-capabilities-across-google-workspace-apps Anthropic: Introducing Claude for Small Business — https://www.anthropic.com/news/claude-for-small-business Anthropic: Investigating unintended model actions in our evaluations and internal use — https://www.anthropic.com/news/investigating-unintended-model-actions Voice narration is AI-generated.

See All (92)

About

AI explained in bits. Each episode takes one concept, like tokens, embeddings, hallucinations, or prompt injection, and explains it in about five minutes. No jargon, no filler. Just the idea, why it matters, and what to remember. If you're curious about AI or already building with it, you'll come away understanding how these systems work. One concept. Five minutes. That's the whole show.

Information

  • Creator
    Dan W
  • Years Active
    2026
  • Episodes
    92
  • Rating
    Clean
  • Copyright
    © Dan W
  • Show Website
    Learn AI in Bits

You Might Also Like