Kubernetes Bytes

Ryan Wallner & Bhavin Shah

Kubernetes Bytes is a podcast bringing you the latest from the world of cloud native data management. Hosts Ryan Wallner and Bhavin Shah come to you from Boston, Massachusetts with experienced backgrounds in cloud-native tech. They'll be sharing their thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud-native ecosystem.

  1. 3d ago

    Running LLMs is easy. Scaling Inference on Kubernetes is Hard!

    Running an open-source LLM locally is increasingly easy.   Running that same model efficiently for hundreds or thousands of users is an entirely different engineering problem.   In this episode of Kubernetes Bytes, Bhavin sits down with Cedric Clyburn from Red Hat to unpack what actually happens when organizations move from experimenting with local models to operating AI inference at scale on Kubernetes.   They discuss why traditional application metrics don’t fully describe LLM performance and introduce the three important inference metrics: time to first token, inter-token latency, and throughput.  Cedric also explains what inference servers such as vLLM actually do, how KV cache consumes GPU memory, why agentic workloads dramatically increase inference demand, and how optimizations such as continuous batching and paged attention help make expensive GPUs more efficient.   The conversation goes deeper into GPU architecture and memory movement, including HBM, SRAM, prefill and decode, before exploring model compression and quantization as another way to reduce infrastructure requirements.   Finally, they discuss llm-d and the emerging Kubernetes-native distributed inference stack, including KV-cache-aware routing, Day 2 model operations, model upgrades, and why AI sovereignty and compliance may drive more organizations toward self-hosted inference.   Topics include: Hosted AI APIs vs. running your own models AI inference economics Time to first token (TTFT) Inter-token latency (ITL) Token throughput Why inference servers exist  vLLM and SGLang KV cache GPU memory and HBM  Prefill vs. decode Continuous batching Paged attention  Distributed inference Model compression and quantization  LLM evaluation and benchmarking  vLLM vs. llm-d Kubernetes inference-aware routing  Day 2 operations for AI models AI sovereignty and private inference  Running smaller models locally on CPUs and edge devices  If you’re a Kubernetes, platform engineering, infrastructure, or AI engineering practitioner trying to understand how production LLM serving actually works, this episode provides a practical introduction to the infrastructure underneath modern AI applications.   Show Notes:  Cedric Clyburn - https://www.linkedin.com/in/cedricclyburn/ IBM Technology Youtube - https://www.youtube.com/@IBMTechnology https://github.com/vllm-project/vllm https://llm-d.ai/  https://github.com/kserve/kserve  https://gateway-api.sigs.k8s.io/  https://github.com/vllm-project/llm-compressor  https://github.com/vllm-project/guidellm

  2. Oct 26

    We Built an AI Developer Platform on Kubernetes

    What does it actually look like when a platform team starts using AI agents in production engineering workflows?  In this episode of Kubernetes Bytes, we sit down with Berkay from Futureplay to talk about how a small infrastructure and server engineering team is using Kubernetes to support mobile games, internal developer platforms, and increasingly AI-assisted software development.   The conversation starts with Futureplay’s Kubernetes architecture and its philosophy of keeping infrastructure lean and avoiding unnecessary complexity. Berkay explains how the team uses a decision journal to evaluate technology choices, how they decide between managed services and self-hosting, and how they provide self-service environments for developers.   Then we get into how AI is changing their engineering workflow.   Futureplay has built an internal system where users can submit requests through Slack, an orchestrator interprets and scopes the request, and Kubernetes Jobs spin up specialized AI workers for building, evaluating, and verifying the change.   The system can ask clarifying questions, create code, open a pull request, deploy a sandbox preview, collect feedback, and iterate — while still keeping human approval and security boundaries in place.   Topics include: Why Futureplay chose Kubernetes Keeping Kubernetes infrastructure lean Buy vs. build decisions for databases and infrastructure Developer self-service and ephemeral environments Using AI for Terraform, Helm, CI/CD, and infrastructure changes Why AI-generated production changes still require validation Agentic software development with Codex and Claude Code From Slack requests to pull requests and live previews Running AI workers as Kubernetes Jobs Building an in-house AI orchestrator Agent isolation, network policies, identity, and permissions Why Futureplay uses a dedicated cluster for AI workloads Why they currently choose hosted models over self-hosted GPUs Measuring platform engineering success Running a small team with near-five-nines availability  If you’re trying to understand what practical AI adoption looks like inside a real engineering organization — beyond demos and vendor promises — this episode offers a detailed look at one team’s evolving approach.   Subscribe to Kubernetes Bytes for conversations about Kubernetes, cloud native infrastructure, platform engineering, AI, security, and the CNCF ecosystem.

  3. Oct 2

    What if Kubernetes Could Resume Instead of Restart? | Checkpoint / Restore Explained

    Kubernetes usually handles failures and rescheduling by restarting workloads. But what if Kubernetes could save the exact running state of a workload, move it somewhere else, and resume from where it stopped?   In this episode of Kubernetes Bytes, Bhavin sits down with Radostin Stoyanov, maintainer of CRIU (Checkpoint/Restore In Userspace) and a contributor helping lead the Kubernetes Checkpoint/Restore Working Group.   They explore how checkpoint/restore works, why it is becoming increasingly relevant for Kubernetes and AI workloads, and how the community is working toward pod-level checkpoint and restore.   The conversation covers how checkpointing can help reduce expensive AI inference cold starts, improve fault tolerance for large GPU training jobs, enable faster model swapping, and potentially influence future Kubernetes scheduling and preemption workflows.  They also discuss the challenges of checkpointing distributed applications, GPU memory, compression, security, encryption, and integrating checkpoint/restore directly into Kubernetes APIs.   Topics include:  What checkpoint/restore actually saves Application-level vs. infrastructure-level checkpointing CRIU and Kubernetes Why AI inference workloads can take minutes to initialize Saving CPU and GPU execution state  Pod-level vs. container-level checkpointing  Kubernetes Checkpoint/Restore Working Group  Distributed checkpoint/restore  Fault tolerance for large GPU training jobs  Accelerating model startup and model swapping  Memory compression for large checkpoints Kubernetes APIs and controllers for checkpoint/restore  Security risks of checkpoint files  Encryption and validation  Scheduler preemption and workload migration  Checkpoint/restore for AI-agent sandboxes   If you work with Kubernetes, GPUs, AI infrastructure, inference, distributed training, or scheduling, this episode provides a look at a Kubernetes capability that could become increasingly important as workloads become more expensive to restart.   Subscribe to Kubernetes Bytes for conversations about Kubernetes, cloud native infrastructure, AI, platform engineering, security, and the CNCF ecosystem.

  4. Sep 15

    Your AI Agent Has Too Much Access | Fixing Agent Identity

    AI agents can do far more than answer questions — they can call tools, access data, execute actions, and act on behalf of users. But once agents start taking actions, an important question emerges: who is the agent actually acting as, and what should it be allowed to do? In this episode of Kubernetes Bytes, Bhavin sits down with Maia Iyer, Research Software Engineer at IBM Research, to explore identity, authentication, authorization, and zero trust for AI agents. They discuss why static API keys and simply passing user credentials to agents can create security problems, and how existing standards such as OAuth 2.0, SPIFFE, SPIRE, and Keycloak can help establish verifiable user and workload identities. Maia also explains the work her team is doing around Rosso CTL and Cortex, including using a runtime layer around agents to handle identity, observe inbound and outbound interactions, apply guardrails, and explore intent-based access control. Topics covered include: • The building blocks of agentic applications• Security anti-patterns for AI agents• Static credentials and over-permissioned agents• User identity vs. workload identity• OAuth 2.0 and delegated identity• SPIFFE and SPIRE• Keycloak• Rosso CTL and Cortex• Intent-based access control• Prompt injection and unexpected tool calls• MCP and agent-to-tool communication• Stateful AI agents• Safely running agents that generate and execute code If you're building AI agents on Kubernetes or thinking about how identity and authorization need to evolve for agentic systems, this episode provides a practical mental model for getting started. Subscribe to Kubernetes Bytes for conversations about Kubernetes, cloud native infrastructure, platform engineering, security, AI infrastructure, and the CNCF ecosystem.

  5. Aug 28

    Your CI Tool is NOT a CD Tool!

    In this episode of the Kubernetes Bytes podcast, Bhavin talks to Pushkar, Staff Software Engineer at Snap Inc. The discussion starts by talking about how platform teams support developers by building internal developer platforms, what are the capabilities developers expect in an IDP. They then talk about how reusing a Continuous Integration tool for Continuous Deployment led to issues and pagerduty alerts. Finally the discussion focusses on Argo Rollouts and the benefits it offers for rolling out updates to your production environment. This is a real world scenario where Pushkar shares lessons learnt, best practices, etc. from his experience onboarding Argo inside their organization.     Check out our website at https://kubernetesbytes.com/   Show Notes: https://argoproj.github.io/rollouts/  https://argo-cd.readthedocs.io/en/stable/  https://buildkite.com/home/

  6. Aug 14

    Optimizing AI spend with Cast AI

    In this episode of the Kubernetes Bytes podcast, Bhavin talks to Phil Andrews, Global Field CTO at Cast AI about all things AI. The discussion starts by talking about why you don't need an Opus class or Fable/Mythos class model for each task, and how Kimchi from Cast AI can help manage the quality of your output with the cost associated with routing between different models. They also talk about Omni which allows customers to get GPUs across hyperscalers and neocloud providers. Listen to learn more!  Check out our website at https://kubernetesbytes.com/   Show Notes: https://www.linkedin.com/in/philip-andrews-iii/ https://cast.ai/kimchi/ https://docs.cast.ai/docs/omni-overview  https://cast.ai/blog/

  7. Jul 2

    Securing Kubernetes for the Mythos Era

    In this episode of the Kubernetes Bytes podcast, Bhavin talks to Alex Zenla, Founder and CTO of Edera. The discussion starts by talking about why Alex and team decided to start Edera, where in the stack does it fit in, and how can it work with existing Kubernetes clusters. They also talk about GPU security, what Kubernetes DRA is, and how to plan for the new era of Mythos-class models. Listen to learn more!  Check out our website at  https://kubernetesbytes.com/    Show Notes:  https://www.linkedin.com/in/azenla/ https://edera.dev/ https://github.com/azenla  https://edera.dev/stories

  8. Jun 18

    More Code, More Problems: Software Delivery in the Agentic Era

    In this episode of the Kubernetes Bytes podcast, Bhavin talks to Rob Zuber, CTO at CircleCI about all things Software Delivery. The discussion starts by talking about the evolution of Software Delivery over the last decade, but then dives into how Vibe coding impacts CI pipelines. Rob also shares his insights about running an AI-first engineering organization at CircleCI. Listen to learn more!  Check out our website at https://kubernetesbytes.com/   Show Notes: https://www.linkedin.com/in/robzuber/ https://circleci.com/ https://circleci.com/blog/

See All (100)

Trailer

5
out of 5
13 Ratings

About

Kubernetes Bytes is a podcast bringing you the latest from the world of cloud native data management. Hosts Ryan Wallner and Bhavin Shah come to you from Boston, Massachusetts with experienced backgrounds in cloud-native tech. They'll be sharing their thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud-native ecosystem.

Information

  • Creator
    Ryan Wallner & Bhavin Shah
  • Years Active
    2021 - 2026
  • Episodes
    100
  • Rating
    Clean
  • Copyright
    © Ryan Wallner & Bhavin Shah
  • Show Website
    Kubernetes Bytes

You Might Also Like