Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News

Teller's Tech - DevOps, SRE and Cloud Podcast

Ship It Weekly is a short, practical recap of what actually matters in DevOps, SRE, cloud infrastructure, and platform engineering. Each episode, your host Brian Teller walks through the latest outages, releases, tools, and incident writeups, then translates them into “here’s what this means for your systems” instead of just reading headlines. Expect a couple of main stories with context, a quick hit of tools or releases worth bookmarking, and the occasional segment on on-call, burnout, or team culture. This isn’t a certification prep show or a lab walkthrough. It’s aimed at people who are already working in the space and want to stay sharp without scrolling status pages, cloud updates, and blogs all week. You’ll hear about things like cloud provider incidents, Kubernetes and platform trends, Terraform and infrastructure changes, and real postmortems that are actually worth your time. Most episodes are 15–30 minutes, so you can catch up on the way to work or between meetings. Every now and then there will be a “special” focused on a big outage or a specific theme, but the default format is simple: what happened, why it matters, and what you might want to do about it in your own environment. If you’re the person people DM when something is broken in prod, or you’re building the cloud and platform everyone else ships on top of, Ship It Weekly is meant to be in your rotation.

  1. 1 天前

    Railway US East Outage, Stripe’s Graph-Based Database Recovery, Kata Containers Host Escape, DynamoDB Vector Search, AWS Network Firewall Proxy, Gateway API 1.6, and containerd 2.4

    This week on Ship It Weekly: Railway explains how an upstream network problem turned into a much larger US East outage, including storage traffic falling back onto the management network and stale connections continuing to cause problems after routing recovered. Stripe shares how graph search and state machines helped cut database pager volume by about 30 percent. Kata Containers patches a critical guest-to-host escape, and DynamoDB adds native vector search. The bigger theme this week is what happens after the obvious failure. Fixing the route does not necessarily clear the connections created while it was broken. Automating recovery does not have to mean handing an AI agent unrestricted production access. And stronger isolation does not eliminate the components that still cross the guest-host boundary. In the lightning round: AWS brings explicit forward proxy functionality back through Network Firewall, Gateway API 1.6 moves TCPRoute and UDPRoute to stable, and containerd 2.4 enters beta with new functionality alongside breaking changes worth finding before your next runtime upgrade. Links Railway: July 2, 2026 US East Services Outage https://blog.railway.com/p/incident-report-july-2-2026-us-east-services-outage Stripe: How Stripe uses graph search and state machines to auto-remediate a global database fleet https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet Kata Containers: Guest-root to host-root escape via virtiofs https://github.com/kata-containers/kata-containers/security/advisories/GHSA-2gv2-cffp-j227 Amazon DynamoDB now supports real-time vector search https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/ AWS Network Firewall forward proxy preview https://aws.amazon.com/about-aws/whats-new/2026/08/aws-network-firewall-forward-proxy-preview/ Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard https://kubernetes.io/blog/2026/08/03/gateway-api-v1-6-release/ containerd 2.4 beta https://github.com/containerd/containerd/releases CNCF: Learning Cloud-Native Engineering Beyond Tutorials Through LFX https://www.cncf.io/blog/2026/08/10/learning-cloud-native-engineering-beyond-tutorials-through-lfx/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W33/ Ship It Weekly https://shipitweekly.fm/

    Railway US East Outage, Stripe’s Graph-Based Database Recovery, Kata Containers Host Escape, DynamoDB Vector Search, AWS Network Firewall Proxy, Gateway API 1.6, and containerd 2.4
  2. 8月7日

    AI Agents Target GitHub, Kubernetes 1.37 Deprecations, AWS Transit Gateway Policy Routing, IAM Identity Center Multi-Region, Cloudflare Meerkat, AMOS Mac Malware, and the Risk of Half-Migrated Production Systems

    This week on Ship It Weekly: AI agents from Anthropic and OpenAI took unsanctioned actions on the real internet during UK government cyber testing, including an attempt to push malicious code into a real GitHub project. Kubernetes 1.37 starts retiring IPVS mode, pushes cgroup v1 closer to removal, and brings an SELinux volume change worth testing before upgrades. AWS Transit Gateway gets policy-based routing, and IAM Identity Center expands multi-Region support to organizations using AWS’s built-in directory. The bigger theme: access is not the same thing as authority, and availability is not just about whether your application is running. Agents need boundaries around the actions they can take. Routing policies need enough visibility to explain why traffic went where it did. And regional resilience does not help much if the people responding to the outage cannot authenticate. In the lightning round: Cloudflare’s Meerkat consensus system, AMOS macOS malware, N-able’s incomplete N-central fix, and an AWS CLI bug that disabled SSH host-key verification. Links AISI: Unsanctioned agent behaviour during cyber testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing Kubernetes v1.37 Sneak Peek https://kubernetes.io/blog/2026/07/31/kubernetes-v1-37-sneak-peek/ AWS Transit Gateway Policy-Based Routing https://aws.amazon.com/about-aws/whats-new/2026/07/aws-transit-gateway-policy-based-routing/ AWS IAM Identity Center multi-Region directory support https://aws.amazon.com/about-aws/whats-new/2026/07/aws-iam-identity-center-extends-multi-region-support-to-identity-center-directory Cloudflare: Introducing Meerkat https://blog.cloudflare.com/meerkat-introduction/ Atomic macOS / AMOS stealer infection https://isc.sans.edu/diary/rss/33208 N-able N-central exploitation after incomplete fix https://thehackernews.com/2026/08/n-able-says-attackers-take-over-n.html CVE-2026-18654: AWS CLI EMR SSH host-key verification https://aws.amazon.com/security/security-bulletins/rss/2026-071-aws/ CloudFront VPC Origins half-migrated incident https://www.reddit.com/r/devops/comments/1vdovj6/the_cloudfront_vpc_origins_outage_caught_me/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W32/ More episodes https://shipitweekly.fm/

    AI Agents Target GitHub, Kubernetes 1.37 Deprecations, AWS Transit Gateway Policy Routing, IAM Identity Center Multi-Region, Cloudflare Meerkat, AMOS Mac Malware, and the Risk of Half-Migrated Production Systems
  3. 7月31日

    Telstra’s Time Sync Outage, DoorDash’s 1.5M RPS Cache, Stateless MCP, GitHub and PyPI Supply Chain Delays, and Why the Quietest Infrastructure Often Has the Biggest Blast Radius

    This week on Ship It Weekly: Telstra’s mobile network jumped back to 2006 after a timing device restarted with the wrong date, disrupting calls, data sessions, and hundreds of emergency calls. DoorDash explains how Entity Cache, built with Envoy and Valkey, handles more than 1.5 million requests per second and uses stale-data policies, invalidation, and fallback behavior as a reliability layer. The latest MCP release candidate removes protocol-level sessions, making servers easier to scale behind ordinary load balancers while leaving teams responsible for authentication, tracing, retries, rate limits, and application state. GitHub and PyPI are also adding friction to package automation. Dependabot now delays routine updates by three days, while PyPI blocks new files from releases older than 14 days. Links Telstra outage https://www.telstra.com.au/exchange/our-mobile-network-outage-has-been-resolved-heres-what-happened DoorDash Entity Cache https://careersatdoordash.com/blog/high-performance-proxy-cache-for-doordash-services/ MCP specification release candidate https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/ Dependabot package cooldown https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ PyPI release-file restrictions https://blog.pypi.org/posts/2026-07-22-releases-now-reject-new-files-after-14-days/ Amazon ECS Action Logs https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-ecs-action-logs/ Network Load Balancer listener rules https://aws.amazon.com/about-aws/whats-new/2026/07/aws-network-load-balancer-supports-listener-rules/ Amazon Managed Prometheus limits https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-managed-service-prometheus-1500m-metrics-workspace/ PixelSmash in FFmpeg https://jfrog.com/blog/pixelsmash-critical-ffmpeg-vulnerability-turns-media-files-into-weapons/ SRE Weekly Issue 527 https://sreweekly.com/sre-weekly-issue-527/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W31/ More episodes https://shipitweekly.fm/

    Telstra’s Time Sync Outage, DoorDash’s 1.5M RPS Cache, Stateless MCP, GitHub and PyPI Supply Chain Delays, and Why the Quietest Infrastructure Often Has the Biggest Blast Radius
  4. 7月27日

    Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production

    This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Jay Lark of Hookbridge about webhook reliability, retries, idempotency, replay, security, local development, and what happens when a simple HTTP POST becomes production infrastructure. Jay is a Principal DevOps Engineer and the founder of Hookbridge, a service focused on making webhook delivery more reliable and easier to operate. We start with the basic model: one service sends an HTTP POST to another when something happens. The happy path is easy. The problems begin when an endpoint is unavailable, responds too slowly, receives duplicate events, or gets them out of order. Jay explains what “at least once delivery” means and why receivers must expect duplicates. We talk about idempotency, event IDs, retry behavior, availability during deployments, and why returning a 200 does not prove downstream processing succeeded. We also dig into observability and security. Teams need enough visibility to know whether a webhook arrived, whether signature verification passed, what response was returned, and where processing failed. Jay breaks down HMAC signatures, timestamps, replay protection, and why a valid signature still does not replace normal business-logic validation. Local development is another source of friction. External providers cannot send events directly to localhost, so developers often rely on temporary tunnels, staging deployments, copied payloads, or mocks. Jay explains how Hookbridge uses a fixed URL and local client to forward real webhook traffic to a developer’s machine. We also talk about n8n, self-hosted OpenClaw systems, Hookbridge pull endpoints, and when polling may be simpler or safer than exposing another inbound endpoint. The big takeaway: design the failure path before a webhook becomes business-critical. Verify the sender, expect duplicate and out-of-order events, build enough visibility to debug failures, and have a replay strategy before the first incident. Highlights • Why webhooks are harder than “just an HTTP POST” • What at-least-once delivery means for receivers • Why idempotency, retries, and event ordering matter • What teams need for webhook observability and debugging • Where HMAC signatures, timestamps, and replay protection fit • Why local webhook development is still awkward • When polling or pull-based delivery may be a better fit • When teams should stop building webhook infrastructure themselves Links Hookbridge: https://hookbridge.io Hookbridge local development CLI: https://www.hookbridge.io/cli.html Hookbridge pull endpoints: https://www.hookbridge.io/pull.html Jay Lark on LinkedIn: https://www.linkedin.com/in/jay-lark-ba7a3b5/ OpenClaw: https://openclaw.ai More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com

    Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production
  5. 7月24日

    AWS CloudFormation Express Mode, Spark 4.2 Vector Search, AI Speeds Coding but Not Delivery, Platform Governance Developers Won’t Hate, and Why Could Is Not the Same as Should

    This week on Ship It Weekly: AWS CloudFormation Express mode promises faster infrastructure feedback by reporting deployments complete before extended resource stabilization finishes. Apache Spark 4.2 adds native vector operations and nearest-neighbor joins, giving some teams a way to keep AI data workloads closer to the platforms they already run. GitLab’s latest research says AI is helping developers generate and commit code faster, but review, testing, governance, and deployment are not accelerating at the same pace. Then, a platform engineering case study from Sevdesk shows how minimum viable governance, useful feedback, and progressive enforcement can improve compliance without turning the platform team into another approval queue. The theme this week: making one part of the system faster does not automatically improve the whole system. In the lightning round: OpenShift 4.22.5 receives an important security update, outdated autoscaling thresholds disrupt GitHub Actions, Cloudflare experiences an incident where some POST requests fail to reach customer origins, and GitHub Code Quality becomes generally available—with automatic billing attached. The episode closes with Reid Savage reflecting on their first year managing Honeycomb’s SRE team and the difference between what a capable team could do and what it should do. Links CloudFormation Express mode https://aws.amazon.com/blogs/aws/accelerate-your-infrastructure-deployments-by-up-to-4x-with-aws-cloudformation-express-mode/ Apache Spark 4.2 https://spark.apache.org/releases/spark-release-4-2-0.html GitLab AI Accountability Report https://about.gitlab.com/resources/ai-accountability-survey-2026/ Platform governance at Sevdesk https://www.infoq.com/presentations/platform-engineering-team-compliance/ OpenShift 4.22.5 security update https://access.redhat.com/errata/RHSA-2026%3A37585 GitHub Actions incident https://github.com/orgs/community/discussions/201795 Cloudflare status https://www.cloudflarestatus.com/ GitHub Code Quality GA https://github.blog/changelog/2026-07-20-github-code-quality-is-now-generally-available/ Could vs. Should — Reid Savage https://www.honeycomb.io/blog/could-should-first-year-managing-sre-team This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W30/ More episodes and full show notes https://shipitweekly.fm/

    AWS CloudFormation Express Mode, Spark 4.2 Vector Search, AI Speeds Coding but Not Delivery, Platform Governance Developers Won’t Hate, and Why Could Is Not the Same as Should
  6. 7月20日

    Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

    This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Mat Ryer of Grafana Labs about AI observability, production agents, evals, telemetry cost, guardrails, and what changes once AI moves beyond demos and into systems teams actually depend on. Mat is Senior Director of AI at Grafana Labs, where he focuses on how AI fits into observability and production systems. We talk about Grafana Assistant, why AI observability is not just logs, latency, and HTTP 200s, and how teams can measure whether agents are actually helping. Mat gets into evals, LLM-as-judge patterns, traces as a way to think about conversations, user feedback, tool choice, model changes, and the cost of collecting new telemetry. We also dig into UX and trust. If an AI assistant gives you a wall of text, you still have to decide whether to believe it. If it can show the graph, deep link into Grafana, apply filters, and expose the source data, that becomes a much more useful operating experience. The big takeaway: start small, enhance workflows you already have, build feedback loops, and treat production AI like something you actually have to operate. Highlights • Why AI demos are easy, but production AI is harder • Why agents need observability, evals, and guardrails • How Grafana thinks about AI Assistant and AI observability • Where LLM-as-judge patterns, traces, tool calls, and feedback fit • Why telemetry cost problems may repeat with AI workloads • Why UX matters when operators need to trust the answer • Where AI can help SRE and platform teams today Links Mat Ryer on LinkedIn: https://www.linkedin.com/in/matryer/ Mat Ryer on GitHub: https://github.com/matryer Grafana Labs: https://grafana.com Grafana Assistant: https://grafana.com/products/cloud/ai-assistant/ Grafana AI Observability: https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/ Grafana Adaptive Telemetry: https://grafana.com/products/cloud/adaptive-telemetry/ Grafana MCP server: https://github.com/grafana/mcp-grafana OpenTelemetry: https://opentelemetry.io Prometheus: https://prometheus.io Grafana Loki: https://grafana.com/docs/loki/latest/ More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com

    Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production
  7. 7月18日

    GitHub API Enumeration, Grok Build CLI Data Exposure, AWS Security Hub Network Scanning, AI-Powered Patch Pressure, and Why Visibility Is Not Ownership

    This week on Ship It Weekly: Datadog tracked coordinated GitHub API enumeration, xAI’s Grok Build CLI reportedly uploaded repo data without redaction, AWS Security Hub added Network Scanning and exposure impact analysis, and Microsoft says AI-powered vulnerability discovery is changing patch pressure. The theme: visibility is not ownership. A GitHub API map does not revoke a token. An exposure finding does not close a port. A patch bulletin does not patch the fleet. And an AI coding tool reading your repo is still access. Brian covers GitHub as a production surface, AI coding tool data boundaries, cloud exposure based on reachability and blast radius, and why patching needs to look more like production operations than spreadsheet theater. Also, the Ship It Weekly shop is open at shop.tellerstech.com with Ship It Weekly t-shirt designs. Use coupon code SHIPTHESTORE for 20% off your order for the next few weeks. Links Datadog: Coordinated GitHub API enumeration https://securitylabs.datadoghq.com/articles/coordinated-github-api-enumeration/ The Verge: Grok Build CLI repository upload report https://www.theverge.com/ai-artificial-intelligence/965600/spacexai-grok-build-repository-upload AWS Security Hub Network Scanning https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-network-scanning/ AWS Security Hub impact analysis for exposure findings https://aws.amazon.com/about-aws/whats-new/2026/07/impact-analysis-aws-security-hub/ Microsoft: Windows vulnerability management and AI-powered discovery https://blogs.windows.com/windowsexperience/2026/07/09/evolving-windows-vulnerability-management-to-meet-the-speed-of-ai-powered-discovery/ SRE Weekly Issue 525 https://sreweekly.com/sre-weekly-issue-525/ HalluSquatting / hallucinated package risk https://www.endorlabs.com/learn/slopsquatting-when-ai-agents-hallucinate-malicious-packages AWS Lambda Managed Instances for Java cold starts https://aws.amazon.com/blogs/compute/eliminating-java-cold-starts-with-aws-lambda-managed-instances/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W29/ Ship It Weekly shop https://shop.tellerstech.com/ More episodes and full show notes https://shipitweekly.fm/

    GitHub API Enumeration, Grok Build CLI Data Exposure, AWS Security Hub Network Scanning, AI-Powered Patch Pressure, and Why Visibility Is Not Ownership
  8. 7月10日

    EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

    This week on Ship It Weekly: Amazon EKS added Kubernetes version rollbacks, Novee Security published Cordyceps research on GitHub Actions supply chain risk, Tenet Security showed how fake telemetry can hijack AI coding agents, and Amazon CloudWatch added alarms directly from log queries. The theme: safety nets are getting better, but the blast radius is getting wider. Rollback buttons, log alarms, zone-aware routing, secret scanning, and AI agent workflows all help, but they do not replace ownership. Brian covers why EKS rollbacks are useful but not a substitute for real upgrade discipline, why GitHub Actions YAML is production code with credentials, how fake Sentry telemetry can become hostile agent context, and why easier log-based alarms can also mean easier pager noise. In the lightning round: ECS Service Connect zone-aware routing, etcd 3.7, GitHub innersource advisories, secret scanning metadata improvements, and CloudWatch Application Signals service events. Links Amazon EKS Kubernetes version rollbacks https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/ Novee Security: Cordyceps supply chain research https://novee.security/blog/cordyceps/ Tenet Security: Agentjacking through fake Sentry errors https://tenetsecurity.ai/blog/agentjacking-coding-agents-with-fake-sentry-errors/ Amazon CloudWatch log query alarms https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-cloudwatch-log-alarms/ ECS Service Connect zone-aware routing https://aws.amazon.com/about-aws/whats-new/2026/07/ecs-service-connect-zone-aware/ etcd 3.7 announcement https://etcd.io/blog/2026/announcing-etcd-3.7/ GitHub innersource security advisories https://github.blog/changelog/2026-07-08-innersource-security-advisories-are-generally-available/ GitHub secret scanning extended metadata and multipart validation https://github.blog/changelog/2026-07-07-secret-scanning-extended-metadata-and-multipart-validation/ CloudWatch Application Signals service events https://aws.amazon.com/about-aws/whats-new/2026/06/cloudwatch-service-events/ Our Links This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W28/ More episodes and full show notes https://shipitweekly.fm/

    EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

簡介

Ship It Weekly is a short, practical recap of what actually matters in DevOps, SRE, cloud infrastructure, and platform engineering. Each episode, your host Brian Teller walks through the latest outages, releases, tools, and incident writeups, then translates them into “here’s what this means for your systems” instead of just reading headlines. Expect a couple of main stories with context, a quick hit of tools or releases worth bookmarking, and the occasional segment on on-call, burnout, or team culture. This isn’t a certification prep show or a lab walkthrough. It’s aimed at people who are already working in the space and want to stay sharp without scrolling status pages, cloud updates, and blogs all week. You’ll hear about things like cloud provider incidents, Kubernetes and platform trends, Terraform and infrastructure changes, and real postmortems that are actually worth your time. Most episodes are 15–30 minutes, so you can catch up on the way to work or between meetings. Every now and then there will be a “special” focused on a big outage or a specific theme, but the default format is simple: what happened, why it matters, and what you might want to do about it in your own environment. If you’re the person people DM when something is broken in prod, or you’re building the cloud and platform everyone else ships on top of, Ship It Weekly is meant to be in your rotation.

你可能也會喜歡