Automatic

Eric Lamanna

Podcast for Automatic.co and LLM.co, the AI automation specialists.

  1. -14 h

    Private vs. Public LLMs: What Every CTO Needs to Know

    For CTOs navigating today's AI landscape, few decisions carry more long-term weight than choosing between public and private large language models. This episode of Automatic unpacks that choice in full — moving beyond the hype to examine the real trade-offs across data governance, product performance, and total cost of ownership. The analysis draws on the full private vs. public LLMs breakdown to give technical leaders a practical framework they can actually act on. Here's what the episode covers: Public LLMs: the appeal and the catch. Vendor-hosted models like GPT-4, Gemini, and Claude offer instant capability and zero infrastructure overhead — but your data transits someone else's servers, and fine-tuning options can be limited or non-compliant in regulated environments. Private deployments: control at a cost. Hosting open-weights models like Llama 3 or Mistral on your own infrastructure gives you full data sovereignty, custom fine-tuning, and auditable access logs — but demands serious investment in GPU compute, ML talent, and ongoing maintenance. The three axes that actually drive the decision: data governance and regulatory exposure, product fit and performance requirements, and a 24-month total cost of ownership model that accounts for far more than token pricing. The break-even reality. Research suggests the economics typically flip somewhere between 40 and 150 million tokens per month — below that threshold, public APIs usually win; above it, private infrastructure starts making balance-sheet sense. The hybrid playbook. A growing number of sophisticated organizations are routing low-sensitivity tasks through public APIs while keeping proprietary and regulated data on private models — with orchestration tools handling automatic classification and routing. Edge and on-device models as a third path. Small-parameter models running locally are unlocking new options for mobile and field applications where cloud latency is a constraint and privacy requirements are strict. The core argument: this isn't a binary choice but a portfolio decision. The most effective approach blends public APIs for general-purpose work, private models for sensitive and proprietary data, and edge deployments for offline resilience — with governance and observability baked in from day one, not retrofitted later. If you enjoyed this episode, check out Real-Time Joins: Making SQL Cry for another deep dive into the infrastructure decisions that quietly define how modern systems perform. LLM

  2. -1 j

    Real-Time Joins: Making SQL Cry

    Stream processing looks deceptively approachable right up until you try to join two live data streams. Unlike static database tables, streaming data arrives continuously, out of order, and never quite "done" — and that fundamental reality exposes the limits of traditional SQL thinking fast. This episode of Automatic digs into the engineering challenges behind real-time joins, why they trip up even experienced teams, and what a disciplined approach to building them actually looks like in production. The episode covers the full arc of designing, operating, and surviving real-time joins at scale, including: Why SQL struggles with streams — static query planners were built for complete, bounded datasets, not infinite flows with late-arriving and out-of-order events.Keys, windows, and watermarks — the three core decisions that define the behavior of any real-time join, and why getting each one wrong has outsized downstream consequences.Streaming vs. microbatch trade-offs — how each model handles state and windowing differently, and why understanding which one you're operating in changes your failure modes entirely.State management and data skew — the hidden operational costs of holding partial matches in memory, how hot join keys can silently degrade performance, and practical remedies like salted partitioning.When to skip the live join altogether — making the case for precomputed, denormalized views when relationships are stable, and reserving real-time joins for truly time-sensitive logic.Observability, governance, and on-call culture — why real-time systems demand metrics that tell a story, privacy-conscious key design, idempotent outputs, and an engineering mindset comfortable reasoning in windows and watermarks.The episode also tackles the less glamorous but critical operational details: throttling backfills to avoid collisions with live traffic, schema validation at ingress, versioned topics, and the importance of testing restore procedures against realistic data cardinality rather than toy examples. The recurring theme is that real-time joins reward deliberate, unglamorous engineering far more than clever tricks. For more from the show, check out LLMs Behind Closed Doors: Building Secure, In-House AI Models, which explores another high-stakes infrastructure challenge — keeping powerful AI models inside your own walls. Automatic

  3. -2 j

    LLMs Behind Closed Doors: Building Secure, In-House AI Models

    Data sovereignty, regulatory compliance, and competitive secrecy are forcing a rethink of how enterprises deploy large language models. This episode of Automatic explores the full picture of building secure, in-house AI models — from the strategic rationale to the infrastructure decisions to the day-to-day operational discipline that keeps a private LLM deployment trustworthy over time. Here's what the episode covers: Why enterprises are pulling workloads off public cloud: regulatory frameworks like GDPR, HIPAA, and PCI-DSS create hard constraints on data handling that vendor APIs struggle to satisfy cleanly.The competitive-intelligence risk: crown-jewel assets — contract drafts, R&D documents, strategic plans — flowing through shared, multi-tenant models carry a non-zero risk of unintended exposure.Latency as a business case: on-premises inference can cut round-trip query times from ~420 ms to ~45 ms, with even steeper gains on RAG-augmented and batch workloads.Layered zero-trust security: network segmentation, fine-grained role-based access control, immutable audit logs, and end-to-end encryption each compound one another to meaningfully shrink the residual attack surface.Infrastructure tradeoffs: pure on-premises, colocation, and private cloud tenancy each carry distinct CapEx-versus-OpEx profiles — the right choice depends on regulatory environment, budget structure, and how quickly AI ambitions are scaling.Ongoing operational discipline: differential privacy during fine-tuning, automated red-team testing, real-time guardrails for PII and policy violations, and post-mortem feedback loops that make the system smarter about its own failure modes.The episode makes clear that hosting a model behind the firewall is only the starting point — the organizations that do this well treat private LLM deployment as a continuous, layered engineering commitment, not a one-time infrastructure decision. The payoff is genuine: the freedom to push these models toward their full capability without routing sensitive data through a third-party black box. More from the show: if you enjoy episodes on the architectural tensions that shape modern data systems, check out ACID vs. BASE: The Database Cold War, which covers another foundational tradeoff every engineering team eventually has to navigate. LLM

  4. -3 j

    ACID vs. BASE: The Database Cold War

    Every digital system quietly takes sides in a debate most users never see: should the database prioritize ironclad correctness, or resilient availability? This episode of Automatic tackles that tension head-on, drawing on the original deep-dive article on ACID vs. BASE database philosophy to unpack why both models exist, what each one costs, and how mature engineering teams navigate the choice. Here's what the episode covers: ACID unpacked — how Atomicity, Consistency, Isolation, and Durability work together to make transactions auditable and correct, and why that discipline carries real coordination overhead in distributed systems.BASE explained honestly — why Basically Available, Soft State, and Eventual Consistency isn't sloppiness but a deliberate acknowledgment of what distributed networks can physically guarantee.The cold war metaphor — how each camp has built up its own arsenal of deterrents (serializable isolation and write-ahead logs on one side; quorum reads, vector clocks, and CRDTs on the other), and why the stalemate is actually productive.Bridging patterns — practical techniques like Sagas for distributed transaction recovery and idempotent operation design that help teams get the benefits of both models without betting everything on one.Polyglot persistence — the case for pairing an ACID core (handling money, identity, and inventory) with BASE edges (serving search, recommendations, and metrics), and why clear documentation makes or breaks that architecture.Labeling data by sensitivity — a framing for deciding where to spend strictness and where to spend speed, so the whole team knows the rules without having to memorize the database internals.The episode closes with a reminder that neither model is a religion. The real differentiator isn't which flag an engineering team salutes — it's whether they've written down their non-negotiables, mapped those requirements to the right model, and built the observability to know when the promises are breaking. Technology follows culture, and the habits determine the outcome far more than the tooling does. More from the show: if questions about infrastructure control and compliance are on your mind, check out The Rise of On-Prem LLMs: Control, Compliance, and Customization for a related look at how those themes play out in the AI layer. Automatic

  5. -4 j

    The Rise of On-Prem LLMs: Control, Compliance, and Customization

    For many organizations, the appeal of cloud-based AI is hard to argue with — until a compliance audit, a data classification question, or a runaway API bill forces a reckoning. This episode of Automatic examines why on-premises large language models are moving from niche workaround to mainstream enterprise strategy, drawing on this in-depth look at on-prem LLM adoption trends. The result is a clear-eyed look at who should be considering the shift, what it actually takes to execute, and where the real trade-offs live. The episode works through the full landscape of on-prem AI deployment in 2026 — from what "on-premises" even means today to the operational discipline required to make it stick. Key topics include: Data sovereignty redefined: Why "on-prem" now spans private co-location cages, GPU appliances, and hybrid architectures — not just basement server racks — and what stays constant across all of them.Four compounding drivers: Direct control over sensitive data, regulatory compliance simplicity, deeper model customization, and long-term economics that favor ownership at scale.The real cost math: How a high-volume inference workload can hit seven figures annually in cloud fees — and why the three-year total cost of ownership for a comparable on-prem cluster often breaks even before year two.Latency as a business metric: What a drop from ~300ms to ~40ms round-trip actually means for voice assistants, fraud detection, and user retention — and why it rarely appears in a CapEx vs. OpEx spreadsheet.Hardware and software foundations: GPU sizing guidelines for models ranging from 7B to 70B parameters, recommended OS and container tooling, and what a mature LLMOps observability stack looks like.A phased rollout approach: Starting with a scoped audit and four-GPU pilot, hardening the environment, scaling horizontally with RAG pipelines and CI/CD fine-tuning, and closing the feedback loop through annotation-driven model improvement.The episode also calls out practical pitfalls — underestimated cooling loads in high-density GPU racks, underused prompt optimization techniques that can cut inference costs by 30–40%, and the maintenance debt that accumulates from one-off model forks. And it closes by reframing the decision not as cloud versus on-prem, but as a deployment spectrum where workloads should be placed based on their actual risk profile and latency requirements. More from the show: if you enjoy episodes about elegant infrastructure under the hood, check out Bloom Filters: The Tiny Data Structure With a Big Job. LLM

  6. -5 j

    Bloom Filters: The Tiny Data Structure With a Big Job

    Speed and efficiency don't always come from adding more power — sometimes they come from adding a smarter gatekeeper. This episode of Automatic dives into Bloom filters, the compact probabilistic data structures quietly doing heavy lifting inside search engines, streaming platforms, and distributed databases. Drawing on this in-depth article on Bloom filters and their real-world impact, the episode unpacks the elegant tradeoff at the heart of the design: no false negatives, ever — and only a small, tunable chance of false positives. Here's what the episode covers: The core mechanic: How a bit array and a handful of hash functions combine to answer "is this item possibly in the set?" in constant time, with minimal memory overhead.The asymmetry that makes it useful: Why guaranteed negatives are so valuable — and why a small false-positive rate is an acceptable, intentional tradeoff rather than a flaw.Sizing and lifecycle management: How to calculate the right array size and hash function count for a target error rate, and why filters need planned rotation before they saturate.Variants worth knowing: Counting Bloom filters (which support deletion) and cuckoo filters (which offer compact fingerprint storage), plus when the classic design is still the right default.Where they show up in production: Guarding databases from empty-key disk reads, protecting caches from pollution, deduplicating event streams, and shielding rate limiters — anywhere wasted lookups add up.Pitfalls to plan for: Validating false-positive maybes downstream, normalizing inputs before hashing, communicating error rates clearly to teammates, and monitoring filter health over time.The episode also touches on the cost dimension — how cutting wasted operations across many layers compounds into real savings — and the broader design philosophy Bloom filters embody: restraint, predictability, and knowing exactly which kind of wrong you can afford to be. More from the show: if you're thinking about data and privacy, check out the episode Your ChatGPT Chats May Not Be as Private as You Think. Automatic

  7. -6 j

    Your ChatGPT Chats May Not Be as Private as You Think

    Most people treat AI chat windows like a private notebook — a safe place to think out loud, troubleshoot problems, and process sensitive information at speed. But this deep-dive on ChatGPT data privacy risks tells a more complicated story. From the moment you hit send, your input begins a journey through servers, logs, and potentially human review queues that most users never consider. This episode of Automatic maps that journey and explains what you can do to stay protected without abandoning the productivity gains these tools genuinely offer. Here's what the episode covers: How chat data actually travels: Why your message doesn't stay on your device — and the chain from volatile memory to application logs to long-term training archives.Real-world breaches, not hypotheticals: The 2023 Redis misconfiguration that exposed active conversation titles and partial transcripts, and the prompt-echo exploits that occasionally surface other users' inputs.The insider access reality: Human reviewers are a legitimate part of how AI systems improve — which also means real staff can encounter real, unredacted user content.The data people accidentally share: A rundown of the sensitive categories that slip into chat windows almost by reflex — API keys, patient notes, financial credentials, proprietary documents, and more.Practical countermeasures for individuals and teams: The "public bulletin board" mental test, data masking and tokenization strategies, when to consider local or self-hosted models, and why organizational policies need regular auditing as vendor defaults shift.The regulatory horizon: How GDPR, CCPA, and HIPAA are nudging AI providers toward data minimization — and why, for now, the burden of protection still sits largely with the user.The core takeaway the episode leaves you with: generative AI treats your input as raw material, and that's exactly what makes it powerful. But until privacy by design is an industry default rather than a premium add-on, every chat window deserves the same caution you'd apply to speaking in a crowded café. The tools are worth using — just worth using with open eyes. For more on the evolving relationship between AI capability and responsible practice, don't miss the earlier episode Fine-Tuning LLMs: Brilliance or Burnout. LLMVDR.ai

  8. 4 août

    Fine-Tuning LLMs: Brilliance or Burnout

    Fine-tuning a large language model is one of those ideas that sounds like an obvious win the moment it's proposed — and then reveals its full complexity once the work begins. This episode of Automatic unpacks the real trade-offs behind fine-tuning LLMs, helping teams understand not just what the technique can achieve, but what it quietly demands in return. Whether you're weighing a first training run or revisiting a model that's already gone stale, this episode offers a grounded framework for making the call. Here's what the episode covers: What fine-tuning actually delivers: How steering a pre-trained model on curated examples shapes tone, vocabulary, formatting habits, and domain-specific behavior in ways that build genuine user trust.The data quality trap: Why inconsistent or messy training examples don't get resolved by the model — they get averaged into unpredictable outputs that satisfy no one's original intent.The hidden cost of ongoing maintenance: Fine-tuned models aren't a one-time investment; policies change, products evolve, and without scheduled refresh cycles, yesterday's expert becomes today's liability.Drift, confidence, and safety: The same fluent confidence you trained the model to project can become a problem when it's confidently wrong — and why safety checks, escalation paths, and graceful uncertainty still need to be built in explicitly.Privacy and data governance: Why every training example should be treated as if it could be reviewed by a regulator or the customer it came from, and what that means for consent, redaction, and audit trails.When to reach for other tools first: The episode makes a clear case for exhausting prompt engineering, tool use, and retrieval-augmented generation before committing to a full training pipeline — and offers concrete questions to determine genuine readiness.The throughline is practical: fine-tuning shines for stable domains, repeating tasks, and teams with real feedback loops and clear success metrics. It invites burnout when it's treated as a shortcut rather than a discipline. For more on the strategic case for keeping models in controlled environments, check out the episode Why Private LLMs Matter Far Beyond Privacy. The source article for this episode is linked above. Automatic

Tout afficher (72)

À propos

Podcast for Automatic.co and LLM.co, the AI automation specialists.

Informations

  • Création
    Eric Lamanna
  • Années d’activité
    2026
  • Épisodes
    72
  • Classification
    Tous publics
  • Copyright
    © 2026 Automatic.co
  • Site web de l’émission
    Automatic