BEAM There, Done That

Plangora

BEAM There, Done That is a podcast about building real systems with Elixir, Erlang, and the BEAM. We’ve built it before — distributed systems, fault‑tolerant services, event pipelines, real‑time apps, production nightmares, and the supervision trees that saved them. Each episode dives into practical lessons from shipping software on the BEAM: architecture decisions, scaling challenges, operational failures, and the patterns that actually work. No hype. No theory without scars. Just hard‑won experience from engineers who’ve been there.

  1. 6 days ago

    Agents on the BEAM: Mike Hostetler on Jido

    In the first version of Jido, every agent was a DynamicSupervisor. Each GenServer also kept a private queue in its own state so an action could push follow-up work to the front of the line instead of waiting in the mailbox. Mike Hostetler calls both decisions things he still cringes at, and he rebuilt the framework for 2.0 to get rid of them. Jido began as bot infrastructure before there was a model to call, and the LLM is still kept out of the core on purpose. In Jido an agent is a validated struct. The process that runs it is a separate concern, and side effects are returned as directives for the runtime to carry out. Mike's estimate is that an agent is about 20% LLM and 80% patterns the BEAM community has relied on for decades. Allen and Francesco take one support-desk scenario through the whole episode: a customer asks where their order is, and the shipping provider times out halfway through. The scenario covers retries, crashes eight tool calls into a reasoning loop, models that are confidently wrong, and 10,000 customer agents of which only 200 are in memory at any moment. Mike also covers ReqLLM, the weekend side project that now handles 11 providers and 665 models, and how Jido and Ash ended up working together instead of competing. How Jido defines an agent: data enters a stateful process, changes it, and emits a resultSignals (CloudEvents envelopes), actions, instructions and directivesThe scars from 1.0: a mailbox-bypassing queue in GenServer state and agents as DynamicSupervisorsDefining the agent turn, and why one GenServer was doing too muchThe OTP 28 priority messages experiment and where reentrancy is heading in Jido 3Why the core supports behaviour trees and FSMs alongside ReAct and tree of thoughtsHibernate and thaw: idle agents written to a persistent store after 30–60 secondsWhat happens when an LLM call times out (async task, three retries by default) and when the orchestrating process crashesHallucinations: evals, DSPy/GEPA prompt-level guardrails, and tool-level guardrails written in codeReqLLM and LLMDB: normalising inference across OpenAI, Anthropic, Gemini, Bedrock, Azure and OpenRouterAsh Jido: Ash resource actions exposed as agent tools with authorization policies intactHosting agents in Python and TypeScript: one agent per Docker container, or per 256 MB sandboxMaintaining dozens of packages with agents, GitHub Actions and optional dependenciesRecorded 9 November 2026. Resources mentioned:Jido and documentation https://jido.run/Jido AI https://jido-ai.hexdocs.pm/ReqLLM https://reqllm.hexdocs.pm/LLMDB https://llmcatalog.dev/Zoi https://github.com/ash-project/zoiJido Discord https://jido.run/discordMoatless https://moatless.store/

    Agents on the BEAM: Mike Hostetler on Jido
  2. 6 Nov

    A Declarative Language for the BEAM: Alejandro Ramallo on Bondy

    Alejandro Ramallo spent months losing sleep over NIF memory leaks that only appeared under high load and were almost impossible to reproduce. The fix in the latest Bondy Connect release: remove every NIF, go pure Erlang, and replace the storage layer with an operation-based CRDT database running causal+ consistency. A decade of running that distributed router in production, and maintaining Partisan underneath it, is the background to Bondy, the declarative language Alejandro introduced at Code BEAM a week before this recording. It targets a specific pain: business rules spread across gen_statem states and clauses until nobody can say where a rule lives. Bondy is statically typed, compiles to ordinary BEAM files and calls Erlang and Elixir directly. It combines Datalog for queries with Constraint Handling Rules for reacting to events and writing solvers. At Leapsight, rules that took around 800 lines of Erlang come down to 40–50 lines of Bondy, and a fluid-delivery scheduling solver for upstream oil and gas took one week instead of three months. Using a fleet of refrigerated trucks as the running example, Allen and Francesco work through what runs today, what is designed but not built, and what is on the roadmap. • NIF memory leaks and control-plane conflict-resolution bugs in Bondy Connect, and how both were removed• Why a library wasn't enough: rules, data and types sharing one language• Shapes: one definition that is both type and runtime validator, including cross-field invariants• Unification for sensor data that arrives in pieces• Structured, path-based errors written for humans and coding agents• The effect system: capabilities, pure guards and synthesis holes• gen_reactor, a new OTP behaviour that runs rules to a fixed point under normal supervision• How CHR semantics, propagation history and termination/confluence analysers keep rules from fighting or looping• Signals, fluents and logical time for latching alarms and queryable history (designed, not yet implemented)• Datalog relations for escalation chains that terminate even when the hierarchy loops• Semi-naive evaluation today; magic sets, cost-based optimisation and incremental view maintenance on the roadmap• Well OS, CHR-based solvers, and where coding agents help or break architecture Recorded October 30, 2026. Resources:Bondy language: https://www.bondy.io/languagePartisan, Constraint Handling Rules (Thom Frühwirth, 1991), and the upcoming Bondy public release are discussed in the episode.

    A Declarative Language for the BEAM: Alejandro Ramallo on Bondy
  3. 30 Oct

    Multiplayer games with Netcode on the BEAM

    Hernán Rivas Acosta's first Erlang game server, a football game for Cartoon Network, reached 50,000 concurrent players on one machine. The BEAM didn't stop it there. The OS file descriptor limit did, and the team chose not to raise it. The Java option they had considered would have managed a few thousand. Roughly fifteen years later, Hernán, now an architect at Erlang Solutions, has put the game loop itself on the BEAM. Spawndemonium, shown at Code BEAM Europe in Haarlem, is a top-down shooter whose simulation is written once in Rust, compiled to WebAssembly, and run both in the browser and inside an Elixir GenServer via Wasmex. Client and server can't disagree about the rules, only about time. The scar: AstroBeam's 30 Hz tick needed 33.3 ms, the Erlang timer took 33, and lag crept upward through every match. One-minute playtests never caught it. The fix was 40 ticks a second at 25 ms each, plus rewriting every per-tick constant in the game. With Allen Wyma and Francesco Cesarini, Hernán covers ticks versus timestamps, why jitter breaks more games than latency, how to fake a bad network convincingly, and why the demons in Spawndemonium act on inputs five ticks old. Topics include:• From Flash games for Cartoon Network and Nickelodeon to Erlang at 18• 50,000 TCP connections per machine, capped by file descriptors• Peer-to-peer anti-cheat built on mutual reporting• Ticks, tick rate, ping vs latency, and jitter• Authoritative servers vs peer-to-peer, and Modern Warfare 2's host advantage• BEAM processes explained through game-engine time slicing• Why browser games are stuck on TCP via WebSockets• Borrowing netcode from Gran Turismo for a multiplayer Asteroids• Delay input, apply and rewind, or meet in the middle• Clock sync using the smallest round trip• Simulating jitter with a log-normal distribution and TCP head-of-line blocking• One server tick: queue inputs, run Wasm, decode binary events, push to sockets• A custom binary protocol with Elixir pattern matching instead of JSON• Wasmex vs a Rustler NIF: a crash costs one match, not the node• Bots as laggy players, deciding moves five ticks ahead• Small universes vs MMO-scale worlds• Multiplayer first, game second Recorded October 26, 2026. Resources:Spawndemonium on GitHub: https://github.com/HernanRivasAcosta/SpawnDemoniumEpisode 15 — Ellyse Cedeno on MMO servers: https://youtu.be/4MDObD_R5E4Episode 10 — Florian Gilcher and Leandro Pereira on Rust and Elixir: https://youtu.be/w5Pl09lpSmEMat Trudel on Bandit: https://youtu.be/XXQcoz_dTE0

    Multiplayer games with Netcode on the BEAM
  4. 16 Oct

    Data Sovereignty & Local-First AI: Building Eldrim with Erik Stenman

    On Christmas Eve, the BEAM's scheduler throttled down to save power. A batch job arrived. A Christmas worth of XML took down all three failover nodes. The Klarna CEO called. Erik Stenman was already at his desk. That incident sent him deep into the BEAM runtime — and eventually led to the BEAM Book. Now he's applying the same habit of asking where resources really go to a different question: where does AI compute actually run, and who controls it? Erik Stenman (known as Happi) has been with Erlang since 1994. He wrote the BEAM Book, built Klarna's payment core, and is now building Eldrim — a sovereign AI compute router based in northern Sweden that guarantees your model calls stay within the jurisdiction you define. Topics include: the Christmas Eve scar: how scheduler throttling, a batch job, and a 20-reduction XML conversion took down all three Klarna nodes simultaneously — and why that incident eventually became the BEAM Bookthe three concepts everyone confuses: data residency (where your data sleeps), data sovereignty (who holds the keys), and local-first (keeping the primary copy close)why "my provider has an EU region" is not the same as EU sovereignty — and why an Amazon data center in Sweden is still subject to US lawthe US Cloud Act and China's national intelligence law: why both reach companies regardless of where their servers are physically locatedwhat Eldrim actually does: an OpenAI-compatible API router that enforces jurisdiction rules, fails with a 503 rather than routing quietly outside approved boundaries, and returns headers telling you exactly where and on which server your request ranopen weight models and the myth of portability — why you never get the same answer twice, and why licenses complicate things furtherdistrict heating in northern Sweden: how a data center in Haparanda can pipe waste heat under the streets instead of dumping it as a climate problemthe sustainability-as-sovereignty argument — and why where you run compute is now also an environmental decisionwhat a developer shipping Elixir and calling a model API can do todayCompanion episodes: local-first with Brooklyn Zelenka and Robert Virding; BlueSky's data plane with Chris Beck; Hologram with Bart Blast. Recorded October 2, 2026. Resources mentioned: eldrim.aihappyhacking.blog — Erik's writingThe BEAM Book

    Data Sovereignty & Local-First AI: Building Eldrim with Erik Stenman
  5. 9 Oct

    Ingela Andin on the TLS Stack Running Under Every BEAM App

    Every secure connection made from an Erlang, Elixir, or Gleam application runs through code she wrote. Most developers have never heard her name. Ingela Andin has spent more than two decades on the OTP team at Ericsson, maintaining the TLS stack — the piece of infrastructure underneath HTTPS, LDAP, SMTP, and every other secure protocol in the BEAM ecosystem. She is the first point of contact when an AI-generated vulnerability report lands in the OTP inbox. And this year, that inbox changed entirely. At the start of 2026, maintainers were still laughing at AI bug reports. Daniel Stenberg, creator of Curl, shut down his bug bounty because the reports were hallucinating junk. Then in the space of weeks, the table turned. The reports became real and they kept coming. Linus Torvalds called the kernel security list almost entirely unmanageable. The BEAM did not escape. Topics include: * what separates a vulnerability from a bug — and what separates a bug from someone doing something dumb * what Ingela's inbox looked like at the start of 2026 and what it looks like now * what a hallucinated AI vulnerability report actually looks like — the asymmetry example that sounded serious and wasn't * why the domain knowledge of the person running the scanner matters more than the scanner itself * how the OTP team triages reports: domain knowledge first, then AI agents to help narrow it down, then human verification * the full fix pipeline: from GitHub advisory to coordinated patch release across three maintained OTP versions to CNA publication * what happens when a bug fix has a bug — and why that's more embarrassing than a partial fix * property-based testing as the tool the team uses to stop bugs from coming back * the one thing Ingela wishes every developer understood before filing a report: write the reproduction case in Erlang, include the full crash report, and don't leave details out * what companies depending on OTP can actually do to help — starting with the "Help Wanted" label on GitHub Companion episodes: BEAM Security Wake-Up Call with Jonathan Machen and Peter Ullrich; Bruce Tate on AI, juniors, and coding agents. Recorded October 1, 2026.

    Ingela Andin on the TLS Stack Running Under Every BEAM App
  6. 2 Oct

    Alex Koutmos on Elixir for Finance, Trading & Blockchain

    Brex runs on Elixir. Klarna runs on Erlang. Mastercard is believed to use it. And most of the community that builds on these systems watches them with PromEx - a library written by today's guest. Alex Koutmos has been in the Elixir ecosystem for over a decade. He wrote PromEx - the Prometheus metrics library most of this audience runs in production - and he's just written Elixir for Finance with his brother Dimitrios, a tenured finance professor at Texas A&M. The book takes Livebook, Explorer, NX, and Scholar and points them straight at markets, trading strategies, and blockchain infrastructure. Topics include: why switching money and routing a stock order are the same problem as switching a phone call - and why the BEAM was built for both the ISO 8583 protocol story: 100 lines of C per message frame, dropped to 10 in pure Erlang, dropped to 4 with the bit syntax - and why nobody talks about this Explorer as an Elixir wrapper around Polars (a Rust data frame library) - why the BEAM delegates the heavy compute and keeps the orchestration how to use Livebook, NX, and Scholar to backtest a moving average crossover trading strategy without leaving Elixir why "let it crash" doesn't mean "let the money vanish" - and exactly how a ledger, Oban, and durable event storage keep the truth safe across process restarts the difference between a ledger and a database - and why a bank built on plain Postgres misses the point blockchain as a distributed ledger: the same distributed state problem the BEAM community was chewing on long before Bitcoin existed why fintech companies use Erlang and Elixir as a competitive advantage - and tell nobody BlockScout, Aeternity, and where BEAM languages already run in blockchain infrastructure what the Python and JVM financial tooling stacks miss that Explorer and NX cover - and where they still win Recorded August/September 2026. Resources mentioned: Elixir for Finance - the book PromEx on Hex BlockScout - open source Ethereum explorer written in Elixir Aeternity - blockchain built in Erlang

    Alex Koutmos on Elixir for Finance, Trading & Blockchain
  7. 25 Sept

    Kip Cole on Time, Localization and Building Agenda

    The Navajo Nation inside Arizona observes daylight saving. The Hopi reservation inside the Navajo Nation skips it again. There's a corner of the Yucatan Peninsula that takes 20 minutes to drive through and costs you an hour on your sat nav. No compiler warns you about any of this. Kip Cole has spent 10 years building the Elixir libraries that handle what software gets wrong about human reality: localization, time, and scheduling. CLDR/ex_cldr gave the ecosystem 173,000 Unicode characters and locale-aware formatting for numbers, dates, currencies, and units. Tempo remodelled time as an interval instead of an instant. And this week, Agenda shipped - the scheduling engine four years in the making. Topics include: why localization is not translation - the three things that go wrong in an e-commerce search before a single word has been mistranslated the 10-year weekend project: how a localization library for Elixir turned into 28 packages and a full rewrite into Localized why compile-time code generation with Elixir macros hit a wall - 1,000+ function clauses and quadratic compile times - and why moving to runtime fixed it the astronomical algorithms scar: three years of learning a new branch of mathematics to implement calendar software the BEAM was never designed for what Tempo makes explicit that everyone does by accident: time as an interval, not an instant why scheduling is just set intersection over everyone's free and busy intervals - and why that single insight collapses almost every hard scheduling problem how "sometime on Tuesday" is not a time but a domain error - and what Tempo forces you to say instead Unicode version 18, 173,000 human language characters, and why most language ecosystems still can't represent them correctly the difference between CLDR (the Unicode standard) and Localized (the Elixir implementation of it) what Agenda actually ships this week and what's next Recorded September 8th, 2026.

    Kip Cole on Time, Localization and Building Agenda

Ratings & Reviews

5
out of 5
5 Ratings

About

BEAM There, Done That is a podcast about building real systems with Elixir, Erlang, and the BEAM. We’ve built it before — distributed systems, fault‑tolerant services, event pipelines, real‑time apps, production nightmares, and the supervision trees that saved them. Each episode dives into practical lessons from shipping software on the BEAM: architecture decisions, scaling challenges, operational failures, and the patterns that actually work. No hype. No theory without scars. Just hard‑won experience from engineers who’ve been there.

You Might Also Like