[AI Agents will Crash] TL;DR - Durable execution helps an AI agent recover its progress after a crash. It can avoid repeating expensive model calls, research, or work a human already approved. But that does not mean the agent’s actions happen exactly once. An external system may successfully create a ticket, send a message, or update a record just before the worker crashes. If the workflow never records that success, recovery may run the action again. The agent has gone back. The outside world has not. AI agents make this harder because a restarted model may generate a slightly different tool request instead of replaying the original one. For production agents, builders need to design these as two separate problems: recovering computation and recovering external side effects. Give every important action a stable identity before the model generates the request, understand what duplicate protection the receiving system actually provides, persist the exact artifacts later decisions depend on, and crash-test the integration at the point where the external action succeeds but your workflow has not recorded it yet. That boundary is where durable execution stops and distributed-systems design begins. A bank runs an AI agent to review new business partners. The agent gathers public information, reads documents, writes an assessment, waits for an analyst to approve it, and then creates a review ticket in the bank’s case-management system. The example is made up. The architecture is not unusual though. Now imagine the worker running that agent dies during a deployment. A new worker comes up and resumes the workflow. For a short question-answering task, restarting may be cheap. Here it is not. The agent may repeat paid searches and model calls. It may regenerate an assessment that differs from the one the analyst approved. It may also create a second ticket for work that already exists. Those are two different recovery problems. The first is preserving completed work. The agent should not redo everything it already finished. The second is safely resolving actions whose outcome is uncertain. The agent may have changed another system just before it crashed, without recording that the action succeeded. Durable execution is very good at the first problem. The second problem is where things get really interesting. What durable execution actually promises Durable execution saves enough state so that a crashed program can continue from where it stopped. That is enormously useful for long-running agents, but the guarantee is narrower. DBOS saves workflow inputs and the output of each step in Postgres. When a workflow is interrupted, DBOS finds the interrupted workflow, starts it again with the same saved inputs, and checks each step for an existing result. Completed steps are skipped. The first step without a saved result is treated as the point where execution stopped, so that step runs again. The important guarantee is in the wording. Steps are attempted at least once, and a completed step is never run again. Database transactions managed by the runtime can commit exactly once. The phrase at least once is what matters for calls to systems outside the workflow engine. Suppose a step creates a ticket in a customer’s case system. The case system saves the ticket successfully. A few milliseconds later, the worker crashes before the workflow records that the step completed. When recovery begins, there is no saved result for that step. The runtime has no proof that the ticket already exists, so it runs the step again. Temporal documents the same boundary from a different angle. A worker can successfully perform an activity and then crash before reporting completion. The server’s history never records the success, so the activity is scheduled again. Temporal makes a useful distinction here. The activity is reported as completed exactly once, but the activity code itself may execute more than once. It may even partially succeed multiple times. That difference sounds small until the activity changes a customer system. Your workflow runtime counts recorded completion. The customer’s system counts actual executions. No workflow engine can make someone else’s API safe to call twice. The runtime can preserve your state, replay your code, and give you stable identifiers to work with. The receiving system still decides what a repeated request means. This also explains why deterministic workflow code matters. Anything unpredictable, including API calls, random values, or reading the current time, belongs inside a step. The workflow itself needs to replay consistently. Unhandled failures matter too. In DBOS, an error that escapes the workflow is recorded and the workflow moves into ERROR. It is not automatically recovered. Calls that are expected to fail temporarily should therefore live inside steps with intentional retry behavior. Retry configuration creates its own failure mode. The Pydantic AI integration notes that DBOS does not classify model errors as final in the same way some engines do. A poorly configured agent can therefore consume its full retry budget before stopping. Durability gives us a powerful recovery model. It does not remove the need to reason about what happened outside that model. The Acknowledgment Gap There is a small but important period between an outside system accepting an action and your workflow durably recording that the action succeeded. I call this The Acknowledgment Gap. In the bank example, the ticket already exists in the case system, but the agent has not recorded that fact yet. The outside world has moved forward. The agent’s durable state has not caught up. That gap exists in every long-running agent that performs real actions. Saving progress more frequently can make the window smaller, but it cannot make the window disappear. The outside action and your local acknowledgment are two different operations. Something can always fail between them. That is why checkpoint frequency is only part of the design. LangGraph makes the tradeoff visible through its durability modes. With exit, progress is written only when the graph finishes, so intermediate progress is not durable. With async, persistence happens while later work continues, leaving a small window where the save may not land. With sync, the save completes before execution proceeds, which reduces that window at the cost of latency. The strongest mode makes the window smaller. It still does not make an external write exactly once. The price of finer-grained durability is measurable. DBOS performs one database write for every step, plus two writes per workflow for the workflow inputs and final result. DBOS reports more than 40,000 workflows or steps per second on a single Postgres database, but the practical cost also depends on how much data each step returns. That is why the same documentation recommends returning a pointer to large objects rather than writing files directly into workflow state. This is a useful engineering tradeoff because we can reason about it directly. More save points mean less work is lost after a crash. They also mean more persistence overhead. What they do not provide is certainty about an external action whose success has not yet been recorded. Why AI agents make ordinary idempotency harder The traditional solution to repeated requests is idempotency. An idempotency key is a stable identifier attached to an operation. When the receiving service sees the same identifier again, it knows that the caller is retrying earlier work rather than requesting a new action. Stripe is a good example. Stripe stores the first result associated with an idempotency key and can return that result when the request is retried. But there are two important details. The first is retention. Stripe may remove keys after 24 hours. If a workflow waits three days for human approval and then resumes, a 24-hour duplicate-protection window may no longer help. The second is payload consistency. Stripe compares a retried request with the… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit theairuntime.com
