Checkpointing and Resuming Multi-Hour Agent Tasks on Self-Hosted VMs
Preserve long-running agent progress with dual checkpoints at application and VM layers.

Multi-hour agent tasks need persistent execution infrastructure. Checkpointing and resuming these tasks on self-hosted VMs requires two layers working together, with application checkpoints preserving workflow progress and VM snapshots preserving the execution environment. To recover correctly, the task must also check committed side effects and refresh resources before it runs again.
Why multi-hour agent tasks expose a structural gap in stateless execution models
An agent crashes at step 7 of 12 and restarts from zero. Its earlier tool outputs, intermediate decisions, and completed work existed only in the process that disappeared. The replacement process has no record of its position. Every expensive LLM call before the crash must run again, adding token spend and wall-clock delay without advancing the task.
The failure is architectural. Short tasks can finish before the infrastructure interrupts them, but multi-hour workflows run long enough to face crashes, timeouts, and rate limits. Planned interruptions reveal the same weakness when a human approval gate requires the process to stop. Under a stateless execution model without durable checkpoints, neither interruption preserves a continuation point. The fix belongs in infrastructure and orchestration, where progress can survive the process that produced it.
The difference between agent state and agent memory, and why confusing them breaks recovery
Agent state records where the current run is in its execution. It includes completed steps, tool outputs, intermediate variables, relevant message history, and flags that determine the next branch. A step counter belongs here, as does the result of the last tool call. This state can become disposable when the run ends.
Memory serves a different purpose across the agent’s lifetime. User preferences, domain knowledge, and prior decisions must remain available to future runs after the current checkpoint is no longer useful. An agent can therefore resume its workflow correctly yet know nothing about an earlier session.
Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents distinguishes task ledgers, permissions, credentials, triggers, provenance, and external commitments from retrievable memory. These categories need separate recovery and governance rules. Treating everything as conversation history obscures those differences. It also makes recovery depend on an expanding transcript.
What a checkpoint must actually capture to make resumption indistinguishable from continuation
The checkpoint design should capture the minimum complete state needed to resume safely. A step counter records where the run is, and saved intermediate results keep it from doing the same computation again. Branching conditions explain why the next step was selected. Provenance records establish which steps ran and what they produced.
A saved session reference or authentication token identifies a dependency that the restore path must validate. Large working context should remain outside the checkpoint as files, pointers, or summaries. The Ralph Loop pattern keeps working state in files this way and starts each iteration with a fresh model context.
External commitments require more than a position marker. A database write or sent email must have a durable completion record so recovery does not repeat it. The design rule is idempotency. Every resumable step must be safe to run again, and non-idempotent operations need a guard that checks whether their effect was already committed.
Application-layer checkpointing: graph frameworks and the state-store pattern
Graph-based orchestration makes workflow checkpoints explicit at node boundaries. Nodes are decisions or tool calls, and edges set where data and control go next. LangGraph has typed state, conditional edges, and interrupt points where a human can oversee the work. Its checkpointer binds a thread ID to a sequence of snapshots, allowing the same thread to reload its saved position.
A linear pipeline uses the same basic loop. It selects a unit of work, processes it, writes the result and position to durable storage, then advances. Dapr Actors support this through persisted actor state, with explicit save_state() calls where required and end-of-turn automatic saving in.NET. For a simpler non-actor pipeline, the Dapr state API can write directly under a task-scoped key. In either case, a completed LLM result is reusable only after it has been saved.
The choice of storage affects how easily operators can look at recovery state. PostgreSQL and SQLite provide durable, queryable records suitable for audit trails. Redis gives fast access to working state, but adding persistence can come at a cost to performance. Object storage provides durable storage without a native hierarchical filesystem interface. These stores can capture CPU registers, open file handles, or network buffers only if the application spells out the information it needs to save.
VM-layer snapshot-restore: how Firecracker captures and resumes a running agent below the application
Firecracker snapshots preserve execution below the application boundary. They serialize the microVM’s memory, CPU registers, and device state, then restore those components without a kernel boot or agent startup. On restore, Firecracker memory-maps the snapshot and resumes execution from the saved position.
That distinction broadens what recovery can preserve automatically. In-memory variables, caches, and guest-side open-file and network state remain part of the captured environment. Persistent disk preserves installed tools and filesystem context, and it must remain aligned with the snapshot. The guest’s saved network state does not, however, establish that an external connection remains usable after a long pause.
The latency difference can also change product responsiveness. Snapshot loading can take single-digit to low-double-digit milliseconds internally, while end-to-end restoration takes longer and must not be confused with that internal step. ForgeVM, a Go-based open-source sandbox orchestrator, demonstrates a measurable advantage for subsequent snapshot-based spawns over cold startup. Its example establishes feasibility.
Restarting the application and reloading a state store remains valid for explicitly saved application state. It does not reconstruct everything the application omitted. Self-hosted autonomous agents also benefit from VM-level recovery because they have browser-session state, in-memory caches, and changes to the environment that build up as they run.
Using both layers together: when application checkpoints and VM snapshots serve different roles

The two layers address different recovery units. Application checkpoints record how far the workflow has progressed, so they can be used for step-level recovery, inspection, and side-effect guards. VM snapshots preserve the broader running environment at deliberate pauses and scheduling boundaries. Neither replaces the other’s role.
A layered design saves application state after meaningful work and snapshots the VM at phase boundaries or approval gates. Restoration brings back the VM first, then checks application progress before further actions. This keeps workflow records lean without requiring developers to serialize every environmental detail. A prior VM snapshot can also restore a controlled environment to an earlier position, but it cannot undo an email or external database change. Rollback therefore remains a governance decision.
Checkpoint frequency: how often to snapshot without making overhead the bottleneck
Checkpoint frequency trades runtime overhead against repeated work after failure. Saving too often consumes execution time, while saving too rarely enlarges the recovery window. The design target is a small overhead relative to each work segment. Segments of 10–30 minutes provide a starting heuristic for agent workloads, subject to task boundaries.
Natural boundaries are often more useful than timers. A completed document or pipeline stage provides a clear application checkpoint, while an external mutation needs records before and after its execution. VM snapshots belong at phase boundaries and human-in-the-loop gates. Full memory serialization has a cost even when restoration is fast.
Approval waits make that distinction especially concrete. An agent waiting hours for a manager should not retain an active process throughout the wait. A checkpoint followed by a VM snapshot allows compute to stop while saved state remains in storage. Keeping application records limited to position, results, and references also prevents checkpoint growth from becoming the recovery bottleneck.
Restoring cleanly: the correctness requirements that determine whether a resumed agent behaves correctly
An agent sends an email, crashes, and sends it again after restoration. The process resumed successfully, but the workflow did not. A restored instruction pointer or step counter alone cannot tell whether an external effect already happened.
For the layered design, restoration starts with the VM environment, then loads the application checkpoint and checks committed effects. Only then can the agent identify the last unconfirmed step. Where an action and its completion flag can share a transaction, they should commit together. Otherwise, the workflow still needs an idempotency guard that prevents an uncertain completion record from becoming a duplicate action.
Resources also need a deliberate lifecycle across pauses. File locks, network connections, and browser sessions should be released before a planned pause and reacquired on resume. Short-lived tokens may have expired while the task was stopped. The restore path must refresh them rather than treating captured credentials as proof of current authorization.
Non-deterministic results require similar discipline. Random choices, timestamps, and model outputs must be recorded and replayed where continuation depends on their original values. Re-sampling can make a workflow take another branch after it is restored. The Always-On Agents survey treats provenance as a separate state category, so recovery must also preserve the sequence of steps and outputs that makes the audit trail coherent.
Persistent AI agent infrastructure vs serverless functions for checkpointed workloads
The execution substrate determines which recovery guarantees are available. Stateless functions can reload application checkpoints from an external store, but that does not restore the process that produced them. The required VM-level control includes an independent kernel, persistent disk, and snapshot access. Extending a request timeout does not supply those properties.
Shared containers can likewise host application checkpointing without providing this per-agent VM snapshot boundary. They share a kernel rather than capturing an independent guest environment for each agent. Firecracker microVMs instead provide a kernel per guest and support snapshot-restore with persistent block storage. The deployment must configure and preserve that storage across sleep and redeploy.
Per-agent isolation gives each agent a place of its own for credentials, installed packages, and browser state. It also separates recovery from changes to another agent’s environment. This is the relevant comparison for persistent agent infrastructure versus serverless functions. Short, isolated tasks can fit request-response execution, while autonomous workflows that accumulate environment state need infrastructure that can preserve that environment.
Fleet-scale economics: how snapshot-restore turns idle agents from a billing liability into a tractable cost

When a fleet grows large, much of its cost comes from time spent idle. Agents wait for users, approvals, tool results, and scheduled work, so total agent count can greatly exceed active concurrency. Keeping CPU and memory allocated to each waiting agent means paying for them throughout the wait. Snapshot-based scheduling separates durable identity from active execution.
The cost then comes from active concurrency plus stored state. An approval wait consumes snapshot storage rather than a running guest, making long human delays much less costly. For self-hosted infrastructure, that frees capacity for other agents while the underlying host bill remains. The distinction is essential to understanding the savings.
Commodity bare metal, including Hetzner and OVH, can support this cost structure, but hyperscaler pricing can make it harder to keep paying for dedicated capacity for each agent. Snapshot-restore is therefore both a recovery mechanism and a way to make persistent agents economically tractable.
Putting the architecture together: a practical checkpoint design for a self-hosted multi-hour agent
A reference design gives each agent an isolated microVM with persistent disk and a durable application checkpoint store. PostgreSQL or SQLite can hold progress, intermediate results, committed-effect records, and provenance under task-scoped keys. Coordinating agents can use a shared store instead. Raw conversation history stays outside the checkpoint, with saved references or summaries providing the context needed for continuation.
The application commits progress after each meaningful unit of work and guards every external action against repetition. VM snapshots save the environment at phase entry and human approval gates, keeping it separate from step-level records. Restoration follows a fixed sequence. It restores the VM, loads the application checkpoint, verifies committed effects, refreshes credentials and sessions, and continues from the last unconfirmed step.
An agent that fails at hour five of a six-hour workflow can then recover its environment without replaying completed work. Its remaining recovery window is the current unconfirmed unit, provided completed results and effects were durably recorded. The same design handles a planned approval wait. The agent checkpoints, snapshots, stops consuming active compute, and later resumes through the same correctness checks used after a crash.


