Cold Start Latency in Self-Hosted Agent VM Environments

Cold start latency in self-hosted agent VM environments is an architectural choice. Persistent AI agent infrastructure preserves an agent’s working state between interactions, while serverless functions can force that state to be rebuilt when execution environments go cold. Snapshot-restore keeps persistence separate from running compute, so idle agents can sleep on storage and wake in under a second without the cost of always-on machines.

Why cold starts hit agent workloads differently than web functions

A web function can absorb one cold start and serve many requests from the same warm environment. An agent workflow built from discrete sandbox invocations has a different pattern, because reading a file, running a test, and writing a patch can each trigger a separate environment. When those calls run in sequence, their startup delays accumulate inside one user-facing interaction. The workflow can run past the calling application’s timeout budget even when each function finishes within its own limit.

This is a structural limitation of using short-lived functions for a stateful, long-running agent. Google’s RAIL model puts an instantaneous response at around 100ms, so a delay of seconds is well beyond that. For Python AI workloads, the dependency stack can account for the large majority of cold start time before useful work begins. Local model inference can take much longer because the model parameters have to move from storage into GPU memory. Both kinds of initialization still have to run even with faster kernel boot.

What actually happens inside a VM cold start, layer by layer

The first phase is VM or container initialization, including kernel boot, network setup, and filesystem mounting. Firecracker targets this layer, with roughly 125ms from the start API call to Linux user-space initialization. AWS originally built it for Lambda and Fargate, where workloads needed stronger isolation than containers without the overhead of full VMs. Its security model combines KVM virtualization, seccomp filters, cgroups and namespaces, and a jailer process. Production orchestration adds time beyond the boot target, but this layer is no longer the dominant cost for most agent workloads.

The second phase starts the language runtime. Python itself starts quickly, while frameworks based on managed runtimes, such as Spring Boot, have often taken seconds to scan the classpath and initialize their dependency injection container. The third phase runs agent initialization code outside the handler to load configuration, create database connection pools, and initialize SDK clients. The fourth phase is the invocation itself, which is the only phase that runs on every request. For agents, the dependency and initialization work in phases two and three tends to dominate, making those phases the decisive target.

That breakdown also explains why isolated startup measurements can mislead. Sequential creates under ideal conditions do not establish how a system behaves when many users arrive after a quiet period. Running tests over and over during development keeps environments warm and hides work that production requests have to do once those environments are gone. A customer service platform built on Step Functions encountered this compounding latency only in production. Tracing the whole agent workflow, rather than individual function durations, exposed the cumulative delay.

Why the two conventional responses — always-on and scale-to-zero — both fail for agent fleets

Always-on deployment keeps every agent’s VM resident and running, so the next interaction has no startup wait. Its cost, however, follows the number of provisioned agents, including those doing no useful work. A coding assistant that handles traffic during business hours still has to pay for provisioned compute overnight. For a bursty workload, those idle periods can dominate the bill. Keeping everything ready solves latency by charging for the gaps between requests.

Scale-to-zero makes the opposite exchange. It stops charges for idle compute, but the next user-facing interaction has to run the full initialization stack again. If several tool environments must start during the next session, that bill compounds. Neither posture fits sparse activity well when both responsiveness and cost matter.

Each user has a persistent agent with its own state, credentials, and memory, but most agents are inactive at any given moment. As the industry moves toward one cloud computer per user, the size of the fleet grows apart from the number of agents doing work at once. Always-on deployment charges as though those quantities were the same, while cold scale-to-zero treats every return as a fresh beginning. Persistent agents need a deployment posture that preserves continuity without reserving running compute throughout every idle gap.

How snapshot-restore works and why it breaks the always-on vs. scale-to-zero tradeoff

Snapshot-restore introduces that third posture by saving a fully initialized microVM for reuse across activations. The system boots the VM, loads its dependencies, runs initialization, and snapshots its memory and block device state to local NVMe. Subsequent creates from that template restore the prepared state. For an existing persistent agent, suspension preserves its current working state for the next wake. Template preparation avoids repeated initialization, while per-agent snapshots preserve the work performed after initialization.

Copy-on-write memory mapping makes the restore fast. The snapshot’s memory is mapped with MAP_PRIVATE, allowing the guest to use snapshot pages until a write requires the kernel to copy an individual page. The system does not duplicate gigabytes of RAM before execution can resume. Instead, pages arrive as needed, and the agent’s initial working set is smaller than its total memory footprint. Page-cache warm-up consequently matters more to restore latency than the hypervisor alone.

Maritime implements hibernate-and-wake by snapshotting each agent’s full VM state to disk, preserving its persistent disk and own kernel while releasing idle compute. Wake latency is measured in hundreds of milliseconds, and dependency-stack initialization does not rerun on resume. State that must be fresh still needs restore-time handling, but the expensive application startup is no longer part of each interaction.

The economics of snapshot-restore at agent fleet scale

Always-on billing charges for the overnight gap and the active session at the same provisioned rate. With snapshot-restore, sleeping agents still have storage costs, but compute spending pays only for active work. The fleet needs enough compute for the agents that work at the same time. This distinction changes the economics of giving every customer a persistent machine.

The active-to-idle ratio determines the size of the saving. The supplied modeled bursting workload puts suspend-and-restore at 93.8 percent cheaper than always-on, an order-of-magnitude difference. Long idle gaps create the opportunity because storage replaces compute for most of the agent’s lifetime. As agents spend more of their time active, there is less spending on idle compute to cut.

For a fleet managed by Maritime’s snapshot-and-restore scheduler, the same principle keeps the idle per-agent cost low while compute capacity follows concurrency. Agents sleep on storage and wake when work arrives, without keeping a machine running just to hold their context. Persistent agents’ stored state carries a cost separate from continuous execution. That separation is what makes a per-user agent product financially plausible beyond a small, heavily provisioned deployment.

What AWS AgentCore Runtime v2 reveals about where the industry landed on this problem

AWS’s AgentCore Runtime v2 release on September 18, 2026, provides a concrete production example of this architectural direction. AWS’s release notes put v1 cold start latency between 5.4 and 30 seconds, depending on how large the container image is. Runtime v2 keeps p75 cold start latency fast across images from 200MB to 2GB. The relevant change is the removal of image size as the driver of startup time described for the earlier runtime.

AWS pairs microVM-based isolation with elastic memory management in the new runtime. Each session starts with a small memory profile, pages in additional memory on demand, and releases it when it goes cold. The runtime’s consumption billing also charges only for seconds of active processing.

These results support the move toward snapshot-restore, though how it is built still matters. The published comparison establishes more consistent startup across image sizes and a runtime designed around demand-driven resource use. The published comparison gives no latency guarantee that holds for all self-hosted infrastructure. Improving the execution model can remove costs that remain when only the initial VM boot is optimized.

The objections that matter: snapshot replay correctness and warm pools as an alternative bet

Snapshot replay creates a genuine correctness boundary. Multiple environments restored from the same initialized memory can inherit identical random-number generator state, stale startup timestamps, and cryptographic session state that should not be reused. Agents with rotating credentials, session tokens, or timestamped actions are directly exposed to this problem. Preserving memory is not the same as making every value in that memory valid after a pause or clone. The system must tell which context can be kept and which state needs to be renewed.

SnapStart provides beforeSnapshot and afterRestore lifecycle hooks for this boundary. Equivalent hooks in other implementations serve the same purpose, with post-restore handling used to reseed entropy, refresh tokens, and reset clocks. Developers should test specifically for state that must be fresh after enabling snapshot-restore. Infrastructure cannot automatically guarantee the meaning or uniqueness of application values. Correct resume behavior remains a developer responsibility at the hook boundary.

Warm pools offer a legitimate alternative when avoiding even restore latency is worth paying for ready capacity. Pre-booted VMs accept requests without a restore step, but their availability depends on how much capacity has been reserved. Predictable, high-concurrency workloads can make this economical because little of the pool sits unused. Sparse agent fleets favor snapshot-restore because most dedicated capacity would otherwise be idle. A burst that exceeds the warm pool still cold-starts the overflow, whereas snapshot-restore does not depend on forecasting a supply of pre-booted environments, though its storage and restore path still matter under load.

A separate limit concerns local inference. Firecracker does not support GPU passthrough or general PCIe device passthrough, and its virtio PCI hot-plug Developer Preview does not provide GPU/VFIO passthrough. Agents that need direct GPU access have to use bare metal or a different architecture. Agents calling external inference APIs do not face that constraint. For local inference, GPU access must be settled before snapshot wake latency becomes the deciding criterion.

What self-hosted agent VM infrastructure needs to get right to make snapshot-restore work in practice

A useful snapshot-restore deployment requires the VM, storage, and scheduler to work as one system. Each agent needs an isolated microVM with its own kernel so that memory and filesystem state have a clear tenant boundary. Firecracker provides hardware-level isolation through KVM for agents that install packages, execute arbitrary code, and maintain their own filesystems. Fast local NVMe supports the restore path, while network-attached storage introduces round-trip latency that can erode the sub-second advantage. A fast hypervisor cannot compensate for a slow path to the pages the agent needs first.

The scheduler must also manage suspension and restoration. That includes putting idle agents to sleep according to policy and restoring them when requests arrive. Maritime’s infrastructure handles these transitions for agents running in Firecracker micro-VMs on bare metal, with persistent disks, real kernels, and package installation privileges that survive sleep and redeploy. This removes the need for application developers to maintain the suspend-and-restore machinery themselves. When requests arrive in concurrent bursts, keeping initialized agents avoids starting their dependencies again, but the restore path still has to keep up with those requests.

Finally, persistence must cover the state that makes the agent useful, including files, credentials, browser sessions, installed packages, and memory. Restoring a clean template quickly is not equivalent to preserving a user’s agent. The former saves initialization time, while the latter lets the agent accumulate context and capability across sessions. The infrastructure choice should therefore be judged by continuity as well as wake latency. Persistence is the goal, and snapshot-restore is the mechanism that makes it responsive without paying for idle compute.