3

Virtual machines make it possible to run multiple workloads on the same physical infrastructure. This improves resource utilization and provides flexibility, but it also introduces performance characteristics that systems engineers must understand.

One of the most useful metrics when troubleshooting CPU performance inside a virtual machine is CPU steal time.

CPU steal time represents time when a virtual CPU (vCPU) is ready to run, but the hypervisor is giving the underlying physical CPU to another virtual machine or workload. From inside the guest operating system, this time can look like unexplained CPU performance loss.

Understanding steal time can help engineers distinguish between a problem inside the guest operating system and CPU contention occurring at the virtualization layer.

What Is CPU Steal Time?

A physical server can host multiple virtual machines. Each virtual machine is presented with one or more virtual CPUs.

When a VM needs CPU time, its vCPU must be scheduled by the hypervisor onto a physical CPU. If the physical CPU is busy servicing another VM, the waiting VM cannot immediately execute.

Linux can report this waiting period as steal time.

In tools such as top, CPU steal time is commonly represented by %st.

For example:

%Cpu(s): 10.0 us, 5.0 sy, 0.0 ni, 80.0 id, 0.0 wa, 5.0 st

Here, 5.0 st means the guest’s CPU accounting recorded time as stolen by the virtualization environment.

The exact interpretation depends on the virtualization technology and guest environment, but the fundamental idea is the same: the guest was ready to run but did not receive the physical CPU time it expected.

Why Does CPU Steal Time Matter?

A VM can appear healthy from inside the operating system while applications still experience poor performance.

For example, an application may require CPU time to process requests, but the VM repeatedly waits for access to a physical CPU. The application may therefore experience increased response times even when traditional guest CPU utilization does not appear exceptionally high.

Steal time can be particularly useful when investigating:

• Unexpected application latency
• Slow batch processing
• Inconsistent VM performance
• Increased request-response times
• Database performance degradation
• Periodic CPU starvation
• Performance differences between similar VMs

It is important to remember that steal time is a symptom of scheduling contention, not necessarily evidence of a problem with the guest operating system.

How to Check CPU Steal Time in Linux

One of the simplest tools is top:

top

Look for the %st value in the CPU summary.

You can also use mpstat if the sysstat package is available:

mpstat

For per-CPU information:

mpstat -P ALL 1 5

This can show whether steal time is affecting particular virtual CPUs.

Another useful source is /proc/stat:

cat /proc/stat

The CPU statistics include a steal-time field on systems that support it.

For continuous monitoring, export CPU steal time to your normal observability platform rather than relying on occasional manual checks.

High Steal Time Does Not Always Mean the VM Is Broken

A common troubleshooting mistake is to assume that high steal time automatically means the VM itself has a configuration problem.

The guest operating system cannot directly control how the hypervisor schedules physical CPU resources.

High steal time may instead indicate contention elsewhere in the virtualization environment.

Possible contributing factors include:

• Too many workloads competing for physical CPUs
• Host-level CPU saturation
• VM overcommitment
• Noisy-neighbor workloads
• Host scheduling contention
• CPU limits or quotas
• Cloud-provider resource conditions
• Oversized or poorly balanced VM configurations

Therefore, guest-level metrics should be correlated with hypervisor or cloud-platform metrics whenever those metrics are available.

Distinguish Steal Time From Other CPU Metrics

Understanding the difference between CPU states is essential.

User time (us) represents CPU time spent executing user-space processes.

System time (sy) represents CPU time spent executing kernel code.

I/O wait (wa) generally represents time CPUs spend waiting for I/O operations.

Idle (id) represents CPU capacity that is not being used.

Steal (st) represents time taken from the virtual machine by the virtualization layer.

This distinction matters during troubleshooting.

A VM showing high us may simply be running CPU-intensive applications. A VM showing high wa may have an I/O bottleneck. High st, however, points toward contention for physical CPU scheduling.

Investigating High CPU Steal Time

Start by establishing whether the elevated steal time is temporary or persistent.

Run:

mpstat 1 10

Observe whether %steal remains consistently elevated.

Next, check application performance. If steal time increases at the same time as application latency, the two events may be related.

Then compare the VM with other instances running on the same platform, where that information is available.

If multiple VMs experience elevated steal time simultaneously, the problem may be associated with the underlying host or resource pool rather than one guest.

Also check whether the VM has recently changed workload characteristics. A previously CPU-light workload may start competing heavily for CPU resources after a deployment or traffic increase.

CPU Steal Time in Cloud Environments

Cloud environments add another layer to the investigation.

A cloud VM runs on infrastructure shared with other workloads, although the exact isolation model depends on the provider and instance type.

When CPU steal time increases, review the cloud provider’s available monitoring metrics alongside guest-level measurements.

Useful information may include:

• CPU utilization
• Instance CPU limits
• Throttling metrics
• Host-level performance indicators
• VM size or instance type
• Scheduled maintenance events
• Historical performance trends

Do not assume that a high steal-time value always indicates a provider fault. Resource configuration, workload behaviour, and scheduling policies can all influence observed performance.

What Can Systems Engineers Do?

If sustained CPU steal time correlates with application performance problems, possible actions include reviewing the VM’s CPU allocation, workload placement, resource limits, and virtualization configuration.

Depending on the environment, an organization may consider moving the workload to infrastructure with more available CPU capacity, adjusting resource allocation, or reducing unnecessary CPU demand.

Any infrastructure change should be validated using performance measurements before and after the change.

The objective should be to address the underlying resource contention rather than simply reacting to one CPU metric.

Monitor Steal Time Proactively

CPU steal time is most useful when viewed historically.

Create monitoring dashboards that show:

• CPU utilization
• CPU steal time
• Load average
• Application latency
• Request rates
• I/O wait
• Memory utilization

Correlating these metrics makes it easier to identify whether performance degradation coincides with CPU scheduling contention.

For production systems, alerts should also consider duration and impact rather than triggering solely because steal time briefly increases.

Conclusion

CPU steal time is an important metric for understanding virtual-machine performance. It describes CPU time the guest expected to receive but that the virtualisation layer allocated elsewhere.

When %steal is consistently elevated, systems engineers should investigate the virtualization environment, workload behavior, resource allocation, and application performance together.