8

Introduction

Monitoring AI workloads is critical for ensuring optimal performance, preventing resource bottlenecks, and diagnosing issues in real time. Whether you’re training deep learning models or running inference pipelines, visibility into GPU, CPU, and memory usage can make the difference between efficient execution and system failure.

This guide walks through three essential tools—nvidia-smi, htop, and system logs—and explains how to use them together to effectively monitor AI workloads.

Why Monitoring AI Workloads Matters

AI workloads are resource-intensive by nature. They rely heavily on GPUs, consume large amounts of RAM, and often involve multiple processes running in parallel. Without proper monitoring:

• GPU memory can silently fill up
• CPU bottlenecks can slow down training
• Processes may hang without obvious errors

Effective monitoring helps you:

• Detect anomalies early
• Optimize resource allocation
• Improve system reliability

Monitoring GPU Usage with nvidia-smi

What is nvidia-smi?

nvidia-smi is a command-line tool included with NVIDIA drivers. It provides real-time insights into GPU usage, memory consumption, temperature, and running processes.

Key Command

nvidia-smi

What to Look For

• GPU Utilization (%): Indicates how actively the GPU is being used
• Memory Usage: Helps track how much VRAM your model consumes
• Processes Section: Shows which processes are using the GPU

Real-Time Monitoring

To continuously monitor:

watch -n 1 nvidia-smi

This refreshes the output every second, allowing you to observe workload behavior dynamically.

Practical Insight

If GPU utilization is low but memory usage is high, your model might be inefficient or waiting on CPU operations. This is a common bottleneck in poorly optimized pipelines.

Monitoring CPU and Memory with htop

What is htop?

htop is an interactive process viewer that displays real-time insights into CPU, memory, and running processes.

Launching htop

htop

Key Features

• CPU Bars: Show per-core usage
• Memory and Swap Usage: Helps identify memory pressure
• Process List: Displays running processes with resource consumption

Filtering AI Processes

Press / in htop and search for your process name (e.g., python, torch, tensorflow) to isolate relevant workloads.

Practical Insight

High CPU usage with low GPU utilization often indicates data preprocessing bottlenecks. This means your GPU is underutilized while waiting for data.

Using Logs for Deep Diagnostics

Why Logs Matter

While nvidia-smi and htop provide real-time metrics, logs give historical context. They help you understand what happened before a failure or slowdown.

Types of Logs to Monitor

1. Application Logs
   • Output from training scripts
   • Errors, warnings, and progress updates
2. System Logs
   • Located in /var/log/
   • Useful files include syslog, dmesg
3. GPU Driver Logs
   • Capture GPU-related errors such as memory faults or crashes

Useful Commands

tail -f /var/log/syslog

dmesg | grep -i error

Practical Insight

If your training job crashes unexpectedly, dmesg may reveal GPU resets or out-of-memory (OOM) events that aren’t visible in your application output.

Combining All Three Tools

The real strength lies in using these tools together: 

• Use nvidia-smi to monitor GPU health and utilization
• Use htop to track CPU and memory behavior
• Use logs to investigate issues over time

Example Workflow

1. Start your AI training job
2. Open three terminals:
   • Terminal 1: watch -n 1 nvidia-smi
   • Terminal 2: htop
   • Terminal 3: tail -f on relevant logs
3. Observe correlations:
   • GPU idle + CPU high → preprocessing bottleneck
   • GPU memory full → batch size too large
   • Sudden crash → check logs for OOM or driver issues

Common Issues and How Monitoring Helps

1. GPU Underutilization

• Symptom: Low GPU usage
• Cause: Slow data pipeline
• Fix: Optimize data loading or use parallel workers

2. Out-of-Memory Errors

• Symptom: Job crashes
• Cause: Excessive batch size
• Fix: Reduce batch size or optimize model

3. System Slowdowns

• Symptom: Training takes longer than expected
• Cause: CPU or RAM bottlenecks
• Fix: Monitor with htop and adjust workload

Best Practices for Monitoring

• Run monitoring tools in parallel with workloads
• Log all training output for future analysis
• Set up alerts for critical thresholds (GPU temp, memory)
• Regularly review logs even if no issues occur

Conclusion

Monitoring AI workloads is not optional—it’s essential for performance, stability, and debugging. By combining nvidia-smi, htop, and system logs, you gain a complete picture of how your system behaves under load. This holistic approach allows you to quickly identify bottlenecks, prevent failures, and optimize your AI pipelines for maximum efficiency. Whether you’re working on small experiments or large-scale deployments, mastering these tools will significantly improve your workflow.