3

A Slowdown With No Obvious Cause

A server doesn’t need a CPU spike or a memory leak to grind to a halt. Sometimes, the real problem is something much simpler: a log file that quietly grew until there was nowhere left to write. That’s exactly what happened in a recent production incident, and it’s a pattern support teams run into more often than most dashboards would suggest.

The issue first came to our attention when users reported that the application had become unusually slow. Page loads were dragging, scheduled jobs were taking far longer than usual, and even SSH sessions felt sluggish. None of it pointed to an obvious culprit; CPU usage sat comfortably under 30 per cent, and memory looked unremarkable. When the usual suspects come up clean, the next place to look is storage.

Tracing the Bottleneck to Disk

A quick check with df -h told a different story from the CPU and memory graphs. The root partition was nearly exhausted, and almost all of that space had disappeared into /var/log. Running du -sh against the subdirectories one by one eventually pointed to a single application log file that had ballooned to several hundred gigabytes on its own.

That discovery reframed the entire investigation. The performance issue wasn’t a hardware ceiling at all; it was a storage subsystem buckling under the weight of one runaway log.

Reading the Log Before Deleting It

It would have been tempting to just clear the file and move on, but that approach only hides the problem until it resurfaces. Looking inside the log revealed millions of nearly identical error entries, repeating every few seconds. A short-lived database connectivity hiccup had triggered the application’s retry logic, and every single retry was being written to disk at the most verbose logging level available.

That’s the part that catches teams off guard. The database hiccup itself was over in minutes, but nobody had told the logging to stop, so it just kept writing the same error over and over for days afterwards.

Why a Full Disk Causes So Much Collateral Damage

Once disk usage climbed toward 100 per cent, the symptoms multiplied well beyond slow page loads:

• Applications struggled to write new log entries and temporary files.
  
• Database writes slowed as available space shrank.
  
• Backup jobs began failing outright due to insufficient storage.
  
• System processes spent more time blocked on disk I/O, inflating response times across the board.
  

CPU and memory stayed within normal ranges throughout the incident, which is precisely why this kind of problem is so easy to miss during a first pass. The real issue wasn’t compute; it was storage, and storage rarely gets the same real-time attention that CPU and RAM do.

The Toolkit Behind the Diagnosis

Nothing complicated was required to find the root cause, just a disciplined order of operations. df -h flagged the partition nearing capacity, and du -sh narrowed the search to the exact log directory and file. Grepping the log for repeated patterns and timestamps revealed the error loop and enabled estimation of how quickly the file was growing. When we pulled up the history in our monitoring tool (Zabbix, in this case), the trend line told the real story: disk usage had been creeping upward for days, well before anyone reported a slowdown.

That combination- df/du for disk metrics, grep for log content, and Zabbix for historical trending turned a vague “the server is slow” ticket into a precise, evidence-backed diagnosis.

Fixing It Without Causing a New Outage

With the cause confirmed, the fix was deliberate rather than rushed. Old archived logs were compressed and cleared in accordance with the existing retention policy. The oversized active log was rotated using logrotate rather than deleted outright, since a quick lsof check confirmed the application still had the file open; removing it wouldn’t actually free the space until the process released the handle or was restarted. The application’s logging level was dialled back to reduce unnecessary debug noise in production, and the underlying retry logic was corrected so that future connectivity blips wouldn’t flood the logs in the same way.

Disk usage and application performance were both monitored closely afterwards to confirm the fix held. Within a short window, response times normalized and scheduled jobs were running cleanly again.

Keeping This From Happening Again

Recovering from a full disk is always more disruptive than preventing one. A handful of operational habits go a long way:

• Set up automatic log rotation with logrotate and sensible retention periods.
  
• Compress old logs once they’re archived so they don’t sit around eating disk space for no reason.
  
• Keep an eye on filesystem and inode usage as a matter of routine, not just when there’s already a problem.
  
• Set your alerts to fire well before you hit a critical threshold, not right at the edge.
  
• Don’t leave debug-level logging switched on in production; turn it on when you need it, then turn it off again.
  
• Make a habit of glancing at log growth periodically instead of waiting for something to break first.
  
• Where it makes sense, move toward centralized log management so you’re not relying entirely on local disk for something this important.
  
• Re-check rotation policies after application updates, since upgrades can silently reset or break them.
  

Conclusion

Disk-filling incidents are a useful reminder that performance problems don’t always announce themselves through CPU or memory graphs. A log file growing in the background can be just as damaging as a runaway process, and it often takes longer to notice precisely because it doesn’t trigger the alerts teams are watching most closely.

The fix here wasn’t complicated: rotate the log, tune the logging level, and fix the retry behaviour that caused the flood. The real value was in the process: checking the disk before assuming compute, reading the log before deleting it, and treating the fix as a chance to prevent a repeat rather than just clearing space. That same discipline, applied consistently, is what keeps small issues from turning into the next outage.