Linux Disk I/O Monitoring with iostat, iotop, and Real Bottleneck Hunting

Disk I/O problems are some of the easiest Linux performance issues to misread. CPU looks fine, memory is not completely exhausted, load average is high, and the server still feels sticky. Commands hang for a second or two. Databases stutter. Backups start slowly and then drag everything else with them. That is usually when I stop staring at top and start looking at storage latency.

I/O troubleshooting is less intuitive than CPU troubleshooting because high utilization does not always mean the same thing on different storage types, and the old “disk busy percentage” mental model can lead you straight into the weeds. This guide is the workflow I actually use on Linux systems to figure out whether storage is the bottleneck, which process is causing it, and whether the problem is throughput, queueing, latency, or swap pressure.

If you are also dealing with disk capacity rather than performance, server disk space management and LVM usage are good follow-ups. Capacity and I/O pain often show up together.

Install the tools first, because some minimal images do not have them

On Debian or Ubuntu, iostat comes from sysstat and iotop is its own package.

sudo apt update
sudo apt install -y sysstat iotop

On RHEL, Rocky, or AlmaLinux:

sudo dnf install -y sysstat iotop

I do this early because too many cloud images ship without the tools I need at the moment I need them.

Start with iostat: it shows me whether storage is actually unhappy

The command I use most often is:

iostat -xz 1

Flags:

• -x extended statistics
• -z omit devices with all-zero activity
• 1 refresh every second

That gives me a moving view instead of a one-time snapshot. The first report is since boot, so I pay more attention to the later lines.

The columns I care about most

Different versions vary a little, but these are the usual signal columns:

• r/s reads per second
• w/s writes per second
• rkB/s and wkB/s throughput
• rrqm/s and wrqm/s merged read/write requests
• await average time for I/O requests to complete
• %util percentage of time the device was busy

I also watch aqu-sz or similar queue depth fields on newer versions.

What high %util means, and what it does not

People love %util because it looks authoritative. I treat it as a clue, not a verdict.

On spinning disks, %util near 100% often really does mean the device is saturated. On SSDs and especially NVMe, high %util by itself is less definitive. Modern devices can handle huge parallelism. I care more about latency and queueing than the percentage alone.

If %util is high and await is rising, users are probably feeling it.

If %util is high but latency is low and stable, the device may simply be busy doing fine.

await is where the pain usually shows up

If I had to choose one column to explain user complaints, it would often be await.

There is no universal “bad” threshold because workloads differ, but this is my rough practical feel:

• low single-digit milliseconds is usually healthy for SSD-backed workloads
• tens of milliseconds starts getting noticeable for databases and interactive applications
• hundreds of milliseconds is usually serious trouble

When await jumps during write-heavy bursts, I suspect one of these first:

• database checkpoints or flush storms
• runaway logging
• backup jobs competing with production traffic
• swap thrashing

rrqm/s and wrqm/s: merged requests are not just trivia

These columns show read and write requests merged by the kernel or device layer before being sent down. People often ignore them, but they help me understand access patterns.

Higher merge rates can be normal for sequential workloads and can improve efficiency. Very low merge rates on spinning media during random workloads can hurt performance. On SSDs the story is different, but the fields still give context.

I do not stare at rrqm/s and wrqm/s first, yet when I am comparing a healthy period to a bad one, those numbers can explain why the same throughput now causes more latency.

Use iotop to find the process causing the I/O spike

Once I know storage is busy, I want the culprit.

sudo iotop

This usually requires root privileges because it reads kernel accounting data per process.

To show only processes actively doing I/O:

sudo iotop -o

That is the mode I use most often. Otherwise a quiet system shows too much irrelevant noise.

What I look for in iotop

• a database process writing heavily during complaints
• a log shipper or app process flooding writes
• a backup or compression job unexpectedly running during business hours
• package management, antivirus, or indexing work on systems that should be quiet

One recurring lesson from production: the top I/O consumer is not always the one with the biggest business impact. A backup job may be writing steadily and legally, but the real problem is that it overlaps with the database flush window.

vmstat adds context iostat does not show alone

I almost always pair iostat with vmstat.

vmstat 1

The fields I care about most for I/O diagnosis are:

• bi blocks received from block devices
• bo blocks sent to block devices
• si swap in
• so swap out

If si and so are moving, I pay attention immediately. Swap activity changes the whole interpretation of the system. What looks like generic disk slowness may actually be memory pressure turning into I/O pain.

Swap thrashing looks ugly in practice

Typical clues:

• vmstat shows repeated swap in/out
• iostat shows high write activity and rising latency
• the server feels uneven or frozen in bursts
• top shows processes in uninterruptible sleep (D state)

At that point, tuning storage alone will not save you. You may need to fix memory pressure, query behavior, worker count, or caching.

dstat is useful when I want one dashboard view

Some systems still have dstat, though it is not as universal as it once was.

dstat -cdngy 1

That can show CPU, disk, network, and paging together. I would not rely on it as my only tool, but when I want quick correlation across subsystems, it is handy.

Because packaging varies, I do not assume it is present. When it is missing, I stick with iostat, vmstat, and ss or sar depending on the incident.

Historical data helps when the spike already passed

Live tools are great when the problem is happening now. They are less helpful when somebody tells me “the server was awful at 03:10” and the box is calm again by the time I log in.

That is when I use sar, which is also part of sysstat on many systems:

sar -d 1 5

And for stored daily reports:

sar -d -f /var/log/sysstat/sa22

The exact file name varies by day and distribution. The point is that historical disk counters can prove whether a slowdown was a one-off burst, a recurring backup window, or an all-day storage problem.

/proc/diskstats is the raw source when I need to get serious

Sometimes I want the kernel counters directly.

cat /proc/diskstats

This is not a friendly view, but it is the underlying source for a lot of higher-level tools. I use it when:

• I am validating a monitoring tool
• I want to script a custom comparison
• package availability is limited

A lot of people never need to read it manually, which is fine. I am mentioning it because understanding that these tools are reading counters rather than magic helps when outputs disagree.

ioping is good for quick latency checks

Throughput is not the whole story. Sometimes the complaint is “the server pauses randomly,” which is often latency rather than bandwidth.

Install if available:

sudo apt install -y ioping

Then test a mount point:

sudo ioping -c 10 /var/lib/postgresql

Or current directory:

ioping -c 10 .

This gives me a feel for responsiveness. It is not a substitute for real workload analysis, but it is a useful sanity check.

If the latency reported by ioping looks bad while iostat is relatively quiet, I start thinking about underlying storage layers outside the guest OS: noisy neighbors, SAN contention, cloud volume credits, or a hypervisor problem I cannot fix from inside the VM.

Deep block-layer analysis with blktrace and blkparse

Most incidents do not need this level. Some do.

On systems where the tools are installed, the workflow is usually:

sudo blktrace -d /dev/nvme0n1 -o - | sudo blkparse -i -

Or trace to files first if I need post-analysis:

sudo blktrace -d /dev/sda -o traceout
sudo blkparse traceout.*

I reserve this for cases where normal monitoring does not explain latency spikes and I need to see the block-layer behavior in more detail. It is powerful, but not a first-response tool.

SSDs and NVMe changed the way I read storage metrics

Old instincts from spinning disks can mislead you badly on SSD-backed systems.

What matters more on SSDs

• latency consistency
• queue depth behavior
• IOPS patterns
• write amplification under real workloads

What matters less than it used to

• simple “disk is 100% busy so it must be dead” conclusions
• sequential-vs-random fear in the old spinning-disk sense

On NVMe, I can see large IOPS counts without obvious pain, then get real trouble from bursts of latency caused by contention, garbage collection, or an overloaded virtualization layer underneath.

That is why I focus so much on await and practical user symptoms.

Choosing the I/O scheduler with intent

Schedulers are less dramatic than they used to be, but they still matter.

Check the current scheduler:

cat /sys/block/nvme0n1/queue/scheduler

You might see something like:

[none] mq-deadline kyber bfq

My rough guidance

• NVMe: none or mq-deadline are common sensible choices
• general server workloads on block devices: mq-deadline often behaves well
• desktop or interactive workloads: bfq may help responsiveness

I do not change the scheduler during an incident unless I know exactly why I am doing it. But when tuning a persistently unhappy workload, it is worth testing.

Temporarily set one:

echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Remember that changes made this way do not survive reboot unless persisted through bootloader or udev/system configuration appropriate to the distribution.

Real problems I see most often

Database-heavy writes

Symptoms:

• high w/s
• growing await
• database latency complaints
• iotop shows the database process at the top

What I check:

• checkpoint settings
• WAL or redo log placement
• backup overlap
• query patterns causing write amplification

Log flooding

Symptoms:

• huge write throughput from app or logging processes
• sudden log growth
• CPU may still look normal

I have seen simple bugs write thousands of lines per second and drown otherwise healthy SSDs. That is where logrotate management and application log discipline matter.

Validation commands I use:

sudo iotop -o -b -n 3
sudo du -sh /var/log/*
sudo find /var/log -type f -size +100M -ls

Those three commands usually tell me whether the “storage problem” is really just uncontrolled logs filling disks and forcing constant writes.

Swap pressure

Symptoms:

• vmstat shows si and so
• system is erratic, not just slow
• iotop may show kswapd influence indirectly

The fix is usually memory-side: reduce workload, tune services, add RAM, or stop overcommitting.

Backup jobs colliding with production

Symptoms:

• predictable time-of-day slowdown
• rsync, snapshot, compression, or database dump process near the top in iotop

This is why I schedule backups carefully and watch contention, especially when using rsync for backups on the same host as the application.

Validation commands I use before making changes

Before I tune anything, I capture a baseline.

iostat -xz 1 10
vmstat 1 10
sudo iotop -o -b -n 5
cat /sys/block/sda/queue/scheduler

The -b batch mode in iotop is useful for documentation and comparisons.

If the device mapping is unclear, I verify it:

lsblk -o NAME,TYPE,SIZE,MOUNTPOINT
findmnt

On virtualized systems, I also want to know whether the storage layer below me is shared or throttled, because guest tuning cannot fix host-side saturation.

Common mistakes during I/O troubleshooting

Looking only at load average

High load does not automatically mean CPU. Processes waiting on disk can drive load high while CPUs sit fairly idle.

Treating %util as a universal truth

Useful, yes. Sufficient, no.

Ignoring swap activity

If swap is active, you are not just debugging storage anymore.

Killing the loudest process without understanding its role

Sometimes the process doing the I/O is the database writer that exists because another service is misbehaving. The visible actor is not always the real cause.

Tuning the scheduler before identifying the workload

Scheduler tweaks can help, but they are not a substitute for finding the process and pattern actually causing latency.

My practical I/O triage flow

This is the sequence I use most:

1. iostat -xz 1 to confirm device pressure
2. vmstat 1 to see swap and block activity
3. iotop -o to find active processes
4. lsblk to map devices to mounts
5. validate scheduler and storage type
6. if needed, ioping or deeper tracing

Example:

iostat -xz 1
vmstat 1
sudo iotop -o
lsblk -o NAME,TYPE,SIZE,MOUNTPOINT
cat /sys/block/nvme0n1/queue/scheduler
sudo ioping -c 10 /var/lib/mysql

That usually gets me to the right hypothesis quickly enough.

If the process is obvious but the timing is not, I correlate with the service manager:

systemctl list-timers --all
crontab -l
sudo ls /etc/cron.d

Scheduled jobs are responsible for more I/O spikes than people like to admit.

Conclusion

Disk I/O monitoring is really latency investigation with extra steps. iostat tells me whether the device is busy and whether waits are growing. iotop tells me who is pushing the workload right now. vmstat tells me whether memory pressure is part of the story. Everything beyond that depends on how stubborn the problem is.

The practical lesson I keep relearning is that storage bottlenecks are often workload-design problems wearing a hardware costume. A bad backup window, uncontrolled logging, swap pressure, or write-heavy query pattern can make excellent storage look terrible. If you watch await, identify the active process, and keep the workload context in mind, you solve these faster and make fewer pointless tuning changes.