Disk Io Troubleshooting

by Sam Hadow

On my server I self-host quite a lot of services but only have 5900rpm HDDs for the data, and a SSD only for the OS and binaries.
Sometimes these HDDs struggle to keep up with the I/O operations of all my services.
So in this short blog post I’ll show you the troubleshhoting steps to find the culprit of a high disk I/O and how to limit its disk usage.

Check disk usage

To check disks usage we can use the tool iostat (provided by the package sysstat on fedora, debian and archlinux)

to see the extended stats every second:

iostat -x 1

you’ll then get an output like this:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.37    0.00    6.94   21.85    0.00   66.84

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-1             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-2             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-3             0.00      0.00     0.00   0.00    0.00     0.00  524.00   8384.00     0.00   0.00   47.82    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   25.06 100.00
dm-4             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
dm-5             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sda              0.00      0.00     0.00   0.00    0.00     0.00  524.00   8384.00     0.00   0.00   46.02    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   24.11  99.30
sdb              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdc              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
zram0            1.00      4.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00

let’s explain each column:

The most interesting columns for us are:

• %util : high means device is saturated (I/O-bound).
• r_await, w_await, f_await: high means requests are waiting a long time (latency issue).
• aqu-sz: high means the I/O operations queue is growing.
• r/s, w/s: high means lots of small I/O operations.

fun fact: Although iostat displays units corresponding to kilobytes (kB), megabytes (MB)…, it actually uses kibibytes (kiB), mebibytes (MiB)… A kibibyte is equal to 1024 bytes, and a mebibyte is equal to 1024 kibibytes. source

note

In the previous example, dm-* devices are actually virtual block devices managed by LVM (the device mapper).

To identify which physical volumes they correspond to, we can run this command:

ls -l /dev/mapper

Or this command as root:

dmsetup ls --tree

Find the process causing a high I/O usage

For that we can use the iotop command (package is named iotop on fedora, debian and archlinux). We usually need to run iotop as root as it needs elevated privileges.
With the following options it’s easier to spot processes causing a high I/O usage:

sudo iotop -aoP

what these options do:
-a = accumulated I/O since start
-o = only show processes actually doing I/O
-P = show per-process, not per-thread

We can also use pidstat (provided by the package sysstat on fedora, debian and archlinux). It’s better to run this command as root, otherwise you’ll only the processes from your the user running the command and not all the processes.

to show per process read/write operations, updating every second:

pidstat -d 1

We can then write down the PID, or the command, corresponding to the line with a lot of disk writes, or disk reads.

Limit disk usage

podman

With podman we can use arguments in the run command to limit disk I/Os as mentionned in the documentation

These may not work in rootless mode unless I/O delegation is enabled.

You can verify resource limit delegations enabled with this command:

cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"

In our case we need io in the output.

If it’s not present you can create the file /etc/systemd/system/user@.service.d/delegate.conf with the following content:

[Service]
Delegate=io

You can also add the other resource limit you want to delegate to users, for example: memory pids cpu cpuset, the file would look like this:

[Service]
Delegate=io memory pids cpu cpuset

You then need to log out and log back in to have the correct limits permissions.

systemd

To limit disk I/O for a systemd service we can use slices

The most useful options we can put in a slice section are the following, you can see all the available options in the documentation

To create a slice unit, for example: /etc/systemd/system/io-limited.slice

[Unit]
Description=Slice for IO-limited services

[Slice]
IOAccounting=yes
IOWriteBandwidthMax=/dev/sda 20M
IOReadBandwidthMax=/dev/sda 20M

We can then assign services to this slice, in the service section we would have:

[Service]
Slice=io-limited.slice