In this lesson,
we'll look at how we can monitor the performance of our storage devices
just like CPU and RAM storage devices have their limits.
If a process is overusing the CPU, keeping it at 100%
the system will slow down considerably.
And the same can happen if a storage device is overused.
Now,
we have utilities like top and HTOP to look at how processes are using the CPU in a ram.
But what about storage devices?
How can we see what's reading and writing to these?
There are many tools that can monitor, read, write operations.
Let's look at some of the simpler ones.
There's a package called
cat
that contains a few such tools
and we can install it with the usual command pseudo
app
install
CSAT.
This contains a few other system statistic utilities.
But the ones we're interested in right now are called iostat.
And Pidstat,
iostats name comes from IOS statistics
and IO is short for input and output because we can
input data to a storage device when we're writing to it
and the storage device outputs data when we read from it.
Pidstat is short for process ID statistics.
Each process on Linux has a unique identifier.
So this tool will show us statistics for each process on our system
along with their ID numbers.
Let's start with the simplest way to see a summary of how storage devices are used.
We can use the IOSTAT command with no arguments.
So if we type in
Iostat at
the command-line, we'll see output like this.
Now, let's figure out what this output shows us.
First of all, this displays historical usage of our storage devices.
That is, it tells us how the devices were used since the system was booted up.
Not necessarily how they're used currently.
And this is a bit tricky to understand when looking at the perse
fields here.
But let's simplify
these fields show an average of usage divided by
the total time since the system was started.
For example, let's say the system was booted three seconds ago
in the 1st 2nd, something read from the disk at 100 kilobytes per second.
In the next second,
something read from the disk at 200 kilobytes per second.
In the 3rd 2nd,
nothing was read from the disk anymore.
So the usage was at zero kilobytes per second.
If we add 100 plus 200 plus zero, and then divide by the total number of seconds.
Since the system was booted three seconds,
we get an average usage of 300 divided by three or 100 kilobytes per second.
So I
oat might show us 100 in the kilobytes red field.
If another second passes and nothing reads from the disk iostat
would show something different the next time we run it.
So now we have 100 plus 200 plus zero plus zero,
divided by four total seconds since the system is up.
So we might see 75 in the kilobytes red field,
the fields which are calculated this way are here,
TPSKB, read slash SKB written slash S and KB DS CD slash s.
When we're looking at what's using the storage devices,
we're interested in the first three fields.
TPS stands for transfers per second.
We can think of it in terms of how many times
did the system tell this device to read or write something
KB reads slash SS refers to kilobytes, read per second,
KB RT N slash S shows kilobytes written per second.
The fields KB red and KB written are much more straightforward.
They simply show the total number of kilobytes
read or written since the system booted,
they don't depend on time. So there's no average to calculate
something to note here. Is that what I
OAT shows us does not always reflect the exact amount of data per second
written or read by a process.
For example, a process can request to write one kilobyte every second,
but IOSTAT might not show the device is used
at thousands of kilobytes per second during that time.
That's because IOSTAT tries to figure out these numbers
by looking at what the device is reporting,
not the process
and the device might work with larger blocks of data.
Even if a process sends just one kilobyte,
the device might report it updated a much larger block.
So estimates can be inflated here, especially for small transfers,
but they become more accurate for larger transfers.
A storage device can be overused or stressed in two different ways.
Something reads or writes to it very often
or something reads and writes to it at a very high speed,
large volumes of data are transferred.
If something uses the devices, often
we'll see a high TPS number.
And for large volumes of data,
we'll see large numbers under these two fields,
the KB read and write fields.
So even if something is writing very little data to the device,
say one kilobyte at a time.
If it does this hundreds of times per second, it's still stressful for the device,
potentially reaching its limits
because each storage device can process a
certain maximum amount of requests per second.
If one process does a high number of transfers per second,
there's not much left for other processes to use.
So one process can starve many other processes by overusing the storage device.
The same goes for large transfers.
If the device can only write two gigabytes of data per second,
and a process is already writing at 1.98 gigabytes,
there's very little left for other processes.
So they will wait for a long, long time until they can finish writing their data,
which can lead to a very slow unresponsive system.
And this is where Iostat
and Pidstat can team up and help us
investigate scenarios where processes overuse our storage devices.
Let's put it on our detective hats and see how by looking at some commandss in action.
First, we'll open up a process that will write constantly to a storage device
and we can use the following command for this purpose.
If stands for input file,
we picked slash dev slash zero here for a file that will
generate an infinite number of zeros of stands for output file.
We'll write our data to a file called delete me
BS equals one tells our DD command to write data one byte at a time
count equals 1 million. We'll make DD write that one byte a million times.
This BS plus count combination makes DD write for a long time.
So we can test our scenario.
Now, O flag equals D
sync makes DD avoid using cache
with right
caching. The right operation could be finished way too early.
But with this argument, the right operation will go on for a long time.
And then finally, the ampersand
sign at the end makes this command right in the background.
So we can continue to use the command-line without waiting for DD to finish.
Now,
let's imagine we don't know what is writing to
the storage device and we want to find out
what should we do?
Well, first, we'll make use of the Iostat command,
but in a new way than we learned previously, we'll type this at the command-line,
Iostat
space one.
This makes Iostat output how our storage devices are used every one second.
Basically, it refreshes every one second
to refresh every five seconds.
We could write five at the end or whatever we need for our purposes.
It's important to know that.
Now, Iostat doesn't show the average usage since the system booted up.
So KB underscore W RT N slash s here doesn't
show us the average right speed since the system booted.
Instead, it shows us the average right speed for the last one second.
Now, as we leave our IO stat command running, we'll keep seeing output like this
and it helps us identify our first clues.
We can see that DM slash zero and SD A are being written to.
We'll find out later how to identify what DM dash zero belongs to
at this point.
All we know is that something is writing to DM dash zero and slash DEV slash SD A.
But we don't know what process is actually doing this.
That's where the next command pidstat can step in to help with our investigation.
First, we'll press control C to quit our IO stat program.
Then we'll type this command
pidstat
dash D
to remember the dash D option. Think of devices.
If we forget to add this option,
pidstat won't show us how processes are using our storage devices.
Now, the output of pidstat shows us our next clue.
It's easy to figure out that KB underscore RD slash S column refers to kilobytes,
read per second
and KB plus wr slash S refers to kilobytes written per second.
Now note that the KB underscore Wr slash S column,
we can see that the DD command is the one with the largest value in this field.
It's the process writing the most kilobytes per second.
We should note and remember this process ID by looking at the PID column.
In our example, we'll have to remember the pid 1411.
It might be useful to know that Pidstat also supports showing stats every X seconds
if we need that feature. So with a command like this,
we can make it output io statistics for processes every one second,
which can be useful if our previous command didn't catch anything interesting
because pits that is a bit similar to I
oat.
When you use it with no command-line options, it outputs average read,
write transfers per second
and its historical average values.
But if you want live results so to speak,
then use the command like pidstat dash D
one to constantly refresh data every one second
with control C, we can exit from pidstat as well. Just like we did with IOSTAT.
At this point, we have two clues,
DM dash zero and slash DEV slash SD A are being written to
the Pid doing the most intensive rights is 141.
So now for the last step, what is this DM zero device?
Well, DM stands for device mapper.
This was actually created by the logical volume
manager as we learned about in a previous lesson
and we can find out what this maps to with the DM setup command.
All right.
And if we follow this up with an lsblk command, we'll see this.
Now in the first command, we can see that DM dash zero maps to something called
Ubuntu dash dash BG
Ubuntu dash dash lb.
It's an L VA logical volume created when we installed our operating system.
And in lsblk,
we can see that this logical volume is created under slash DEV slash SD A three.
So it's a logical volume created on the slash DEV slash SD A disk.
Now we know why both DM dash zero and SD A
showed up as being actively used in our Iostat command at
the end of the day SD A is the real device being used since
DM zero is just an abstraction that uses SD A under the hood.
So our detective story is now concluded and we now know the following
DD is a process writing to slash DEV slash SD A,
the first storage device on the system.
The PID of DD is 1411 in our example.
Now, in our case,
we could use a command like
PS 1411,
which shows us the exact command that was used to start this process.
And finally, if we want to gracefully stop this process,
we can type in kill 1411.
And if that doesn't work, we can force quit it with kill dash 91411.
This sends the sigkill signal which abruptly terminates the process
and this means the process won't be able to
terminate gracefully save its data and so on.
So only use this as a last resort if the first command doesn't work
and that kind of wraps up our detective lesson.
And it's a good example of how we usually investigate things on a Linux system.
On many occasions,
we have to use more than one tool and collect
clues from all of them to reach a conclusion.
To end this lesson. Here are a few command-line options that might be useful.
FOST
dash D makes it show only the device utilization report.
You may have noticed that normally it also displays some CPU usage at the beginning
with slash D. We can omit those details.
Another useful option is slash H to make it
display IO statistics in a human readable format.
Normally, IOSTAT shows us the stats in kilobytes.
And if we see a kilobytes written value of 11311196,
it can be annoying to figure out how many megabytes or gigabytes.
This means dash H will automatically adjust those values to kilobytes,
megabytes or gigabytes,
whichever makes sense for the output.
So instead of a long stream of numbers, we might see 12.7 gig,
which is much easier to read.
The equivalent of this option in PIDS status is
human,
which means we can use a command like this
to see stats in a human readable format.
Don't use dash H with pidstat since dash H
is assigned to a different thing in this case.
And finally,
you might have noticed that iostat shows us activity for entire storage devices,
but not for each partition on them.
For example, we can see here
how slash DEV slash SD A was used,
but we can't see how slash DEV slash SD A one or slash DEV slash SD A two was used
to check out individual partitions as well. We can use either of these commands
I osad
dash
P
all
to see all partitions and devices
or
I osad
dash PSD A
to see all partitions on the SDA device.
And remember you can always use man iostat
and man pidstat
to refresh your memory on the command options you can use.
Now, I hope this was interesting and helpful.
Now, let's move on to our next lesson.