Now having looked at our load averages,
we're now going to look at how we can monitor high CPU and help to
troubleshoot the bottleneck that we might be seeing.
When we've installed sysstat, we have a command line utility called mpstat,
and that can then go through and gather,
collect real‑time information about our processor utilization.
We've also got sar, our System Activity Reporter,
and sar ‑u will show information relating to our CPU performance.
Now obviously if we're gathering information on a regular basis,
such as we have, where we've enabled our system collection,
we're able then to be able to see our CPU data over a period of time.
When we take a look at monitoring our CPU utilization,
first of all in real time, we can go through and use the mpstat command.
If we want to go through and see some activity,
why don't we go through and compress a relatively large directory.
Our /usr/share directory obviously has a heap of resources
including all of our documentation in it.
We can make sure that we can background our task by sending
everything through to our /dev/null device and then background
it with the ampersand at the end.
Now that will take a little while for it to run through,
possibly a minute or so, just depending on the speed of your system.
Now to collect information, I can go through then and use mpstat 1,
so we go through and run every 1 second for 60 seconds.
I can go through and compress it again to give
myself a little bit more activity, and then we can go through,
and if we want to, we can actually drill down to an individual processor.
By default, mpstat is going to show us all of our processors,
but by using the uppercase P and the processor number,
we're identifying the processor we want to interrogate.
Now, tar and its compression utility that we're using here,
xz, will probably run as a single process,
so it's going to be mainly on one of our processors.
We can also look at something similar, but historically,
so looking at our System Activity Reporter. So ‑u without any additional options
will go through and take a look at all of our CPU activity.
Sar ‑u and ‑P 0 is going to be interrogating again,
in this case, processor 0, but we can try it with processor 0,
processor 1, or whatever number of processors that you have.
Now we can also identify which process is taking the most CPU time.
So if we start off our backup process again and then go into our top utility,
by default, top will organize the processors by those utilizing the most CPU,
and we're probably going to see then our xz
compression utility sitting at the top.
Now I think this again introduces to us a time where we can get our
hands busy and we can go through and take a look at using mpstat and
sar to monitor our CPU performance.
So we're going to be taking a look at all processors,
single processors, as well as using top to identify the CPU hog.
Come on, let's go through and find that command line.
Hopefully we can remember from the slides that when we installed sysstat,
we also put on our whole load of utilities.
So we can go through and look at mpstat.
If I go through then and run that, we can get an idea of our current utilization.
The big clue here is our column on the right that says that,
well, all of our CPUs then combined is giving then an idle of 98.42.
So, yes, yes, yeah, we're not really doing very much,
but we can go through and gather information in real
time on a regular basis as well.
So why don't we go through and look at the tar command.
We're going to create a tar archive.
Let's just move this to the top.
We can go through and use xz compression,
so the uppercase J for Juliet, and we're going to write through to the file.
We'll just call it, let's say, usr.tar.accepts,
so that's going to be our backup.
We're going to be backing up the /usr/share directory.
I just want something that's going to be relatively large so
we're going to induce a lot of CPU utilization.
We want to background this,
so I want to make sure that we're not seeing any information coming back,
so I can redirect errors and our standard output through to /dev/null,
and then that's going to allow me to background the whole of
this backup without seeing any information.
Now this then becomes really useful, because at this stage,
we want to go through and start collecting our information.
So that's running in the background now,
and if I go back and run mpstat,
we can collect data every 1 second for 60 seconds,
because that's roughly how long the backup is going to take.
Now as we go through now, we can see that,
well, everything is blue so we're not getting any warnings,
okay,
we're getting a warning now that we've gone to 50% idle and 50% utilization,
but you can see that we are then pretty busy.
But all of this is probably going to be on a single CPU,
because when we get to sort of 50% utilization,
where we've got two CPUs, or at least I have two CPUs,
so it kind of indicates to me that we're probably flat out on one of our CPUs.
But we will continue to collect this information for 1 minute.
We've said then on our iteration that we want to run
every second and 60 iterations of that.
As we take a look at the output now,
we're able to see that we have hit 100% idle and the rest of
the process of utilization has died down,
so this is a good indication now that the backup has
finished and we've done our 60 iterations now,
as the command has finished.
If I go through and execute again,
will we go through and execute the same backup,
so again generating that same activity?
Now I don't really know which processor is going to be busy,
but if I go back up now and specify ‑P and processor 0,
as we're collecting our information on this,
well, we can see that CPU 0 is really not doing any work,
so let's jump out of this and then change it through to CPU 1.
And we can see now that as we were suggesting before,
where we're looking at our 50% utilization,
we're pretty much 90% utilized when we go through
and start looking at output now.
And, again, this is going to run through until our backup has finished.
But it gives us a good idea to get an idea of,
well, which processors are being maxed out,
really looking at things if we've got processors that are multi‑threaded,
so can run across multiple processors to get better performance as well.
This now has all died down, so I can Ctrl+C to jump out of it.
Now if you wanted to look at the activity over a period of time,
then that's where our System Activity Reporter is going to come in.
By default, it's looking at the file for today,
so I can then do a ‑u and it's looking at our CPU performance.
Now, again,
it's all looking pretty low at the moment because we
haven't collected information.
Remember, the information is collected every 10 minutes.
But, again, we could drill down to individual processors,
so to look, say, for processor 1.
And, again, we can see that there's no real utilization on that.
It might appear slightly different if we look at it when
the next element has been collected.
If I go through and clear out of this, I'm going to go back into my backup.
We'll kick this off again, and now we're going to go into our top utility.
When we look at top, it summarizes information at the top of the screen,
which is very good,
so we can see things like our uptime being shown through our load averages.
But the important thing here is that we're able to see the
processors that are taking most utilization.
And looking towards the top, we can see that xz is hogging the CPU at the moment,
taking about 98‑99% of all of our CPU utilization.
So this becomes a really good way of seeing if we have got a busy system.
We're identifying what command is taking the CPU utilization.
Once we're finished with this, we can do a Ctrl+C to jump out.
Now I'm just going to do a Ctrl+L to clear my screen.
Just out of interest,
I'm going to go back and just see if we can see anything now with sar,
we've collected anything.
Well, we've just finished the backup, but,
yes, we can see we haven't collected anything.
We should be collecting in a couple of minutes' time.
I'm seeing here 8:45, so we'll be connecting next at 8:55.
Now having waited a couple of minutes, and very patiently,
we should be able to go back and relook at our activity.
So running our same command as we had before,
when we take a look at the collection at 8:55,
we're able to see then that our average had hit something like 12.6,
so we can see that we have been a little bit busier.
If we go through and just look at all of our CPUs,
then we can see that the average for our user percentage is 12.13,
so we can see then that we've been gathering information
just a little bit more than we had before.
So we're able to see the activity over a period of time,
which can be extremely useful rather than just saying, well, it's doing okay now, isn't it?