when it comes to servers, there's actually not just the idea of hardware but 2 2
also of course the operating system. 3 3
Well, what are issues we'll deal with and 4 4
what are the tools that are available to us? 5 5
Let's find out more next. 6 6
>> You're watching IT pro TV. 7 7
Alright, you made another great choice once again as we get started here on 8 8
operating system issues and the tools that are actually available to us. 9 9
And the great thing is we have Mr Wes Bryan with us who understand some of 10 10
the issues that will deal with as we continue to work on servers. 11 11
So Wes this is probably one of those things, 12 12
the novice that's actually getting started. 13 13
They don't understand. 14 14
They see the hardware, the operating systems on it. 15 15
Everything should be fine at this point. 16 16
Hands off, right. 17 17
>> Unfortunately, no. 18 18
[LAUGH] I wish it would be that way. 19 19
And that's ultimately at the end of the day with our servers running. 20 20
That's what we want. 21 21
Remember our servers are not like your traditional desktops and 22 22
workstations, right. 23 23
They are what provide the resources that allow our company to be productive. 24 24
They go offline, we gotta determine whether it's hardware components that 25 25
are causing the issue or in this case, what we're gonna talk about are some of 26 26
the more common software or OS and software issues. 27 27
Again, keep in mind when we talk about the OS is essentially software that allows us 28 28
to use our hardware, and 29 29
we'll by association kind of talk about application issues too. 30 30
So running what I've done is I've got us a list here and 31 31
this are some of the issues that I would be aware of on the exam, like for 32 32
instance you're gonna see that we have unable to log in. 33 33
We have unable to access resources, unable to access files. 34 34
Now one of the things that you'll see is that they're kind of hinting 35 35
at is that some of these issues could be related to domain access. 36 36
And what we mean by that is keep in mind that if you're gonna access resources that 37 37
are within a domain environment, 38 38
the computer that you log into has to have a trust relationship with the domain. 39 39
Now that's technical jargon for saying it has to be part of the domain. 40 40
Now your domain controller is what's out there and it's listening out for 41 41
those authentication requests, not only from users, but 42 42
also from computers to believe it or not. 43 43
Even though we do not actually enter in a computer name and 44 44
its password when I type in the user name and password. 45 45
And I log into a domain machine, those credentials and 46 46
including the machine itself underneath actually send those credentials back to 47 47
the domain controller and they say, hey, we're part of the domain. 48 48
The main controller says prove it. 49 49
Well, if I can't communicate with the domain controller, 50 50
then I'm having domain issues that might not allow me to log in. 51 51
All right, it could be account issues too, right, keep in mind that part of security 52 52
around your accounts or things like account lockouts. 53 53
Account disablement, which means that if you've tried too many times to log into 54 54
a domain account and it's not working, you're not typing the password right? 55 55
There might be a threshold set that doesn't allow you to log in. 56 56
So you can start to see that it could be a domain access issue. 57 57
Domain issues. 58 58
It could be something like as simple as account issues as well. 59 59
>> Yeah, some of the things that we might not get because we don't see them 60 60
as often as we think that we should, which is that that machine loses that trust with 61 61
the domain either as sat for one reason or another. 62 62
You know, it's just sat idle or something and all of a sudden you go and 63 63
do what you do normally. 64 64
And then it says, hey, you're not really connected domain even though you go and 65 65
check it and says it is domain joined everything else. 66 66
But somehow that actually gives us something that will have to do which is 67 67
a lot of times rejoin that computer back to the domain to get that trust 68 68
reestablished. 69 69
>> Most definitely and it's as simple as a joining and rejoining the domain. 70 70
Now I want you to understand that some of the issues that can cause this, 71 71
it could be network connectivity but it also could be what they call clock skew. 72 72
Keep in mind that anything that any kind of authentication mechanism that you have 73 73
that is time sensitive and 74 74
there's a lot of communications across the network that are time sensitive. 75 75
But we're just focusing in on how does that affect the domain? 76 76
Well you have to be within by default, five minutes of the domains. 77 77
Master time clock and if your computer no longer is then those Kerberos, timestamps, 78 78
authentication protocols, timestamps no longer work, they don't match up and 79 79
the computers. 80 80
Or the domain controller says, now I don't trust you anymore. 81 81
And then that's at that point, like Ronnie says that we have to take that computer 82 82
off of the domain and we have to rejoin it to the domain. 83 83
And then what happens is it synchronizes with the master time clock of the domain 84 84
controller. 85 85
Now we no longer have that clock skew. 86 86
Now when we talk about things like unable to access resources, 87 87
this could be strict access control, misconfigurations could be permissions. 88 88
It could it could be permissions and I don't wanna get too much into 89 89
the networking but if you know topics right here, but it could be the fact 90 90
that you don't have network connectivity wherever that file is actually stored. 91 91
Alright, so it could be domain issues but it could also be permissions issues. 92 92
If you're unable to access files, you see these just kind of keep coming up now, 93 93
this could be domain issues as well, it could be misconfigured permissions. 94 94
It could be the fact, again, even though I didn't put it in here, 95 95
it could be just as basic as network connectivity and 96 96
you cannot communicate with wherever that resource is actually being hosted. 97 97
We talked about system file corruption, system file corruption, some of the major 98 98
ones that I think of our, well malware, we like to blame everything on malware. 99 99
I have to at least mention it. 100 100
But let's not just assume if you've got anti malware software running on 101 101
your machine, let's not assume that it's actually malware because it could just be 102 102
something like maybe had an update and it failed, right? 103 103
And you had to roll back in the part of the rollback Ronnie, 104 104
I don't know if you've ever had to do this, 105 105
I've actually had to roll back a Windows update and then I got into this boot loop. 106 106
If you will not for a loop, boot loop, right? 107 107
And the machine would just constantly loop and I had to boot into safe mode and 108 108
use some of the recovery options to actually reinstall the operating system. 109 109
System file corruption could also be the fact that you have a hard 110 110
drive storage device that's failing as well. 111 111
So these are some of the first few Ronnie, but let's go ahead Ronnie and 112 112
I'd like to bring you into this conversation because we've talked about 113 113
things like end of life, end of support, right? 114 114
And that's the next one we have to talk about, let's talk a little bit about that. 115 115
One of the things that comes to mind Ronnie, 116 116
when I think of end of life machines, I think of lack of support. 117 117
That's the first thing that comes to mind for me. 118 118
What are some of the things maybe that comes to your mind? 119 119
>> Yeah, no, when it comes down to the end of life a lot of times it may literally be 120 120
where that machine actually has been in use for so long that literally 121 121
something happens and it tries to reboot and it doesn't come back up. 122 122
So when it comes to the end of life it could literally be that hey, 123 123
that machine should be replaced at some point. 124 124
So support is probably the commonplace, right? 125 125
That's the reason why we actually do something on that life cycle approach. 126 126
And so that we don't hit those moments were actually have a machine online for 8, 127 127
10 years and now we need to replace it and we find out that we can't replace it 128 128
easily as we think that we could with a brand new machine. 129 129
We have to kind of take some different incremental steps there as well. 130 130
But service is normally the first thing I think of but 131 131
I have seen those machines that are actually just continued to work. 132 132
So nobody touches them until the time when it's now trying a very 133 133
difficult path to reboot and it takes longer than they think because 134 134
they didn't remember the last time they actually ever booted it. 135 135
And it's a lot longer to boot than some of the newer servers that are online. 136 136
>> And most definitely so it not only becomes an OS functionality issue. 137 137
We'll talk about some security issues later that also becomes a security issue 138 138
because now you have potential that you have machines on your network that can no 139 139
longer be patched right. 140 140
One of the big things that we patch for is not only functionality flaws, but 141 141
a lot of times, first and foremost of those security updates. 142 142
And again, I know that's a security concept but 143 143
also keep in mind the continued functionality and 144 144
one of the things that we remedy that with is good asset management. 145 145
And part of that good asset management is determining like Ronnie 146 146
hinted to what is the end to end life system life cycle. 147 147
And are we truly implementing and documenting that? 148 148
So again, back to Ronnie's comment. 149 149
We don't come up on that end of the life. 150 150
And we're like, well, we didn't know right. 151 151
You should know. 152 152
All right, so those are some things. 153 153
Now, I will tell you there's a laundry list when it comes to slow 154 154
performance, right? 155 155
We mentioned some of the ones in here, I think of excessive workload, right? 156 156
If you have more users or more services, more applications, more API calls if 157 157
you will that you're talking about front end communicating with the back end. 158 158
That is hitting that server. 159 159
That can actually, that the server can actually respond to. 160 160
We're gonna get performance issues right. 161 161
Low compute resources kind of works hand in hand, now that doesn't necessarily mean 162 162
that you have going out the gate low compute resources. 163 163
It might be low compute resources after there's an excessive workload put on it, 164 164
but you also have to keep in mind what are the resources? 165 165
I say the compute resources, 166 166
the two that come to mind is the CPU as well as the RAM. 167 167
Networking is important too, but this is not a networking show, so 168 168
we'll get to that coming up a little bit later. 169 169
Insufficient disk space, keep in mind that we've talked about things like page files 170 170
and swap files, right. 171 171
We use those locations of the hard drive as a way to extend a lack of 172 172
physical RAM when the disk space gets too full. 173 173
Guess what? 174 174
You're gonna start to see performance issues as well. 175 175
If you happen to have a print server out in your network, 176 176
remember one of the things that the print server does is. 177 177
It actually spools the jobs and 178 178
sends them to the appropriate destination print devices, right. 179 179
And if you have a full hard drive and that hard drive cannot spool those jobs to 180 180
the disk before it actually sends them across the network and 181 181
routes it to an appropriate printer. 182 182
We now have performance issues that could potentially be happening on that server. 183 183
So disk space is something that you need to keep in mind. 184 184
Other things malware, obviously malware can cause slow performance. 185 185
Another thing that they call out is memory leaks and if you're not aware of what 186 186
a memory leak is keep in mind that most applications that run. 187 187
When they run and they are done, they release the memory two other processes and 188 188
services if you will and applications. 189 189
When there's a bug in an application even though it has shut down, 190 190
it could reserve and hold those memory locations. 191 191
And continue to utilize additional memory until we get slow performance or 192 192
at the end of the day, maybe we have system hangs crashes, freezing lockups. 193 193
>> Yeah, a lot of that when it comes down to memory leaks right. 194 194
Are the idea of applications that may not have either verified that they 195 195
actually work the way they need to. 196 196
In other words, it might be a line of business item, 197 197
line of business application that they didn't just check and verify 198 198
that everything was working exactly the way that need to in terms of the memory. 199 199
So they put it in play, it actually works for us. 200 200
But then all of a sudden we realized something that the application is now 201 201
working a little bit slow and 202 202
now the operating system is working a little bit slow. 203 203
Because constantly consuming additional memory here so 204 204
don't always blame the operating system. 205 205
Because usually those things are not going to be the key cause but 206 206
usually third party apps at this point I would start checking their too. 207 207
>> Most definitely, especially if you've got a server that you're using for 208 208
like GPU rendering you launched that one application of the whole. 209 209
Well, first of all the, all the lights in the building dim and 210 210
then the service performance goes down. 211 211
You might want to check the resources on the server and 212 212
also check the requirements, the vendor requirements for that application. 213 213
Now the next one here, I want to call out this term application specific or 214 214
application level permissions. 215 215
Because one of the issues you'll see here is that you'll notice it says cannot write 216 216
to the system logs we have throughout the series from time to time shown you system 217 217
logs and the information is there? 218 218
So why might a system log not show up if I've got a piece of software 219 219
that's running on a server and it needs to write to a system 220 220
log what its current state is information so that we can audit that. 221 221
If we don't give that third, a lot of times it's third party software. 222 222
Then the native software, 223 223
a lot of times you don't have to worry about the permission it's not a given. 224 224
But usually when you have a third party installation like that, 225 225
it might try to write its information to a system long. 226 226
And if you don't have the right associated application level privileges or 227 227
permissions depending on what systems you are, 228 228
that software might not actually be able to write to that log. 229 229
A lot of times it's going to be a permissions issue last but 230 230
not least in this list we talked a little bit about service management, right. 231 231
We talked about or service issues. 232 232
I'm sorry, is ultimately what it comes down to it service management, right. 233 233
And what we're talking about are the services that run on the machines. 234 234
All right, if a service fails to load at the time you're expecting it to load, 235 235
it could be disabled right. 236 236
Now, don't get me wrong, it could be a service, 237 237
an application that completely fails, right? 238 238
So it could be a startup failure altogether. 239 239
But remember, you also have dependency chains and 240 240
that also goes for things like when we talk about modules and 241 241
we talk about package management which will mention coming up here is that. 242 242
You have a dependency chain and if I have a service that doesn't start up, 243 243
what I need to do is find out is it dependent is that services startup 244 244
dependent on another service that it relies on? 245 245
And maybe that service isn't starting but you're not aware of that, right. 246 246
So I also understand that you have dependency change that you have to keep 247 247
in mind too. 248 248
Now, I also mentioned a little bit about some freezes we have talked about in other 249 249
episodes, we had a dedicated show on system crashes and hangs. 250 250
So I just want to briefly touch on this on the operating system side, 251 251
not that the components of the devices or the hardware has failed. 252 252
It could be a compatibility issue if you've ever installed an application that 253 253
for whatever reason the executable allowed you to install it. 254 254
But is not the application version that should be running on that machine and 255 255
you launch it and 256 256
it freezes the whole machine up chances are you might have a compatibility issue. 257 257
All right, you might have file or system bio corruption and 258 258
I made the distinction here because you have application level files that allow 259 259
the application to run but they run in user mode. 260 260
And don't actually run in kernel mode so you don't write again but a system file, 261 261
you might make modifications to the registry upon installation right now. 262 262
You're talking about a system file, system files, the crucial files that 263 263
the operating system needs to run that could now be corrupted. 264 264
So those are some of the issues we mentioned memory leak. 265 265
Keep in mind that you also have other types of memory issues that might happen, 266 266
a buffer overrun. 267 267
All right, the buffer overrun they call sometimes called a buffer overflow if it's 268 268
an actual attack that's being done. 269 269
And this just means that you've got a logical portion of your memory called 270 270
the memory stack that applications in the operating system run in. 271 271
All right, it is typically a protected and checked locations, right. 272 272
However sometimes they're bad coding bugs, right? 273 273
Lack of patch management, no updates. 274 274
Sometimes you get code that spills into those unchecked portions of memory. 275 275
Now a couple of things could happen here could be a malicious nature doing through 276 276
malware where they're actually trying to crash your system or 277 277
either way at the end of the day. 278 278
It could crash the system and 279 279
it could hang systems as well too keep in mind that these are memory issues. 280 280
And you need to pay attention to the applications that you have installed as 281 281
well as any anti malware software that you might need to run. 282 282
We talked about freezing, freezing again, application hanging if you will a lot of 283 283
times they're run hand in hand, calls them out as two separate things. 284 284
I'm of the mindset that these are you talking about a system crash and 285 285
a system hang. 286 286
We're really talking about one and the same thing. 287 287
All right, so all the memory issues that we've already talked about including 288 288
system file corruption and driver issues. 289 289
Then we have package management and update potential failures, right? 290 290
So when we talk about package management, 291 291
we talked about the packages that are running within Linux systems, right. 292 292
If you go to do package management and you have those issues, 293 293
are you doing the proper privilege escalations? 294 294
Are you running a sudoku? 295 295
All right, they do you have the repositories that you're trying to pull 296 296
the package down for? 297 297
Have you configured those repositories? 298 298
Do you have network connectivity to even connect to those repositories? 299 299
These could cause issues as well, when we talk about patch management issues, 300 300
talk about patch management, patch update failures. 301 301
Patch update failures could be simply network connection. 302 302
It could be insufficient disk space because keep in mind 303 303
you have to download the updates before you actually install the update. 304 304
So if you don't have the disk space to actually install or download the updates, 305 305
the installations early is going to fail too. 306 306
Another thing to keep in mind too we don't typically like to reboot servers, 307 307
right servers should stay online. 308 308
So one of the things that could be a patch update failure that I want you to keep in 309 309
mind, just like we stress at a plus and net plus simple fixes. 310 310
First, did you apply an update and 311 311
you've got a pending reboot before the update actually finishes its installation. 312 312
If you've ever been in a Windows world, you probably have to deal with this. 313 313
Once every Tuesday, right? 314 314
We call it patch Tuesday or rollback Wednesday, depending on your perspective. 315 315
All right, so keep in mind that you might have to have a scheduled deployment and 316 316
a scheduled reboot time, too. 317 317
So Ronnie, these are really just a lot of the issues that we've talked about, 318 318
I know we haven't really touched based on some of the tools that we have, 319 319
we've kinda mentioned some of them. 320 320
But these are some of the issues that they call out on the exam that I would tell you 321 321
and encourage you to be able to identify giving a scenario and 322 322
then we'll talk about some of those tools. 323 323
>> Yeah, so what about those tools, 324 324
what are some of the things we need to focus in on? 325 325
>> And we've kinda mentioned some of it and we've mentioned it here, 326 326
some of the things that come to mind is proper package management. 327 327
We've mentioned some of the things that you can do in order to ensure that 328 328
that happens. 329 329
Proper privilege escalation, 330 330
remember that you have to have privileged access management. 331 331
If you have somebody that needs to be administrator that you need to be running 332 332
Sudu, if you have the user account control, 333 333
they need to be an administrator in order to install stuff. 334 334
So proper privilege escalation. 335 335
The other thing that I would mention too is service and process management, 336 336
starting services, restarting services, stopping services, 337 337
configuring the startup type to automatically start up if it's a service 338 338
that you need knowing what the dependency chain is. 339 339
And are those dependencies prevalent? 340 340
When you use package managers like something like DNF, yum or apt, they will 341 341
actually search for the dependency chains and download the additional dependencies. 342 342
Now that is not a given so you always have to do your due diligence and 343 343
make sure that dependencies are there and they are present. 344 344
Next, we have a few different things. 345 345
The hardware compatibility list is always great cuz if you've got a piece 346 346
of software that you need to run on a machine, 347 347
you need to find out if the hardware that you have to run that on is going to be 348 348
compatible with the operating system of the software that you're going to install. 349 349
Again, I've kinda put package management twice. 350 350
I'm not trying to beat that one to death here. 351 351
That's typo on my part. 352 352
Remember good configuration management tool. 353 353
We can use things like SCCM, right? 354 354
We can use things like the desired state configuration to make sure that we don't 355 355
have configuration drift that can happen at time. 356 356
Remember that you have orchestration and automation type technologies like puppet, 357 357
chef answerable that can help you to do this at large scale and 358 358
reduce the administrative burden of doing this configuration against thousands of 359 359
servers. 360 360
And then finally, at the end of the day, it's not what we set out for but 361 361
we do have recovery options, keep in mind that you can reload 362 362
the operating system and you can remove all of the application data, 363 363
it's gonna require you to reinstall your applications for sure. 364 364
Remember that you have safe mode. 365 365
Remember that the legendary safe mode that if you install a driver and 366 366
it causes an issue an application to hang, you can always move into safe mode, 367 367
find the offending driver, uninstall it, roll it back, 368 368
maybe find the valid driver and start the process over again. 369 369
These are some of the operating system tools that I would say to be aware of for 370 370
the exam. 371 371
>> All right, so let's say for example, 372 372
we have a Linux server that we're actually managing. 373 373
Can you actually help us out here and give us a small demonstration? 374 374
>> Yeah, Ronnie, so 375 375
I've got Ubuntu actually installed on my local machine here, so I'll run it and 376 376
keep in mind, ladies and gentlemen, that when you log in to Ubuntu server, that 377 377
doesn't have the graphical user interface, this is what you're gonna see as well. 378 378
We wanna ensure that we elevate our privileges properly. 379 379
So I'm gonna do like let's say a sudo and 380 380
we'll use the package manager apt here and we'll just do maybe apt. 381 381
Oops, that is not how you do that. 382 382
Let's try to spell update, right? 383 383
Syntax, by the way, 384 384
will not allow package manager [LAUGH] to do its job if you don't type it right. 385 385
All right, so let me see if I remember what my password is. 386 386
So now you can see that it's actually reaching out to the repository, right? 387 387
It's gonna be pulling down those updates. 388 388
So that gives you an example there, Ronnie, 389 389
the proper privilege escalation that you should do. 390 390
Should you be doing something like package management? 391 391
>> So let's say we're actually like in a Windows server then. 392 392
What about services there? 393 393
>> Sure, so we could use the services management console when you have to do 394 394
service management. 395 395
Let me show you here. 396 396
If I right click on my start icon and now what I mean is I'm on my domain 397 397
controller, I can launch the computer management console if I want. 398 398
Keep in mind that sometimes things that you do in different MMCs will actually 399 399
launch the server manager because that's where they want you managing your servers. 400 400
Imagine that it was named appropriately [LAUGH] but under services and 401 401
applications, we can get into the services console and 402 402
we can find various services if we want and we can see whether they are started, 403 403
whether they're manual, whether they're disabled. 404 404
So for instance, let me show you one, application identity is an example of 405 405
a service that is manually, it's startup type if you will, is manual by default. 406 406
This is actually the service that we use in things like AppLocker to 407 407
identify applications that are allowed to run or denied to run and 408 408
if this service isn't started and what I mean by that is turned to 409 409
an automatic start and actually start the service. 410 410
The AppLocker identity service that says, hey, this is a good application, or 411 411
this is a blocked application, it'll fail to run and 412 412
that whole entire platform will be useless without this service running. 413 413
I also mentioned dependency chains. 414 414
It's also important to understand, 415 415
do you have the appropriate dependencies installed like, for instance, RPC? 416 416
About half if not 75% or more of every single service in Windows relies on RPC. 417 417
So if something happens to RPC, you're gonna see a lot of this list fails 418 418
to start so that shows you an idea of the dependency chains, right? 419 419
So give me a couple of examples to some of the OS issues and 420 420
those tools that we've been talking about in this episode. 421 421
>> All right, well the,re you heard it from Wes Brian himself that these 422 422
are some of the issues you're more than likely to encounter as you continue to 423 423
manage different servers and it's very helpful that we actually understand those 424 424
issues as well as the tools that are actually available. 425 425
Well, that will go and end our episode once again but 426 426
there is another episode coming up and you don't wanna miss it and 427 427
we don't want you to miss it, join us there. 428 428
>> Thank you for watching ITProTV. 429 429
[MUSIC]