Once an as it has occurred within our organization and we've cleaned up the mess,
we've repaired everything,
we've brought it back to the original working condition or we've
applied compensating controls to ensure that
that incident has been fully remediated
and protected against. We need to go through a post incident activity.
Usually post incident involves the idea of what can
we learn from the incident that just occurred.
However,
it's more than just that we need to
identify different aspects of how the incident came
into being what vulnerabilities took place and how
it overall function across our enterprise environment,
whether that was from a flaw of vulnerability or misconfiguration,
we need to complete a forensic analysis and go back through after the repairs
have been completed and identify everything that
was affected within our systems themselves.
If a folder was changed and something was left behind,
we need to figure that out and go through the process of identifying not just
what they changed but deleting anything that was in our systems from the get go
that may have constituted or uh emerged from that incident after the fact.
Usually this involves identifying different network changes,
application folders or programs that may have been installed or modified
uh and digital storage to make sure that the
hashing values match up to where they were before.
This has all been gone over before.
But realistically,
what we're really trying to do here is bring our systems back to the original um
operating of what they were beforehand.
Uh But we also need to look at it from the perspective of what was left behind.
We can fix something and an attacker may have left something behind.
Uh This could be a logic bomb. It could be malware that's waiting.
It could be any number of things including backdoors.
And the mere fact is that we have to go back
through after the major incident has occurred and figure out what happened
once that's happened, we can identify root cause analysis,
root cause analysis is understanding the initial
vulnerability or the flaw in the misconfiguration
that went through the process of what started
this entire endeavor in the first place.
This could be something as simple as identifying, hey, they came in through telnet
and we're able to get access to our username
and passwords because it was transmitted in the clear
two is something as complex as somebody clicked
on a phishing email which we thought we caught
and then it
inadvertently downloaded malware into this system which then
expanded or antivirus system didn't work properly.
There could be many layers of flaws within the
uh within the enterprise system,
but finding the root cause and then every step that happened
along the way is vital to understanding what happened in the network as a whole.
This all constitute a root cause. Analys
of
understanding what happened to be. Provide us with this initial attack vector.
Well, we are going to identify log files.
We're gonna go through network diagrams, we're gonna go through workflows,
we're going to go through the entire process and really understand
not only the root cause but what happened along the
way and that all constitutes that root cause analysis.
You could have a problem
that started with say telnet
and then
evolved over time to go from there and then
spread out like a virus over our different systems.
Uh Each of those having their own quote
unquote root cause, but the true root cause would be that initial
telnet.
That doesn't mean we're just gonna identify telnet
as our initial problem and fix that.
We need to fix all the flaws and vulnerabilities along the way.
Finally, at the end of the, at the end of the fix, right?
So you've gone through, you've had this incident,
you've got employees that are working 2025 hour shifts and I'm not joking it,
it's really been that bad sometimes where
you've got employees that are virtually exhausted.
We've fixed the major incident, we've uh fortified our systems,
we've closed down the vulnerabilities, we've stopped the attack,
that's in progress
and there's still a lot left to do.
This is not the time to do the lessons learned capability. Right.
I've seen brand new managers come into play
and we fix all the problems.
Everybody's running on fumes and it's like, oh, well, what did we learn from this?
Nobody cares. I, I hate to be that guy but nobody cares at that point. Right.
We're all exhausted. Let your employees go to sleep,
send them home, tackle it in 8 to 10 hours.
There's still cleanup processes that have to take place.
There's other technical things that need to be done,
but having your employees work on limited
sleep where they're literally falling asleep at the
table is not good for you and it's not good for your employees either.
But once the mess has been cleaned up, once you're fairly confident,
you've got all the vulnerabilities taken care of the flaws have been remediated.
The malware is out of your system now is the time to go through and identify.
What did we learn from this process?
Could we have done something easier
that would have shaved off some hours of time? It's not,
it's not unnatural
to find somebody who made a mistake at the very
IO and they were going down one path only to find out two or three hours were wasted
because they could have gone this direction. Had they identified the hint
that was provided within the malware scheme.
It happens to everybody. This is where experience really comes into play.
This is why somebody that's got 10 years
of experience is paid more than somebody with,
you know, two months of experience because experience matters.
But in order to get those experiences,
we need to do a lessons learned that lessons
learned process allows people to sit at a table.
Identify what could I have done better?
What tools could we have utilized that would have stopped this in the first place.
What could we have done that would have mitigated the problems that we foresaw.
I remember several times when I was a newer employee or even
an experienced employee where we had a disaster take place and we
wasted two or three hours,
even five or six hours going down a rabbit
hole that had nothing to do with the problem.
But all the resources said go this way.
The problem was, the technology was so new that no one really understood the logs.
So we use what the manufacturer told us in the first place should be the root problem.
We only find out that wasn't the root problem at all.
Had we identified the complex nature of the logs.
We could have, we could have uh digress and gone.
Hey, it's actually over here.
Uh And knowing that this one little alarm in combination of this other alarm,
the problem is in this direction, not
in this direction
and that's normal. You shouldn't get mad as a manager that, that happens.
It's part of that learning experience,
especially when new equipment or new technology is coming to bear.
That's why your experienced employees make so much more money than
your brand new employees because that experience really does matter.
So we need to go through lessons learned policy so that
not only can your experienced employees underscore what they learned,
but the newer employees get a feeling of, oh, this is how it's done
and what did they learn along the way?
A lot of times during this lesson learned process,
you'll have brand new employees that don't want
to open their mouth because they feel like
what they say will make them sound stupid or
that they don't deserve to work here and they
really face that syndrome that says I don't belong,
I'm not really qualified.
You need to pull,
pull them out of that shell and reassure them
that it's a learning process to get them to
talk about it because I've had brand new employees
where they come to bear and they're like,
oh, I learned this, this and this.
When I was going through the logs, we had no idea,
but they had literally solved the problem.
Had an experienced employee saw the same logs they had,
we could have shaved off 10 hours, but that new employee was so worried about
getting fired or being looked down upon that. They didn't
you want to share that information?
Now, you can blame the employee but I don't think that's the right way.
That's a cultural issue, not an employee issue.
And so I think with this, let's learn process,
it's the culture of the company that really comes in
play here that says it's ok to make mistakes.
Let's learn from those mistakes.
And if you can constitute that good culture within your company,
yes, you may have some rocky starts to begin with.
But long term wise, your company, your organization can be far better for it.