Remember the message.
The future is not set.
I flew in yesterday after getting up at 4 o'clock in the morning.
But anyway, so this is going to be my talk on firewalls and fire alarms.
Here's the agenda.
I'm going to talk a little bit about me, very briefly.
And then I'm going to tell you a story.
Actually, I'm going to tell you two.
And then I'm going to show you some comparisons to illustrate some points that I think are really important to understand.
And we'll talk about some case studies that we get out of those particular stories.
And some lessons learned.
And ultimately, we're going to talk about preparing for failure.
Anybody here ever had a control in place and it failed?
Never, right?
That never happens.
And if you're one of those few people that are like, oh yeah, no, everything I put in place works.
Well, either you're wrong or you just haven't figured that out yet.
So this is me.
I really like sloths.
That's the most important thing on this slide.
This is my buddy Flash.
He's with the Buffalo Zoo.
I work at the University of Buffalo.
I've been there over 25 years.
I'm also, what is not on this slide, I've been a volunteer firefighter almost 30 years, which is where some of these ideas come from, putting these two things together.
In this space, in terms of technology, I volunteer with a whole bunch of things.
I speak all over the place, and as I mentioned, I love sloths.
Who's ready for story time?
Ready for stories?
All right.
Who's heard of the Iroquois Theater or the fire it had?
Yeah, probably just a couple of Chicagoans.
Yeah, okay.
So don't add too much if you know a whole lot about this particular story.
Yes, it was a fire.
We'll get there.
So the Iroquois Theater was built, and it opened in 1903.
Now, you have to understand the story behind the building of the Iroquois Theater.
There had been a Brooklyn Theater, which had burned to the ground in the late 1870s in New York, and it was a disaster.
A lot of people died.
I think it was like 275 people died because there weren't a lot of things in place to save people.
Construction was a problem.
There were just numerous issues.
When they went to put this theater together, the person who designed it said, nope, we need to design a theater that will avoid all of those catastrophes.
So if you look carefully, you can see this says absolutely fireproof.
What do we know about absolutes?
Only Sith believe in absolutes.
Yeah, absolutes are crap.
But what's really important here is that it was designed well.
There were a lot of really good controls that it was designed to have.
So it was built, the exterior was all concrete, right?
And concrete's pretty good at preventing fire spread.
There was meant to be an asbestos curtain that had a brass metal frame, a heavy brass frame, because most theater fires, when they occurred, they would occur on or behind the stage.
So if you could bring a curtain down and separate the audience from the backstage area, you could save a lot of lives.
So this was really a pretty good idea.
They had what were called ventilators.
Ventilators were really skylights.
They were giant skylights that had a mechanism that you could open, so that if there were smoke and hot gases and you had this problem, you could open them and that would get sucked up and outside and it would be a whole lot safer.
They also hired what they called a house fireman.
A house fireman was a man at this time who was a fireman who was hired to literally be in the theater during every production and be watching for problems.
So that seems good, right?
You have a human who's there who's watching.
It's not just relying on some of these other things.
There were more than 30 large emergency exits.
The Brooklyn Theater did not have a lot of exits.
That was one of the major problems.
So they said, yeah, we're going to go overboard.
There's going to be a ton of exits.
It's going to be great.
They had multiple fire escapes.
All theaters in Chicago at the time were required to have sprinklers.
Not only was this theater going to have sprinklers, it was going to have what's called a standpipe.
And a standpipe, if you've ever been in any construction, even recent in a large structure, there are these big pipes, and firefighters bring in hose and connect to them.
And there's one of a couple different ways it can work.
You either hook it up, and there's actual, when you open the valve, water flows, or it allows you to flow water into it.
So there's a couple different ways this can work.
But it adds to, so you have the sprinklers from above, and you have these standpipes that firemen could hook into to give them extra water.
And building inspections, right?
That was a thing that was happening in Chicago, and in theory, it passed building inspection.
This is the theater in 1903.
We'll come back to this picture, but you can see it was not a small place.
In fact, this was the inside.
This was a very opulent theater.
And one of the big differences beyond the safety features of the theater between the Brooklyn and this was the fact that it had a humongous stage, which you can't really see here, but the stage and the backstage area were actually larger.
This was a 1,600-seat theater.
Okay?
However, the day of the fire, there were more than 2,000 people in that space, and we'll talk about why that is in a minute.
So the problem is, there were a lot of assumptions about this building, and this is the result of those assumptions.
There was a very cold day.
It was December 30th.
December 30th in Chicago is when?
What happens around that time of year?
Well, it's close to New Year's, but what else is going on in general?
Winter, and the kids are...
they're on vacation, right?
They're off, they're on a break, and this was a matinee performance.
The bulk of the people in this audience were women and children who had come to see Mr.
Bluebeard.
Now, what I will tell you about Mr.
Bluebeard is that it was actually a very strange choice for children that the original play was about this fellow Mr.
Bluebeard who had killed multiple wives.
I think he had seven or eight of them.
And the eighth or ninth wife figures this out when she discovers the dead bodies of the rest of the wives.
Totally a children's play.
But they change it up, they make it child-friendly, if you will, by having the wives come back to life at the end.
I'm telling you, this was a huge performance.
People absolutely enjoyed it.
The theater was massively packed.
The reason why there were 2,000 people in that theater?
There were 350 people who were there just working.
So between the actors and the stagehands and the makeup people and, and, and, and.
So you have 1,600 capacity seats, you have 350 people working, and then you have the fact that they kept selling standing-room-only tickets.
Because standing-room-only was allowed at the time.
You very quickly get to over 2,000 people.
When all was said and done, 602 people were dead, again, most of them women and children, and 250 were injured.
It is and remains the worst fire in a single building in history.
It was absolutely devastating.
Okay, now that we've talked about that happy story, who here knows anything about the Irish healthcare system and ransomware?
Anybody here about that?
Okay, at least one.
So I'm guessing most of you don't.
That's cool.
So this is a different kind of disaster.
So this is the Irish healthcare system.
4,000 locations with 54 acute hospitals.
This is a big hospital system.
130,000 staff.
They had over 70,000 devices, different varied support levels of IT throughout those, and 350 IT staff.
So, you know, they had a lot of people doing a lot of stuff, and it was classified as critical infrastructure.
What do we assume is the case with critical infrastructure?
What kind of protections do we think are probably there?
Yeah, the good kind.
Not helpful.
Specifically, what kinds of things do we think, you know, should be there, right, in critical infrastructure?
Backups, multi-factor, firewall, EDR, right?
So what they did have, they had a security operations team.
This is good.
They had some kind of cybersecurity solutions providers because this is such a weird distributed environment that they may have had more than one.
It was sort of hard to tell reading the document.
There was a very, very detailed write-up that I'm taking all of this from.
They had an incident provider on retainer, so that's good.
They had AV installed, and they did, in fact, have firewalls.
So, I mean, you know, not terrible.
Unfortunately, they suffered a massive ransomware attack.
So this is what they put out on Twitter when it happened, and they were like, yep, we shut everything down.
Now I'll give you a little timeline of what happened here.
On May 14th at 1 a.m., the attacker executed Conti ransomware on six of the hospitals.
But the attacker had been there a while.
This wasn't their first rodeo.
At 2.50 a.m., there were notifications to the service desk about a bunch of systems that were suddenly encrypted.
That's a bad sign.
At 4.36 in the morning, they discovered not only were they hitting these systems, presumably workstations, now they were in the data center.
This is not good, right?
We all know when that happens.
And then, of course, at 7.28, finally they make this announcement, but at 5.10 is when they actually make the decision to shut the systems down.
So this is not wonderful, right?
We have a company that is in trouble.
So what happened as a result?
Well, they had to move to paper and pencil, and they were used to having access to all those records.
Remember, this hit how many of those hospitals?
Six.
How many had cancellation of services as a result of this?
31.
So just because the ransomware didn't hit the hospital doesn't mean it wasn't impacted.
So they had to cancel a bunch of things, diagnostic services and radiotherapy and elective surgery and, and, and, and.
In addition, they didn't have access to patient information systems.
Because they were doing all of this manually, it was really hard to identify people.
Like, how do you really know this is the right person?
I mean, how many of us have seen shows on TV or had personal experiences or know people where you're supposed to have surgery on one leg and maybe they were going to do surgery on the wrong leg?
Like, this shit happens in real life.
And that's problematic.
Well, they were dealing with way more of this than usual because they didn't, they had no way to even see those records.
They had trouble managing waiting lists for, for their, you know, their waiting areas and the ER.
They couldn't report results easily and, you know, prescribing medications.
Good luck, because you don't know what people were on.
You have to rely on them telling you or bringing you a bottle and showing you.
I mean, this was a mess.
Now, what I will say is that this hospital had been very good at dealing with issues like COVID.
They had gotten pretty good at being able to react and take care of things.
When they did go to the paper-based system, at least they weren't completely struggling because they were like, okay, you know, we, we've dealt with worse things in, with humans, right?
So we, we got this.
But obviously, this was still a pretty big problem.
All right, so I want to, I want to bring up a couple of terms and let's talk about them.
So when you see the word fireproof, what do you think of?
Won't burn.
Anything else?
What?
Can't hear you.
Yeah, no, not alcohol.
I understand the temptation.
So won't burn is, is really the most common thing.
So here's a cool thing that I'll tell you.
What if I told you solid objects don't burn?
What do you think about that statement?
Throw it in the sun.
So what I will tell you is that's actually a true statement.
Solid objects do not burn.
The way solid objects burn is indirectly.
You heat a solid object to a point where it gives off a vapor.
And once that vapor is hot enough, it will ignite.
What is burning is the vapor, not the object.
So solid objects do not burn.
However, when thinking about this term fireproof and this idea that things cannot catch fire or cannot be on fire, this is a misnomer.
How many of you have heard of fireproof safes?
At least a few of you.
Do you think that those safes will never burn?
They're time-rated, right?
That's the key.
So we'll come back to that idea.
What about this word?
Secure.
What does this mean?
Will or can't be hacked, right?
What else?
What?
It's an emotion, yes.
What else?
Can't be hacked, right?
So it's what?
Protected, right?
It's this idea, just like this idea of fireproof, it's kind of like the ultimate level of security, right?
It's secure.
It's the place we want to go.
And when we build buildings, there's this idea that we want them as fireproof as possible.
The problem is, and the themes we have seen with both of these stories that I told you, both about the HSE and about the theater, we often think of controls as preventions.
But here's the reality.
Controls are safeguards.
They're meant to delay disaster, not fully prevent it.
If you get lucky, you might prevent it, but most of the time, it's just going to delay things.
And ideally, if you can delay it long enough, that's a good thing, because it allows you to get in and do what you need to do before things get bad, in both security and in the fire service.
So we need to remember that our controls will fail, and that is the assumption that I want to drive home to you.
And in fact, even in my own fire department, I've asked them, I'd like you to run a drill, because we have a lot of new tools, right?
And this is true in security.
We get new gadgets.
We get new blinky boxes.
We get all kinds of cool stuff.
And we get really good at relying on that.
Who has EDR?
Anybody have EDR in place?
We get really good at relying on EDR.
We get really good at relying on these tools.
What happens when the tools stop working?
Do you have anything that even alerts and tells you it's not working?
How do you know if it's even working?
So I've asked in my fire department, for example, let's do a drill where instead of using the jaws of life, right, to do extrication from vehicles, why don't we say, okay, we're going to do extrication, and you can't use the jaws.
What are you going to do now?
Well, there are lots of things we can do.
We did it long before the jaws of life existed.
We used tools like air chisels, right?
I mean, we used all kinds of other things.
We didn't use the jaws.
Now, that doesn't say we don't want those as our first line of defense if we can, but we need backups.
And controls really require review, much like what I was saying about EDR.
If you don't know the answer to how do I know my tool is working the way it's supposed to, you have a problem.
You should be able to answer that question.
That is what offensive security, by and large, is designed to teach you, but there are ways you can do it on your own.
So these are the three types of basic controls we're going to talk about, and then I'm going to talk about one additional idea, but this is our foundation.
So we're going to talk about prevention, detection, and containment.
So using the case studies from the fire and from HSE and their ransomware, we're going to look at each of these.
So here's my little boy scout, right?
We want to always be prepared, so we want to prevent the bad things.
So in general, these are the kinds of things that usually we put in place, right?
So the fire service, building codes, inspection, education, InfoSec, firewalls, access controls.
Again, this isn't everything, but these are, you know, just some things we usually think of.
So here are some failures in terms of prevention in the fire service.
So that building that was built in 1903, yeah, there were a whole bunch of mistakes made along the way.
The biggest was that they were in a hurry.
They were in a hurry because the theater season kicks off in October, and they wanted that theater open by October.
And they got it.
It kept getting delayed.
There were all kinds of problems with strikes, strikes, things that didn't even make sense because it was like bricklayers in New York City, and what does that have to do with Chicago?
But it didn't matter.
The point is, they had one delay after another.
So they kept pushing and pushing and pushing.
Remember that curtain?
Well, that curtain had some asbestos in it, but most of it was wood pulp, and it never did have that magical frame.
So it caught fire like everything else.
If you look carefully, you will see this was the light that arced, that caused the spark.
That spark caught some gauze-like curtains.
Those gauze-like curtains decided to hang out with some scenery that was there, right?
And what do you think that scenery was painted with?
What?
Oil.
Yes, oil-based paints.
And when they brought that scenery in, remember the fireman I told you about?
He was like, get that crap out of here.
We don't want that here.
And they were like, but this is the show.
And he eventually relented, right?
So the flammable scenery gets allowed in, and then, remember I told you, all of this was done in a humongous hurry.
So the inspectors were told, hey, bring your kids to the show.
We'll give you free tickets if you just look the other way.
The man, the fireman who was there, actually did point out a whole bunch of problems, and we'll get to that later.
So in HSE, they had a lot of legacy systems.
End of life, this is a problem.
They had no patch maintenance, because that's always a good idea, which doesn't mean nothing was patched, but there wasn't any kind of consistency.
And in some places, yes, they were running antivirus in monitor mode, because that's helpful.
So if we analyze these failures, and you're going to see a repeat of this concept, so we're going to look at the cause, how we can resolve it, and some key takeaways.
So there was a false sense of security here, right, in both cases.
We thought the healthcare system was in good shape, because they were considered at the level they were.
We thought that theater had all those protections in place.
Nobody had verified the expected controls in either case, which is why everything fell apart.
They also assumed those controls would act the way they were supposed to.
They had the curtain, the ventilators were there, but they didn't work the way they were supposed to, and nobody was testing anything.
So how do you resolve that?
Well, obviously, you know, if they claim there's a control in place, if you think you have something that does a thing, not only should you verify that it, you know, first you want to verify it works, then you want to make sure, is it even really there?
Like, maybe there's something that works the way you expect the tool to work, but there's something else doing that.
And I've seen this kind of thing with firewalls, right?
Oh, it's being blocked because of the firewall, and it comes to find out, no, actually, it wasn't the firewall, it was some other tool, so you think the firewall is doing what it's doing because you see something's blocked, but in reality, it was some other tool blocking it,
and your logs don't differentiate.
Yay.
So you want to audit your controls.
This is your key takeaway.
Relying on inspect and controls is not good.
You need to verify and maintain them and test them all the time.
And I know that's a lot.
You have a question?
It does cost money, but there are some ways to, you know, automate things, and you can take little pieces.
I mean, you don't have to do it all at once, right?
Right.
No, I get it.
It is.
It can be expensive.
All right, so let's move to detection.
So in terms of the fire service, the thing we most know about, right, smoke and heat detectors and fire alarms.
And while this is not actually in the presentation itself and didn't play a role necessarily at the theater, just as an FYI, because I've been a firefighter for a long time, you should know whether your smoke alarms, what kind they are.
Because they don't all detect the same things.
Some of them detect heat.
Some of them detect ionized particles.
And some of them do both.
And the problem is, you think you have a smoke detector that will do the thing you want it to do until the fire happens.
And then you're like, why didn't my smoke detector go off?
I put batteries in it.
I pressed the button.
It worked right.
So just a little PSA.
Check your smoke detectors.
Know what you have in the house.
I always recommend dual to make sure you have that full protection.
Anyway, back to the show.
So in terms of InfoSec, some sort of intrusion detection and regular monitoring, honey pots, honey tokens, these are standard detection things.
So in the fire service in this particular case, there was no fire alarm system.
And not only was there no fire alarm system in this theater, at the time, has anybody seen the pull boxes that are in old cities?
There was no pull box.
At all.
So the only reason that folks knew there was a fire, and I'll go back to this toward the very end of my talk, that first picture that I showed you of the theater, it was already on fire.
In the upper left-hand corner, there was actually smoke showing.
So it wasn't until the firemen yelled and had somebody run down multiple blocks to actually do something about it that they found out about it.
And those sprinklers, yeah, they never actually installed them, and this fan pipe was never connected to anything.
That's a problem.
In InfoSec, this is a problem I see all the time that drives me nuts.
We see our tool has remediated things, and we go, oh, it must be good.
For example, my EDR detected Mimi Cats, and it stopped it.
Excellent.
That is what it should do.
Fantastic.
What does an attacker do when your tool finds a way to stop that sort of thing?
Try it again, and try it a different way.
There are a bazillion ways to run Mimi Cats, not just one.
So if your tools detect something like that, you should be looking for a secondary way in.
Look for attackers somewhere else.
There wasn't any kind of continuous monitoring.
Yes, they had AV.
Yes, they had folks who supposedly were watching things, but they had no formal monitoring thing in place.
They had, you know, no humans that were actually looking at those logs, and they assumed that the AV was working the way it was supposed to, you know, blocking things, not in monitor mode.
So let's analyze these failures.
Again, a false sense of security in both cases.
The tools were not working as expected, and we didn't verify those controls.
And in the case of the AV, right, it was misconfigured because no one ever actually put it in block mode.
So how do we resolve that?
Well, like I said before, verify those controls actually work, and do regular testing, and verify those tools work the way you expect them to.
So again, detection systems, you got to verify them.
I know this sounds redundant, but that's part of the point, right?
I'm saying the same thing over and over because we don't do it.
What about containment?
This is one of my favorites.
So in the fire service, we have fire extinguishers, sprinklers.
We have what are often called fireproof walls, but they're really fire-resistant walls.
Like we heard with those safes, they are time-rated.
They are meant to allow people to escape or get out before they fail.
And in InfoSec, in theory, we have incident response, and HSE did have IR on retainer, and when they did figure out they had a problem, they did call them in, and it was Mandiant, and they started working this case.
If you don't have a way to isolate systems, that's not ideal because that's another thing you want to be able to do.
So you should ask yourself, if you know a system is doing something bad, do you have a way to isolate that in any case?
So in terms of the fire service, this is fabulous.
So look at this little stick thing.
This is cool.
This is what they gave the firemen.
This is a fire extinguisher.
And you're like, what?
So this is called a kill fire, and it is really meant you throw some powder on a small fire and it puts it out.
It is not meant for the fire to be up on top in a curtain.
So, you know, this poor guy is, like, throwing...
he had, like, six of them, and he's, like, throwing the powder.
It's doing nothing, right?
Wrong tool for the job.
He wasn't given any other tools, just those.
Those ventilators that were such a good idea to allow those heat and gases to escape, yeah, they never actually finished them, so they just nailed them shut, because otherwise cold air would get in, and I did mention it was winter, right?
Yeah.
I mentioned the standpipe never connected, and there wasn't a sprinkler system.
So there was no way to contain that fire.
So in terms of InfoSec, well, they had a flat network, a completely flat network, which is not cool, and so they had no visibility at all beyond certain specific systems.
Again, I mentioned the monitor mode, but they also had a problem with communication.
They had no governing structure, no CISO, no, you know, nobody in charge of security.
So there were all these, like, disparate little IT community bits where none of them were talking to each other.
So what I didn't cover is that there were, in fact, hospitals that detected some badness and prevented this from being even worse in their environments, but it was only their environments, because who were they going to call?
Because there was no central place to call.
At one point, their security operations folks called when they saw active, active activity from an attacker in a system.
They called the sysadmins and they said, I want you to reboot the server immediately.
Boy, that was effective.
Yeah, like, what?
So clearly these folks hadn't gotten the training they needed.
It was discovered later on that they found at least 16 systems with issues that were never, ever addressed.
And again, you know, you assume AV, EDR is going to save you in all cases, you're making a huge mistake.
So we'll look at the analysis of the containment space.
Obviously human error here played a huge role in all of the things.
That fireman tried to tell people, Houston, you have a problem, because he walks through the theater and he's like, he clearly can see there's a problem.
But here's the thing, he'd been working for a different theater at one point or some other facility, and he'd been fired because he was too aggressive.
That's what they called it.
He was too aggressive in talking about what the problems were that he discovered.
So he tried to tell people, but he was like, I don't want to go too far because I don't want to get fired.
Instead, you know, all these people die.
Yay!
Inadequate design, right?
So in the HSC case, we certainly have a situation where a flat network is going to be a disaster.
In the case of the theater, you know, it's well designed, but not as well implemented.
But we're going to see there were some design flaws, too.
Again, lack of verification.
There were inconsistencies in tool configurations.
Some places had AV set in monitor mode.
Some didn't.
And again, the wrong tools are provided.
So how do we solve that?
Well, again, we verify those controls actually are there.
Don't just assume because someone tells you they're there.
Go look.
If you're supposed to have some sort of EDR on every system, go and look.
Is EDR on every system?
You're going to need eyes on that stuff.
And at a minimum, you can look right in the tool itself and see, am I missing any systems, right?
And that might require multiple people looking at this.
You want to make sure you have the right tools.
Because if you don't have the right tools, and they don't have to be fancy, this is why I brought up the whole fire service and, you know, this business about let's try a drill without the fancy tools.
You might need a not-fancy tool to do the right kinds of things.
So make sure that you have the right tools, whatever they are.
So containment is all about having those tools and education.
Your folks need to know, if they see something like Mimikatz in their environment, you don't just go, yay, it was detected and caught and we're done.
And you need some sort of solid communication plan.
Otherwise, you will never contain the badness.
Not really.
Okay, so this isn't a formal control, but I think it's worth discussing, is that in both cases, there were delays and failures when responding.
So the controls have now all failed in these cases, but you still need to respond.
You still need to do something.
So this is funky.
These are doors.
These were some of the doors in that theater.
Look at this weird-ass locking mechanism.
Now imagine you're in a smoky environment and you've never seen that before.
Good luck getting the hell out of here, because you're never going to figure this out.
And there were a bunch of double doors like this with these weird configurations.
There were draperies covering the exits, and no exit signs, because exit signs are ugly.
So we don't want any of that.
This is this brand-new fancy theater.
There were, in fact, locked doors.
And who's seen Titanic, the movie?
Yeah, good number of you.
What keeps people getting out from steerage up to getting on those lifeboats?
Gates.
Well, guess what this theater had?
They had gates, because they didn't want the people in the upper levels, in the cheap seats, coming down and sitting in the orchestra.
So as soon as the performance would start, they'd pull those gates across to keep those folks where they should be with the ticket they purchased.
Not good.
It was also, as I mentioned, December, and it was cold.
So weather played a factor.
Nobody was trained on what to do in an emergency, much less a fire.
So you have all these people who are ushers and actually working with the public, bringing them in and seating them.
Not one of them had been prepared for what to do in an emergency.
That was very evident.
So in InfoSec, we see this all the time, right?
There's no incident response plan.
Have something, even if it's rudimentary.
Just think about it.
I mean, it doesn't have to...
Yes, ultimately, you will want a formal incident response plan, but that can take time to develop.
So just come up with something, even if it's a basic game plan.
If you know that something bad happens, who do you go to?
How do you deal with it, right?
Something really basic.
As I mentioned, with the HSC, there was no formal security oversight at all.
No CISO, nothing.
Clearly, there were inconsistent levels of knowledge because some of those hospitals had people that recognized there was a problem and actually knew what to do about it and prevented the badness.
And then you had the ones who were like, yes, there's an attacker there.
Just reboot your server.
You'll be good.
There was no out-of-band communication mechanism at all.
So when they're all down, now how are they supposed to communicate?
Think about this long before you have an incident, not in the middle of it.
And there was no SOC, so there was nobody to call and nobody to escalate it to.
So in terms of analyzing that, again, missing significant resources in both cases.
Communication played a role in both the fire and what happened in HSC, right?
How do you communicate, hey, there's a fire, without, like, standing up and screaming, hey, there's a fire.
And there was no communication to anywhere else.
So this is fun.
Not only was there no firebox, there was no telephone in that theater, no telegraph, nothing.
So literally the only reason anybody knew there was a problem was the person who ran up the street to the firehouse, which was horses, was horse-drawn carriages back then.
So, not good.
And of course, architecture flaws, right?
Staircases.
We have a situation where a flat network means it's going to blow through a lot of it.
And in the case of this theater, there were also some interesting architectural flaws in terms of where staircases were.
And so there were points of egress that were very tiny and a lot of people wound up in front of them.
So have some sort of clear plan, even if it's not fancy.
Please educate your folks on the basics of IR.
They do not need to be DFIR specialists, but they do need to know who to call and they need to know when they should call somebody.
Not just, hey, we got an alert and it's resolved.
They need to understand, hey, we got an alert and it's resolved, but maybe this is more serious.
And I know that for some folks it's like, but what about our SOC 1 people?
Our level 1?
Yes, they should know when they see this stuff.
They should, you know, hair on the back of their head stand up or something.
They should recognize this.
And a lot of folks do not get the training they need.
Documentation, I'll just leave it there.
Everybody understands that.
And you need a way to impart critical information quickly.
If you want to have effective incident response, you've got to plan, you've got to drill, and you need to have access to stuff.
So let's talk about some lessons learned from both of these.
I think this is particularly interesting.
These were things that the fire service learned from this fire.
Anyone heard of UL, these stickers you see on all the electronic crap that we buy except some of the stuff that was made in China?
UL was developed by a man who was inspired from seeing this horrible fire.
That is why UL exists.
You know the crash bars we have here and here?
This fire is the reason those crash bars exist.
Doors were locked in theaters because they thought they didn't want outside people coming in emergency exits when they didn't have a ticket.
So they would lock the door.
Well now, you can do that, but people can still get out.
But that's because of that fire.
By the way, this is the inside of the theater.
This is on the stage.
I purposely did not put any pictures of the 602 bodies, but there were quite a number of photographs, even from that time period.
Exits have to be clearly marked as they are in this auditorium, and not only do they have to be marked, they have to be on a separate electrical system.
So if this building were to lose power, they should not.
They actually have to be on a completely separate thing.
And that was something they learned from this.
And you have to have doors that are easy to get out.
Those doors were crazy.
And the fact that they had the gates, not cool.
So what did HSE learn?
Oh wait, a little more here, I almost forgot.
So they also learned that they needed to use some sort of fireproofing type solution on things like backdrops, not just use oil-based paint, because that's bad.
There should be a fire alarm.
It should actually be connected to the local fire station.
That is not always the case everywhere.
Some are independent.
But this is something they learned from this particular thing.
There are ventilation standards now to prevent the mess that happened here.
And maximum seating capacities were changed.
There's no more standing room only in most theaters as a result.
All right.
So what did HSE learn?
Well, they added a 24 by 7 monitoring service.
That was a good thing.
They added a temporary CISO and CTTO so that there would be somebody to actually coordinate effort.
They actually segmented their legacy systems, which was smart.
And they added some control platforms.
They had their folks go through some security awareness training.
They did things like restrict access to those authorized to use things.
Shocking!
They put in processes for communicating in some kind of incident this way.
And they added incident response playbooks to go with that.
Now, I'm telling you, this didn't happen quickly.
And they also added a bunch of technology.
So they now have a SIM and UEBA and SOAR and automation and vuln scanning and all the things that they probably should have had.
Given they were considered critical infrastructure, you would have thought these things were there.
But you would have thought a theater would protect its patrons too.
So in the end, we need to prepare for failure.
We need to assume our controls will fail.
We should always be thinking, I put this control in place, it's awesome.
Now what happens when it fails?
What's the next thing?
What am I going to do?
In the fire service, we do routine fire drills.
We make sure that kids know how to get out and stay out.
Who remembers stop, drop, and roll?
Right?
When is that useful?
No, when is it useful?
When you're on fire.
How often are your little kids on fire such that stop, drop, and roll is a good idea?
The fire service figured this out and they've started teaching things like get out, stay out and having exit drills and having plans, right?
Because, what a disaster, right?
Kids, you hope they aren't on fire.
That's really not, you know, that would be the worst case scenario.
And there should be inspections to be looking for those weaknesses.
Do something like a catch-me-if-you-can TTX where, you know, an attacker gains admin somewhere and they're trying to persist.
What are you going to do, right?
Some basics.
Can you, as an organization, deal with that?
Just, you know, on the basics.
Stephanie, you had a question or a comment?
Okay.
So, in terms of InfoSec, again, these TTXs are critical.
We should have playbooks for some of the basics that we see all the time.
We should be running drills, looking for what we would do.
We should be doing some kind of detection testing.
Do our detections do what we think they should do?
I have another talk where I talk about the fact that detections are often not doing what you think they are, not because they're not working properly, but they're working at a different level than you think they are.
So, for example, when you detect an attacker doing password attacks using LSAS, there are a bazillion ways to do that.
So, if you have a tool that detects that, it probably detects a particular technique.
That's fabulous.
But what exactly does that mean?
Well, it means that's one technique, which the way you do that technique could be any one of a hundred different ways.
So, it really is just one tiny thing that you can detect.
So, I'm going to wrap up here.
And across the disciplines, I think you can see that we can learn from the fire service.
We've got to have those layered controls, right?
And we need to test them.
We need to know they're there.
We need to validate that.
We need to make sure we have these response plans.
We want to make sure those communications are there.
We need to be adaptable and flexible.
And any kind of proactive, resilient approach is going to be required.
As the military frequently says, improvise, adapt, and overcome.
So, my call to action to you is adopt this mindset where you assume your controls fail and to think ahead about what you're going to do as those controls fail.
One of my biggest pet peeves is when somebody in our network team says, yep, but they can't get to it because.
Great.
Until they do.
So, I always like to end with this quote because I actually got my degree in music business.
I did not get a degree or anything in IT because there was no such thing.
So, ultimately, I may not have gone where I intended to go, I wound up in this space instead, and I think I wound up exactly where I needed to be.
And with that, I will also leave this with you.
I wrote a book called The Active Defender which teaches folks how to think about defense from an offensive perspective.
If your folks do not understand those basic concepts, they're missing half the part of the security story.
And I'll have a bunch of flyers up here that have some more information about the book, if you're interested.
I want to thank CypherCon for having me.
I want to thank you all again so much for being here this late.
I will take, if you guys have a couple of questions, I'm happy to take them now, or if you want to come up, either way.
Thank you.
Any questions?
I know it's late.
All right, I will leave it at that, and feel free to come up and see me because I think I'm the last thing of the night.
CypherCon.
www.cyphercon.com