Good afternoon, everyone. My name is Michael Kafka. Everybody calls me Shecky. If you want
to find out why I got the name Shecky, see me after the talk. It's a long story. Wait,
there can be only one. Interesting title, huh? Expect a lot of Highlander references?
Maybe a couple. What we've got is... let me give you a little bit about myself here. Hold
on. You test beforehand, and it doesn't go, and now it does. Come on. We'll start off
a little bit about me. I've been in the field of professional for seven years. I've been
in IT for over 25 years. That doesn't include a period of time where I was just learning.
I started programming computers back in the early 80s. So I've been around computers for
a very long time. I'm one of the organizers of one of the Birdsects. If those of you who
are down in the Chicago area, you might have heard of us. We're a weekly, basically, meetup
group. We have chapters, meetups, all throughout the Chicagoland area every Thursday. Each
area gets one Thursday per month. I happen to be one of the organizers of the North version
of it. We average about 15, 10, 15 people. So if you are in the area on the second Thursday
of the month, feel free to come out to mine. Or any Thursday, feel free to look up Birdsect
and meet up, and you can find out where we're meeting that week. I volunteer for Hack4Kids.
I have for a number of years. I'll be volunteering tomorrow morning for the Hack4Kids. I enjoy
seeing the children going ahead and getting introduced to all the concepts, learning,
destroying stuff in the destruction village. There's my website, if you're interested.
I blog, but I blog infrequently. Why infrequently? Well, because like the rest of you, I've got
other things on my mind half the time. And finally, you can find me on Twitter. I do
a lot of tweeting. I try to keep it more professional, I would say. I try not to tweet
as much about, say, politics and stuff like that, as some people out there are trying
to keep it more infosec or personal-orientated, comparatively speaking.
So why can't there be only one? What is the whole deal? Well, the one thing is related
to defense in depth. What is defense in depth? The idea of using multiple layers to make
things more secure. It's a very simple definition, and we'll go a little bit more into it as
we move forward with the presentation today. And how are we examining it? We will be examining
it through an actual real-world situation that I was involved in last year. The data
that I found, the reasoning behind defense in depth and how defense in depth actually
relates on downward, beyond what your general thoughts are on it, is actually rather interesting
because it goes through a lot more than you might realize. We will be talking about a
There will be a couple screenshots. All the data has been sanitized, so nothing really
to worry about. You're looking at defense in depth. This is your typical look at it.
You've got your data layer, you've got your application, your host, your internal network,
your perimeter, your physical, and your policies, procedures, and awareness. It's your basic
idea behind it. It's the castle theory. You get your moat, you get your castle walls,
you have your people right behind that, you have your people in front of it. You get through
one layer, the next layer is hopefully going to stop whatever is going on. But defense
in depth goes beyond just these basic layers. And what this diagram doesn't do, it doesn't
go into the depth of the depth. In any of these layers here, you can actually re-layer
things. Maybe we might not have an area of host or internal network. You've seen flat
networks before. Perhaps our policies and procedures are lacking a bit. Maybe it's because
of budgeting, maybe it's because of lack of resources physically and people. But everything
goes through there, and it just layers up. And in each one of these layers, you can have
layers, like onions. Onions have layers. Yeah, I know people prefer parfaits, but onions
do have layers. I know not everyone likes ogres or onions, and you might prefer cakes
or parfaits, but we understand the idea of the layers, but it's not as easy, and it still
isn't, because you can have sub-layers, or layers within layers. The reality is, with
sub-layering, it's usually done at a vendor slash product level, as opposed to a different
type of technology, per se. How many different technologies, on average, do you think your
company uses? Three, four vendors? Five? More? How many of those different technologies,
or layers, wind up going ahead and crossing over with other ones? Do you have an EDR that
does web-blocking? Do you have a firewall that's doing IPS on top of it all? Where you
have another thing going ahead and doing IPS, and then you've got a separate web-blocking
situation. So you get this whole strata of different areas. We used to hate the idea
of using multiple vendors in these layers, as it is. We all remember back in the day
with Norton versus McAfee, versus Avast or any of the free AVs out there. How many AVs
can you have on a machine at once? How many times did those interfere? Did anybody run
into that, where they interfered with each other? Plenty of times. Nowadays, it's not
as prevalent. You have multiple vendors sitting out there, and they're not really interfering
with each other. Or if they are, they're interfering in a way that we don't just quite get, and
it drives us crazy. Yeah, it still happens though sometimes, where they interfere with
each other. But we'll talk a little bit about that later on. So the scenario that I've got
for you is very simple. It's a website. How many people have web-blocking technology out
there? Use Cisco Umbrella or whatever, or Bluecode or Zscaler to go ahead and help prevent
people getting from sites to sites that they aren't supposed to be getting to, including
malicious sites. How many people out there have an EDR? Feel free to raise your hand.
Alright, so one person has an EDR and nobody's using web-blocking? Well, if you know anything
about it when you're dealing with a website, it came from the net, and eventually it was
blocked. And then you go ahead and say, alright, so we blocked this site, we blocked that site.
Don't think much about it. You don't think about what technology is actually blocking it.
Is it the web-blocker? Is it EDR? Is it something like you blocked origin inside of the browser
that you might have distributed out through your company? And why does that matter?
I ran into the situation with mail.ru and its subdomains. They were being blocked and
alerted on by Microsoft security tools in some machines, and Cisco's umbrella in other
machines. Does that make sense to anybody? Hands? Or should I have just been banging
my head against the wall like a crazy man? Would you have banged your head against the
wall like somebody that was just ready to give up? I see a lot of head shaking out there.
So, I ran into a, wait, what? And I didn't think too much about it. Both technologies
would block. There was a reason why one was blocking over the other in different instances.
But why was it happening? We'll start off with the mail.ru. Has anybody ever heard of
that site before? Alright, we've got one person. You know, it's basically like a generic
casual Russian type site. There's legitimate stuff on it, there's illegitimate stuff on
it. A lot of illegitimate stuff on it that's hosted on it. There's also ads that are hosted
on it. It's not really something that most people are going to wind up wanting to go
to from our country or from most corporations, as it is, because the stuff on it is just
what it is. It has some legit things on it, but during my research on this as to what
it was and why it was being blocked in the first place, alienvault.otx.alienvault.com
went ahead and listed out about 20 different malicious items that had been found on that
site over the course of time. And that was just a quick look that I did on it, just to
get an idea of what I was dealing with in here. As it turns out, the initial block and
the initial reason somebody went there was somebody that was an employee that was originally
from Russia and had their own personal mail account on mail.ru. And we looked at them
and said, yeah, you can use your personal devices and your personal machines for that,
not a corporate device. Thank you very much. And, yeah, we asked to it. But then we were
still getting these weird blocks from it. And they were ads. And we're going, okay, so
why is it being blocked one way and not the other, now that we've put this into an actual
block list because we don't want people going to it. So, we found that there were layers
sitting inside of there. The web blocking had layers. And what was it that was blocking
it? It wasn't just the differing technology that we were running into. It was because
of how they were layered is why one was blocking over the other. But that still doesn't make
complete sense, does it? Why would you go ahead and still get, why would, if you've
got the technology in place on two different machines, why would one use one technology
and one use the other technology? Both machines make the exact same way. Well, it's a matter
of knowing where in the stack, where is everything layered. Now, why is that important? You're
on an incident. You've got to do a response on this. You're going through all the logs
and everything. You see that something was blocked. Why would it be important to know
when in the layers it was blocked? Well, it would give you an idea of how fast it was
blocked, what's the chance of it spreading to other machines, what's the chance of something
else going on beyond it, and what technology was actually doing the hard work on it all,
so that way you can start digging into that technology more to get more answers. Not a
bad idea, huh? How many people out here understand what their layering is? Understand if they've
got multiple technologies that can do something, which one comes first, the chicken or the
egg? It's not something that we think of very often, is it? So, let's take a look at
this. This right here are two separate screens. You'll see the one that says Destination Mail
RE, that is from the Cisco Umbrella Portal. You'll see there that it's categorized as
News Media, Portal, Search Engine, Webmail, and our Internal Block List. To the right
you'll see Microsoft Defender blocking, alerting on the site being blocked by Network Filter
Lookup Service through Firefox. We put this into the Cisco Block List, into our Umbrella
Block List, because of the research that I did and Microsoft was naturally trying to
block it. So we figured, well, everybody should get the pop-ups through the corporate screen
saying, we blocked the site and if you think that's incorrect, please report it to us and
we'll take a look into it. That was the only way we could get that to go, but not everybody
was getting it when we started testing. It is real interesting that it says Network Filter
Lookup Service, as opposed to a Web Block. Now, Cisco's Umbrella works off of what used
to be called Open DNS. They use categorizations, they use simplified systems to go ahead and
allow you to set up what sites people can go to, what sites you're going to block down
to, and it doesn't always give you 100% information. Now, while we were working through this, they
actually made a couple of changes inside of the portal. The first question that we
had was, is mail.ru even making it to the Cisco interface? We go in there and we see
it says DNS next to it, and no, I don't have screenshots of this just because there was
proprietary information inside of there that I cannot reveal at this point in time. But
it would say DNS. So when we got on board with Cisco to start looking at this issue,
why wasn't Cisco blocking it? Why was it being blocked by Microsoft? They said, well, the
simple thing is, is that yeah, it's showing it, but it's just a DNS lookup that's going
through and we're not seeing anything else beyond that. At least on the machines that
we weren't getting the block screens on. We'll take a look into it maybe, but there's got
to be something else going on. Not a lot of great information, except for the fact
that even if you've got a website and a URL, it's still going to go through and at least
hit the open DNS servers, the umbrella DNS servers, before it does any block. And that
starts breaking down to, okay, so now we know when I type in a website, we'll go ahead and
we'll take a look at that site and we'll see what the IP address is. We'll get that IP
address first. And then when he tries to go ahead and make a connection to that site,
we'll go ahead and we'll block it through the web locking technology inside of it all.
Okay, makes sense. Would you agree with me on that? Well, the interesting thing is, is
that we're still going ahead, not any closer to understanding why Microsoft was doing this.
Why did it need to be in a manual block listing umbrella? Well, it's a matter of threat
intelligence. Microsoft has their own threat intelligence. Cisco has their own threat
intelligence. AlienVault has its own threat intelligence. Palo Alto has its own threat
intelligence. Threat intelligence is its own threat intelligence. There's no standardization
and there's very little communication between a lot of these companies. So they'll go ahead
and do their best analysis on it. And the analysis boils down to, in a lot of cases,
what a human being will say or what a machine learning algorithm says. Once it passes through
one, the other one takes a look at it and makes a final decision. At least that's how
I've seen it work at this point in time. So we decided to trust Microsoft and put it
into the block list. We knew we weren't going to be stopping anything that we needed to
worry about from a corporate standpoint. We weren't going to be affecting anything from
a business standpoint. It was all going to still be there. And we weren't going to get
any complaints. And sure enough, we put the block in there and we still don't get any
complaints. And yes, I still get these alerts from Microsoft Defender to this day occasionally
if there's an ad being served from mail.ru. You would think that adding it into the umbrella
block list would be the end of it. We thought so and we were wrong.
That does say wrong there.
So what we started to do is, well I started to do the testing. The reason why was I
was asked by the CISO, why is Umbrella not working the way it's supposed to be working?
Or is it Microsoft Defender that's not working the way it's supposed to be working?
One or the other is not working the way it's supposed to be working. Why are we getting
to? So it was tasked to me to go through it. So I started testing mail.ru and enforcing
things. First off, through different browsers. If it was getting blocked by Umbrella, we
would get an umbrella block in all three browsers. Okay, so we know that it's not a
browser-based situation. If it was getting blocked by that network filter on Microsoft
Edge, you would get the smart filter screen. Has anyone here seen that red smart filter
screen? I know that Edge is not always very popular. But it's a nice bright red screen
that says, basically, we're blocking this because of security concerns. But if it was
being blocked in Firefox, like you saw in the slide, or Chrome, or Brave, or any of
the others, all you get is a white screen with a little bit of black text that says
access denied. Well, that tells us a whole lot. Both screens tell us a whole lot, don't
they? Access denied can be just about anything. So, the question is, which one is it going
to actually hit first? What is the order? And order is important because security is
knowledge. Knowing what should be doing what, when it should be doing it, helps you find
out if you've got a misconfiguration or a vulnerability sitting out there that somebody
is going to slide through, or potentially can slide through. If you don't know all of
it, you might end up chasing your own tail forever, which, as you can hear, is what I
was doing with this whole situation. So I was doing these tests. And it made me learn
about both technologies in play. The first thing that I did was I found out that uBlock
Origin, inside the browser, if it's an app that uBlock Origin has in their block list,
because that's what they use as a centralized block list up on the net, that's going to
supersede Umbrella, it's going to supersede Microsoft, it's going to supersede anything.
It's going to see that URL that you punched in and say, hey, this is on my block list,
it is blocked, boom, done. So there's your first layer right there, uBlock Origin. But
as we all know, that is dependent upon things being reported in uBlock Origin, and however
they decide what sites are good and what sites are bad. And it's mostly ad-based sites that
uBlock Origin blocks anyway. They don't necessarily block malicious or anything like that.
You also know that that could very easily be bypassed. It's very easy, even in corporate,
to go ahead and just turn off uBlock Origin to go ahead and allow a site to work the way
that they originally intended it to. We've all done that. We all know our users have
done that as much as they say, oh no, I wouldn't do anything like that. We all know that that's
not true. So let's start taking a look at it.
Cisco Umbrella. Pardon me for a second. I've got to see how the screen looks. I have some
slides here where things will pop up and some where they don't. Having the screen behind
me is a little bit different than how I'm used to giving a presentation. So Cisco Umbrella
was formerly OpenDNS. They allow successful DNS lookups before blocking. And it uses its
own intelligence, Cisco Talos, to go ahead and categorize sites. Basically what Umbrella
is, is a machine in the middle attack. If you really think about it, and it becomes
more evident when there's a problem with the Umbrella root certificate, which I've seen
happen before where the root certificate didn't get installed onto a machine, and
your browsers nowadays with how they're going off about security and certificates, you all
of a sudden get a block in Firefox, you get a block in Chrome, you get a block in Edge
with a security warning on it saying, yeah, this is not a valid certificate. So you know
that there's a middle section there that it's going through. It also means that the
DNS lookup happens before any sort of blocking happens. Now we've got part of what's going
on. So we understand the basics, and that's all that you need to know is the basics of
how Umbrella is doing it. You're going in, you're typing in your website, it's going
ahead and shifting everything over to its DNS and using its certificates, and then allowing
everything to go through or blocking it. Simple, concise, almost pretty in a way.
So then you get into the Microsoft network protection. And as I said, it was Microsoft
Defender for Endpoint that was alerting on this. Network protection, we found out, bases
and things on Microsoft's Intel. Big surprise there. But it has to be turned on. And where
does it get turned on is the next thing. It's part of the exploit guard series of ASR rules,
or attack surface reduction rules. So if I tested a machine, if I tested an ALR on a
machine that did not have those rules applied, but did have Umbrella applied on it, the Umbrella
block screen would show up as expected. If I had a machine that actually had those rules
turned on, well, then we get into a different situation. But again, this is all through
Defender that we're looking at. Now those of you that don't understand, Defender changes
itself every three months right now. And it gets better. And it gets better. But you still
have to go through a lot of these queries to figure out what's really going on behind
the scenes. So we know it's part of exploit guard. We know it's turned off by default.
It's enabled through group policy or Intune. Now why do I bring both of those up? Well,
we'll get to that in a moment. But wait, there's more. It uses Microsoft's threat
and tell, and it will give the smart screen message if you're using Microsoft Edge. And
finally, it will just give you that access denied if you're using a non-Microsoft browser.
Finally, it does alert in Defender. So we've got all this information. So we now know that,
well, if we have exploit guard turned on on a machine, that seems to go first. If we don't
have exploit guard turned on in the machine, it hits umbrella. So we're getting somewhere.
So we now know the order of operations. So the question is, why is exploit guard not
turned on in everything? So Microsoft's protection works similarly in the sense that it runs
off a pure IPS DNS response, and it's not... network filtering is not a web-locking technology.
I don't decide categories on that. It is purely done off of threat intel. If it blocks,
it says either that the site's IP is good or bad, but there's always some network traffic
involved with it. But there's less network traffic, and it's sitting closer to the kernel.
Cisco was the third layer. So if the other two aren't working, then umbrella will wind
up going and blocking it, or if it gets through these. Now we're getting to a point of going,
now I understand why the layering is important. And we're not having technology interfering
with each other. Instead, it's complementing each other. Isn't that what defense in depth
is supposed to be? Isn't that what we're looking for? Is if something fails, we've got something
else to back it up? And now we're doing it with two separate, completely separate ecosystems.
A Cisco ecosystem and a Microsoft ecosystem. And they're actually working together, even
though they don't directly communicate with each other. Defense in depth is layers. So
the ultimate reasoning behind it all, and the final portion of the layers is, so why
were it was only some machines? Well, when you're dealing with a hybrid situation where
you have an on-premise domain and an Azure domain, unless you're joined to both of them,
everything does not get pushed out through one technology. Azure joined machines do not
get GPOs pushed to them. Everything goes through and to. This all started up during a period
of time where we were starting to implement the ASR rules, including exploit guard and
network filtering. We wound up in a situation where machines that were part of the rollout
already, which was mostly on-prem machines joined to the on-prem domain, were getting
those ASR rules. So they were blocking it through Microsoft network filtering. The machines,
the laptops that were being taken home that were part of the Azure joined domain, they
weren't getting that protection yet. Which gave us the time to go ahead and say to our
other teams that were in charge of pushing out these protections, why aren't you pushing
these out yet? You've pushed them out to the most sensitive of the machines and the
machines that would break the easiest. Why have they not gone out to everybody else yet?
To which we got the typical, we haven't had time to go ahead and finish testing them and
we don't have enough resources to do it in a super fast fashion. We'll get to it within
the next month. And they slowly have gotten to it and things have gotten pushed out. In
the meantime, we turned around and said, well, thank God that we've got these multiple
layers sitting there. Shows flat out that defense in depth does work. And it works very
well if set up and configured properly. But again, whether it's the overview of different
technologies from physical to policies, or it's inside of an endpoint solution or solutions,
if you don't know where things are going to happen, you're just going to drive yourself
crazy going, well, or you're just going to say, well, it's blocking, who cares? But somewhere
down the line going that, who cares, can turn around and bite you in the butt when you have
an actual situation. As to where Ublock working comes into play, it blocks ads nicely. I said
it happens before Microsoft's technology. And it looks at the URL name and compares
it to its own list. There are general layers. There are sub-layers. Layers can be different
technologies. So layers upon layers upon different technologies. And sometimes inside of the
same technology stack, you can have different layers sitting there. Microsoft Endpoint nowadays
actually has its own web-blocking feature inside of it. I have not tested it thoroughly
because it is still in its infancy and the way that it works doesn't lend itself to the
corporate situation that I am in at this point in time. But I check it out every so often
just to see if they've gotten it to a point where it's worth me completely testing on it.
Cisco has multiple technologies. They're always acquiring stuff. You can have layers of Cisco
technologies and in between layers of somebody else's technology in there. You could be using
Duo with a Palo Alto firewall. The final idea though is to have the layers complement each
other. So when you're actually doing a POC, when you're actually testing, you have to
make sure that any machine that's being tested on has your current technology on there minus
whatever technology is being replaced, if it's being replaced. So if you're trying to
remove from something say like Defender from Endpoint to Carbon Black, you remove Defender
from Endpoint from the machine but all the rest of your technologies have to still be
on that machine. Two reasons why. Number one, you'll see where things hit. Number two, it'll
make sure that you're not interfering like in the old days because that does happen at
times. It's an instance where you need to go ahead and figure it out and that's part
of our job. Everything is not all glam and not all writing up whatever you're writing
up scripts, thread hunts, scene queries, however you look at it. Each one is just a greater
part of the whole but you need to know how they fit into the whole like a jigsaw puzzle.
There can be only one and that one is order. It is the order of operations, it is the order
that things hit, it is the order that you implement them sometimes. But that's the only
one consistent thing on it all, is that order. If you're going to do Defense in Depth properly,
you need to know what that order is. Otherwise, it's basically spitballing it or throwing
a piece of gum at a wall. Maybe it'll hit this, maybe it'll hit that, I don't know,
maybe it hits here. And then when you have to go ahead and support that technology, you
go, well, I don't know, and then you start wasting time trying to figure it out at that
point. The vendors themselves are not always so helpful in being able to diagnose this
sort of problem. And that's where we can really get better, is if we can get them to all work
together with us. We see this on a regular basis. How many people here deal with scene
technology? Logstash, Slipknot, what have you. How hard is it to go ahead and get things
parsed properly into that scene? Anybody? Anybody have an easy time with it? How long?
It's a mess? Syslog. How many items go in there that are Syslog? In different formatted
Syslogs. In different shapes. You get the Linux, and then you get the Cisco, and then
you get the Microsoft, and then you get this coming in, and you have to write up... I used
to deal with Raylong this way. I had to write up parsers for five different technologies
on Syslog, and a way for it to recognize which technology was coming in. That took me six
months. Six months of testing, and checking, and testing. In the meantime, you're getting
overfilled data on it. Why? Because everybody wants them in their one silo. We need to work
together. And it's not just the corporations, such as Microsoft, or Palo Alto, or Cisco,
or Carbon Black, that need to work together and come up with a standardized way to get
logs in, and standardized fielding, so that way it parses properly. Because all that's
part of defense in depth in its own right. That gives you the views that you need to
see. But if you're not getting the fields that you need, or not able to easily see them,
you're making your work two to three times as hard. But we also need to be able to share
out these sorts of information. If you make a parser for Cisco Firepower, that parser
is going to be different than Cisco ASA, by the way. I went through that. Inside of Cisco,
ASA's syslog is different. The fields are different than Firepower's. Probably because
Cisco bought Firepower and developed ASA. And the two teams don't necessarily communicate
the best together. If we can share that information, if we can share something like this, where
now all of you know that if you're using a Microsoft stack and a Cisco Umbrella stack,
Microsoft's network filter is going to supersede anything that Umbrella does. That way, when
somebody comes back and says, why isn't Umbrella blocking this? Why are we getting an alert
from our EDR on this? Well, that's because Microsoft Threat Intelligence feels that this
is blocked. Or we put it into a block that's inside of our EDR, and our network filter
hits first. It goes ahead and says, oh, this is this? Fine. It hits before Cisco finally
does. Well, we've got the same technologies everywhere. Well, obviously somewhere this
technology is not working properly on the machine. Now you know where to go ahead and
look for it. Now you know, well, wait a second, if that's not working on this one machine,
what else isn't working on it that's part of the EDR? Is the EDR working properly? We
need to dig into this to make sure that it's working right.
CTEK astronomy is prevalent in the world that we live in. Anybody knows what CTEK astronomy
is? Anybody ever see the movie Sneakers, and you don't know what CTEK astronomy is? Too
many secrets. We're trying to protect secrets, but too many secrets are being kept from us
by the vendors. I always liken our world to a set of layers anyways. We have the security
industry, and we have the security field, and they cross over all the time. The security
industry are the vendors, whether it's an MSP or a tool provider. The field is each
one of us. You can work in whatever vertical you want to and be part of the security field.
But that vertical isn't necessarily part of the security industry. We need to get everybody
talking better together. Why? Because it gives us more depth. It gives us more knowledge.
It gives us more information. I'm going to finish off by saying, it's not goodbye, but
I'll see you later. If you have any questions, I'm free to take them right now. Anybody?
All right. Well, that about does it. Again, you can find me on Twitter at SiliconShackie.
You can check out my website at SiliconShackie.com. I'll be here all day if you feel you want
to come up and talk to me. I'm always friendly. Well, try to be friendly at least. Thank you
very much, and have a great day.
Thank you.