Good afternoon, everyone.
My name is Michael Kafka.
Everybody calls me Shecky.
If you want to find out why I got the name Shecky, see me after the talk.
It's a long story.
Wait, there can be only one.
Interesting title, huh?
Expect a lot of Highlander references?
Maybe a couple.
What we've got is... let me give you a little bit about myself here.
Hold on.
You test beforehand, and it doesn't go, and now it does.
Come on.
We'll start off a little bit about me.
I've been in the field of professional for seven years.
I've been in IT for over 25 years.
That doesn't include a period of time where I was just learning.
I started programming computers back in the early 80s.
So I've been around computers for a very long time.
I'm one of the organizers of one of the Birdsects.
If those of you who are down in the Chicago area, you might have heard of us.
We're a weekly, basically, meetup group.
We have chapters, meetups, all throughout the Chicagoland area every Thursday.
Each area gets one Thursday per month.
I happen to be one of the organizers of the North version of it.
We average about 15, 10, 15 people.
So if you are in the area on the second Thursday of the month, feel free to come out to mine.
Or any Thursday, feel free to look up Birdsect and meet up, and you can find out where we're meeting that week.
I volunteer for Hack4Kids.
I have for a number of years.
I'll be volunteering tomorrow morning for the Hack4Kids.
I enjoy seeing the children going ahead and getting introduced to all the concepts, learning, destroying stuff in the destruction village.
There's my website, if you're interested.
I blog, but I blog infrequently.
Why infrequently?
Well, because like the rest of you, I've got other things on my mind half the time.
And finally, you can find me on Twitter.
I do a lot of tweeting.
I try to keep it more professional, I would say.
I try not to tweet as much about, say, politics and stuff like that, as some people out there are trying to keep it more infosec or personal-orientated, comparatively speaking.
So why can't there be only one?
What is the whole deal?
Well, the one thing is related to defense in depth.
What is defense in depth?
The idea of using multiple layers to make things more secure.
It's a very simple definition, and we'll go a little bit more into it as we move forward with the presentation today.
And how are we examining it?
We will be examining it through an actual real-world situation that I was involved in last year.
The data that I found, the reasoning behind defense in depth and how defense in depth actually relates on downward, beyond what your general thoughts are on it, is actually rather interesting because it goes through a lot more than you might realize.
We will be talking about a There will be a couple screenshots.
All the data has been sanitized, so nothing really to worry about.
You're looking at defense in depth.
This is your typical look at it.
You've got your data layer, you've got your application, your host, your internal network, your perimeter, your physical, and your policies, procedures, and awareness.
It's your basic idea behind it.
It's the castle theory.
You get your moat, you get your castle walls, you have your people right behind that, you have your people in front of it.
You get through one layer, the next layer is hopefully going to stop whatever is going on.
But defense in depth goes beyond just these basic layers.
And what this diagram doesn't do, it doesn't go into the depth of the depth.
In any of these layers here, you can actually re-layer things.
Maybe we might not have an area of host or internal network.
You've seen flat networks before.
Perhaps our policies and procedures are lacking a bit.
Maybe it's because of budgeting, maybe it's because of lack of resources physically and people.
But everything goes through there, and it just layers up.
And in each one of these layers, you can have layers, like onions.
Onions have layers.
Yeah, I know people prefer parfaits, but onions do have layers.
I know not everyone likes ogres or onions, and you might prefer cakes or parfaits, but we understand the idea of the layers, but it's not as easy, and it still isn't, because you can have sub-layers, or layers within layers.
The reality is, with sub-layering, it's usually done at a vendor slash product level, as opposed to a different type of technology, per se.
How many different technologies, on average, do you think your company uses?
Three, four vendors?
Five?
More?
How many of those different technologies, or layers, wind up going ahead and crossing over with other ones?
Do you have an EDR that does web-blocking?
Do you have a firewall that's doing IPS on top of it all?
Where you have another thing going ahead and doing IPS, and then you've got a separate web-blocking situation.
So you get this whole strata of different areas.
We used to hate the idea of using multiple vendors in these layers, as it is.
We all remember back in the day with Norton versus McAfee, versus Avast or any of the free AVs out there.
How many AVs can you have on a machine at once?
How many times did those interfere?
Did anybody run into that, where they interfered with each other?
Plenty of times.
Nowadays, it's not as prevalent.
You have multiple vendors sitting out there, and they're not really interfering with each other.
Or if they are, they're interfering in a way that we don't just quite get, and it drives us crazy.
Yeah, it still happens though sometimes, where they interfere with each other.
But we'll talk a little bit about that later on.
So the scenario that I've got for you is very simple.
It's a website.
How many people have web-blocking technology out there?
Use Cisco Umbrella or whatever, or Bluecode or Zscaler to go ahead and help prevent people getting from sites to sites that they aren't supposed to be getting to, including malicious sites.
How many people out there have an EDR?
Feel free to raise your hand.
Alright, so one person has an EDR and nobody's using web-blocking?
Well, if you know anything about it when you're dealing with a website, it came from the net, and eventually it was blocked.
And then you go ahead and say, alright, so we blocked this site, we blocked that site.
Don't think much about it.
You don't think about what technology is actually blocking it.
Is it the web-blocker?
Is it EDR?
Is it something like you blocked origin inside of the browser that you might have distributed out through your company?
And why does that matter?
I ran into the situation with mail.ru and its subdomains.
They were being blocked and alerted on by Microsoft security tools in some machines, and Cisco's umbrella in other machines.
Does that make sense to anybody?
Hands?
Or should I have just been banging my head against the wall like a crazy man?
Would you have banged your head against the wall like somebody that was just ready to give up?
I see a lot of head shaking out there.
So, I ran into a, wait, what?
And I didn't think too much about it.
Both technologies would block.
There was a reason why one was blocking over the other in different instances.
But why was it happening?
We'll start off with the mail.ru.
Has anybody ever heard of that site before?
Alright, we've got one person.
You know, it's basically like a generic casual Russian type site.
There's legitimate stuff on it, there's illegitimate stuff on it.
A lot of illegitimate stuff on it that's hosted on it.
There's also ads that are hosted on it.
It's not really something that most people are going to wind up wanting to go to from our country or from most corporations, as it is, because the stuff on it is just what it is.
It has some legit things on it, but during my research on this as to what it was and why it was being blocked in the first place, alienvault.otx.alienvault.com went ahead and listed out about 20 different malicious items that had been found on that site over the course of time.
And that was just a quick look that I did on it, just to get an idea of what I was dealing with in here.
As it turns out, the initial block and the initial reason somebody went there was somebody that was an employee that was originally from Russia and had their own personal mail account on mail.ru.
And we looked at them and said, yeah, you can use your personal devices and your personal machines for that, not a corporate device.
Thank you very much.
And, yeah, we asked to it.
But then we were still getting these weird blocks from it.
And they were ads.
And we're going, okay, so why is it being blocked one way and not the other, now that we've put this into an actual block list because we don't want people going to it.
So, we found that there were layers sitting inside of there.
The web blocking had layers.
And what was it that was blocking it?
It wasn't just the differing technology that we were running into.
It was because of how they were layered is why one was blocking over the other.
But that still doesn't make complete sense, does it?
Why would you go ahead and still get, why would, if you've got the technology in place on two different machines, why would one use one technology and one use the other technology?
Both machines make the exact same way.
Well, it's a matter of knowing where in the stack, where is everything layered.
Now, why is that important?
You're on an incident.
You've got to do a response on this.
You're going through all the logs and everything.
You see that something was blocked.
Why would it be important to know when in the layers it was blocked?
Well, it would give you an idea of how fast it was blocked, what's the chance of it spreading to other machines, what's the chance of something else going on beyond it, and what technology was actually doing the hard work on it all, so that way you can start digging into that technology more to get more answers.
Not a bad idea, huh?
How many people out here understand what their layering is?
Understand if they've got multiple technologies that can do something, which one comes first, the chicken or the egg?
It's not something that we think of very often, is it?
So, let's take a look at this.
This right here are two separate screens.
You'll see the one that says Destination Mail RE, that is from the Cisco Umbrella Portal.
You'll see there that it's categorized as News Media, Portal, Search Engine, Webmail, and our Internal Block List.
To the right you'll see Microsoft Defender blocking, alerting on the site being blocked by Network Filter Lookup Service through Firefox.
We put this into the Cisco Block List, into our Umbrella Block List, because of the research that I did and Microsoft was naturally trying to block it.
So we figured, well, everybody should get the pop-ups through the corporate screen saying, we blocked the site and if you think that's incorrect, please report it to us and we'll take a look into it.
That was the only way we could get that to go, but not everybody was getting it when we started testing.
It is real interesting that it says Network Filter Lookup Service, as opposed to a Web Block.
Now, Cisco's Umbrella works off of what used to be called Open DNS.
They use categorizations, they use simplified systems to go ahead and allow you to set up what sites people can go to, what sites you're going to block down to, and it doesn't always give you 100% information.
Now, while we were working through this, they actually made a couple of changes inside of the portal.
The first question that we had was, is mail.ru even making it to the Cisco interface?
We go in there and we see it says DNS next to it, and no, I don't have screenshots of this just because there was proprietary information inside of there that I cannot reveal at this point in time.
But it would say DNS.
So when we got on board with Cisco to start looking at this issue, why wasn't Cisco blocking it?
Why was it being blocked by Microsoft?
They said, well, the simple thing is, is that yeah, it's showing it, but it's just a DNS lookup that's going through and we're not seeing anything else beyond that.
At least on the machines that we weren't getting the block screens on.
We'll take a look into it maybe, but there's got to be something else going on.
Not a lot of great information, except for the fact that even if you've got a website and a URL, it's still going to go through and at least hit the open DNS servers, the umbrella DNS servers, before it does any block.
And that starts breaking down to, okay, so now we know when I type in a website, we'll go ahead and we'll take a look at that site and we'll see what the IP address is.
We'll get that IP address first.
And then when he tries to go ahead and make a connection to that site, we'll go ahead and we'll block it through the web locking technology inside of it all.
Okay, makes sense.
Would you agree with me on that?
Well, the interesting thing is, is that we're still going ahead, not any closer to understanding why Microsoft was doing this.
Why did it need to be in a manual block listing umbrella?
Well, it's a matter of threat intelligence.
Microsoft has their own threat intelligence.
Cisco has their own threat intelligence.
AlienVault has its own threat intelligence.
Palo Alto has its own threat intelligence.
Threat intelligence is its own threat intelligence.
There's no standardization and there's very little communication between a lot of these companies.
So they'll go ahead and do their best analysis on it.
And the analysis boils down to, in a lot of cases, what a human being will say or what a machine learning algorithm says.
Once it passes through one, the other one takes a look at it and makes a final decision.
At least that's how I've seen it work at this point in time.
So we decided to trust Microsoft and put it into the block list.
We knew we weren't going to be stopping anything that we needed to worry about from a corporate standpoint.
We weren't going to be affecting anything from a business standpoint.
It was all going to still be there.
And we weren't going to get any complaints.
And sure enough, we put the block in there and we still don't get any complaints.
And yes, I still get these alerts from Microsoft Defender to this day occasionally if there's an ad being served from mail.ru.
You would think that adding it into the umbrella block list would be the end of it.
We thought so and we were wrong.
That does say wrong there.
So what we started to do is, well I started to do the testing.
The reason why was I was asked by the CISO, why is Umbrella not working the way it's supposed to be working?
Or is it Microsoft Defender that's not working the way it's supposed to be working?
One or the other is not working the way it's supposed to be working.
Why are we getting to?
So it was tasked to me to go through it.
So I started testing mail.ru and enforcing things.
First off, through different browsers.
If it was getting blocked by Umbrella, we would get an umbrella block in all three browsers.
Okay, so we know that it's not a browser -based situation.
If it was getting blocked by that network filter on Microsoft Edge , you would get the smart filter screen.
Has anyone here seen that red smart filter screen ?
I know that Edge is not always very popular.
But it's a nice bright red screen that says, basically, we're blocking this because of security concerns.
But if it was being blocked in Firefox, like you saw in the slide, or Chrome, or Brave, or any of the others, all you get is a white screen with a little bit of black text that says access denied.
Well, that tells us a whole lot.
Both screens tell us a whole lot, don't they?
Access denied can be just about anything.
So, the question is, which one is it going to actually hit first?
What is the order?
And order is important because security is knowledge.
Knowing what should be doing what, when it should be doing it, helps you find out if you've got a misconfiguration or a vulnerability sitting out there that somebody is going to slide through, or potentially can slide through.
If you don't know all of it, you might end up chasing your own tail forever, which, as you can hear, is what I was doing with this whole situation.
So I was doing these tests.
And it made me learn about both technologies in play.
The first thing that I did was I found out that uBlock Origin, inside the browser, if it's an app that uBlock Origin has in their block list, because that's what they use as a centralized block list up on the net, that's going to supersede Umbrella, it's going to supersede Microsoft,
it's going to supersede anything.
It's going to see that URL that you punched in and say, hey, this is on my block list, it is blocked, boom, done.
So there's your first layer right there, uBlock Origin.
But as we all know, that is dependent upon things being reported in uBlock Origin, and however they decide what sites are good and what sites are bad.
And it's mostly ad-based sites that uBlock Origin blocks anyway.
They don't necessarily block malicious or anything like that.
You also know that that could very easily be bypassed.
It's very easy, even in corporate, to go ahead and just turn off uBlock Origin to go ahead and allow a site to work the way that they originally intended it to.
We've all done that.
We all know our users have done that as much as they say, oh no, I wouldn't do anything like that.
We all know that that's not true.
So let's start taking a look at it.
Cisco Umbrella.
Pardon me for a second.
I've got to see how the screen looks.
I have some slides here where things will pop up and some where they don't.
Having the screen behind me is a little bit different than how I'm used to giving a presentation.
So Cisco Umbrella was formerly OpenDNS.
They allow successful DNS lookups before blocking.
And it uses its own intelligence, Cisco Talos, to go ahead and categorize sites.
Basically what Umbrella is, is a machine in the middle attack.
If you really think about it, and it becomes more evident when there's a problem with the Umbrella root certificate, which I've seen happen before where the root certificate didn't get installed onto a machine, and your browsers nowadays with how they're going off about security and certificates,
you all of a sudden get a block in Firefox, you get a block in Chrome, you get a block in Edge with a security warning on it saying, yeah, this is not a valid certificate.
So you know that there's a middle section there that it's going through.
It also means that the DNS lookup happens before any sort of blocking happens.
Now we've got part of what's going on.
So we understand the basics, and that's all that you need to know is the basics of how Umbrella is doing it.
You're going in, you're typing in your website, it's going ahead and shifting everything over to its DNS and using its certificates, and then allowing everything to go through or blocking it.
Simple, concise, almost pretty in a way.
So then you get into the Microsoft network protection.
And as I said, it was Microsoft Defender for Endpoint that was alerting on this.
Network protection, we found out, bases and things on Microsoft's Intel.
Big surprise there.
But it has to be turned on.
And where does it get turned on is the next thing.
It's part of the exploit guard series of ASR rules, or attack surface reduction rules.
So if I tested a machine, if I tested an ALR on a machine that did not have those rules applied, but did have Umbrella applied on it, the Umbrella block screen would show up as expected.
If I had a machine that actually had those rules turned on, well, then we get into a different situation.
But again, this is all through Defender that we're looking at.
Now those of you that don't understand, Defender changes itself every three months right now.
And it gets better.
But you still have to go through a lot of these queries to figure out what's really going on behind the scenes.
So we know it's part of exploit guard.
We know it's turned off by default.
It's enabled through group policy or Intune.
Now why do I bring both of those up?
Well, we'll get to that in a moment.
But wait, there's more.
It uses Microsoft's threat and tell, and it will give the smart screen message if you're using Microsoft Edge.
And finally, it will just give you that access denied if you're using a non-Microsoft browser.
Finally, it does alert in Defender.
So we've got all this information.
So we now know that, well, if we have exploit guard turned on on a machine, that seems to go first.
If we don't have exploit guard turned on in the machine, it hits umbrella.
So we're getting somewhere.
So we now know the order of operations.
So the question is, why is exploit guard not turned on in everything?
So Microsoft's protection works similarly in the sense that it runs off a pure IPS DNS response, and it's not... network filtering is not a web-locking technology.
I don't decide categories on that.
It is purely done off of threat intel.
If it blocks, it says either that the site's IP is good or bad, but there's always some network traffic involved with it.
But there's less network traffic, and it's sitting closer to the kernel.
Cisco was the third layer.
So if the other two aren't working, then umbrella will wind up going and blocking it, or if it gets through these.
Now we're getting to a point of going, now I understand why the layering is important.
And we're not having technology interfering with each other.
Instead, it's complementing each other.
Isn't that what defense in depth is supposed to be?
Isn't that what we're looking for?
Is if something fails, we've got something else to back it up?
And now we're doing it with two separate, completely separate ecosystems.
A Cisco ecosystem and a Microsoft ecosystem.
And they're actually working together, even though they don't directly communicate with each other.
Defense in depth is layers.
So the ultimate reasoning behind it all, and the final portion of the layers is, so why were it was only some machines?
Well, when you're dealing with a hybrid situation where you have an on-premise domain and an Azure domain, unless you're joined to both of them, everything does not get pushed out through one technology.
Azure joined machines do not get GPOs pushed to them.
Everything goes through and to.
This all started up during a period of time where we were starting to implement the ASR rules, including exploit guard and network filtering.
We wound up in a situation where machines that were part of the rollout already, which was mostly on-prem machines joined to the on-prem domain, were getting those ASR rules.
So they were blocking it through Microsoft network filtering.
The machines, the laptops that were being taken home that were part of the Azure joined domain, they weren't getting that protection yet.
Which gave us the time to go ahead and say to our other teams that were in charge of pushing out these protections, why aren't you pushing these out yet?
You've pushed them out to the most sensitive of the machines and the machines that would break the easiest.
Why have they not gone out to everybody else yet?
To which we got the typical, we haven't had time to go ahead and finish testing them and we don't have enough resources to do it in a super fast fashion.
We'll get to it within the next month.
And they slowly have gotten to it and things have gotten pushed out.
In the meantime, we turned around and said, well, thank God that we've got these multiple layers sitting there.
Shows flat out that defense in depth does work.
And it works very well if set up and configured properly.
But again, whether it's the overview of different technologies from physical to policies, or it's inside of an endpoint solution or solutions, if you don't know where things are going to happen, you're just going to drive yourself crazy going, well, or you're just going to say,
well, it's blocking, who cares?
But somewhere down the line going that, who cares, can turn around and bite you in the butt when you have an actual situation.
As to where Ublock working comes into play, it blocks ads nicely.
I said it happens before Microsoft's technology.
And it looks at the URL name and compares it to its own list.
There are general layers.
There are sub-layers.
Layers can be different technologies.
So layers upon layers upon different technologies.
And sometimes inside of the same technology stack, you can have different layers sitting there.
Microsoft Endpoint nowadays actually has its own web-blocking feature inside of it.
I have not tested it thoroughly because it is still in its infancy and the way that it works doesn't lend itself to the corporate situation that I am in at this point in time.
But I check it out every so often just to see if they've gotten it to a point where it's worth me completely testing on it.
Cisco has multiple technologies.
They're always acquiring stuff.
You can have layers of Cisco technologies and in between layers of somebody else's technology in there.
You could be using Duo with a Palo Alto firewall.
The final idea though is to have the layers complement each other.
So when you're actually doing a POC, when you're actually testing, you have to make sure that any machine that's being tested on has your current technology on there minus whatever technology is being replaced, if it's being replaced.
So if you're trying to remove from something say like Defender from Endpoint to Carbon Black, you remove Defender from Endpoint from the machine but all the rest of your technologies have to still be on that machine.
Two reasons why.
Number one, you'll see where things hit.
Number two, it'll make sure that you're not interfering like in the old days because that does happen at times.
It's an instance where you need to go ahead and figure it out and that's part of our job.
Everything is not all glam and not all writing up whatever you're writing up scripts, thread hunts, scene queries, however you look at it.
Each one is just a greater part of the whole but you need to know how they fit into the whole like a jigsaw puzzle.
There can be only one and that one is order.
It is the order of operations, it is the order that things hit, it is the order that you implement them sometimes.
But that's the only one consistent thing on it all, is that order.
If you're going to do Defense in Depth properly, you need to know what that order is.
Otherwise, it's basically spitballing it or throwing a piece of gum at a wall.
Maybe it'll hit this, maybe it'll hit that, I don't know, maybe it hits here.
And then when you have to go ahead and support that technology, you go, well, I don't know, and then you start wasting time trying to figure it out at that point.
The vendors themselves are not always so helpful in being able to diagnose this sort of problem.
And that's where we can really get better, is if we can get them to all work together with us.
We see this on a regular basis.
How many people here deal with scene technology?
Logstash, Slipknot, what have you.
How hard is it to go ahead and get things parsed properly into that scene?
Anybody?
Anybody have an easy time with it?
How long?
It's a mess?
Syslog.
How many items go in there that are Syslog?
In different formatted Syslogs.
In different shapes.
You get the Linux, and then you get the Cisco, and then you get the Microsoft, and then you get this coming in, and you have to write up... I used to deal with Raylong this way.
I had to write up parsers for five different technologies on Syslog, and a way for it to recognize which technology was coming in.
That took me six months.
Six months of testing, and checking, and testing.
In the meantime, you're getting overfilled data on it.
Why?
Because everybody wants them in their one silo.
We need to work together.
And it's not just the corporations, such as Microsoft, or Palo Alto, or Cisco, or Carbon Black, that need to work together and come up with a standardized way to get logs in, and standardized fielding, so that way it parses properly.
Because all that's part of defense in depth in its own right.
That gives you the views that you need to see.
But if you're not getting the fields that you need, or not able to easily see them, you're making your work two to three times as hard.
But we also need to be able to share out these sorts of information.
If you make a parser for Cisco Firepower, that parser is going to be different than Cisco ASA, by the way.
I went through that.
Inside of Cisco, ASA's syslog is different.
The fields are different than Firepower's.
Probably because Cisco bought Firepower and developed ASA.
And the two teams don't necessarily communicate the best together.
If we can share that information, if we can share something like this, where now all of you know that if you're using a Microsoft stack and a Cisco Umbrella stack, Microsoft's network filter is going to supersede anything that Umbrella does.
That way, when somebody comes back and says, why isn't Umbrella blocking this?
Why are we getting an alert from our EDR on this?
Well, that's because Microsoft Threat Intelligence feels that this is blocked.
Or we put it into a block that's inside of our EDR, and our network filter hits first.
It goes ahead and says, oh, this is this?
Fine.
It hits before Cisco finally does.
Well, we've got the same technologies everywhere.
Well, obviously somewhere this technology is not working properly on the machine.
Now you know where to go ahead and look for it.
Now you know, well, wait a second, if that's not working on this one machine, what else isn't working on it that's part of the EDR?
Is the EDR working properly?
We need to dig into this to make sure that it's working right.
CTEK astronomy is prevalent in the world that we live in.
Anybody knows what CTEK astronomy is?
Anybody ever see the movie Sneakers, and you don't know what CTEK astronomy is?
Too many secrets.
We're trying to protect secrets, but too many secrets are being kept from us by the vendors.
I always liken our world to a set of layers anyways.
We have the security industry, and we have the security field, and they cross over all the time.
The security industry are the vendors, whether it's an MSP or a tool provider.
The field is each one of us.
You can work in whatever vertical you want to and be part of the security field.
But that vertical isn't necessarily part of the security industry.
We need to get everybody talking better together.
Why?
Because it gives us more depth.
It gives us more knowledge.
It gives us more information.
I'm going to finish off by saying, it's not goodbye, but I'll see you later.
If you have any questions, I'm free to take them right now.
Anybody?
All right.
Well, that about does it.
Again, you can find me on Twitter at SiliconShackie.
You can check out my website at SiliconShackie.com.
I'll be here all day if you feel you want to come up and talk to me.
I'm always friendly.
Well, try to be friendly at least.
Thank you very much, and have a great day.
Thank you.