[Music]
all right I'm going to kick off here uh
hey I'm Ryan leov I'm talking about
hunting before Day Zero uh a little bit
about me I'm a founder and CEO of
startup security and I enjoy eny doing
Disaster Recovery storms the hard
drives um startup security um which is
relevant to the topic today is we help
secure the tools Technologies and
companies that are building the future
we want uh in practice this means
helping series a b andc startups
Kickstart their Security Programs
meaning they have no security hires but
have built something worth securing and
so we get to tag in and help um in
practice this looks like a lot of
security observability research um from
our end and uh my background which is
also relevant I was originally a
software engineer worked for a bunch of
SAS companies um eventually I got
tricked into being an SRE which is
keeping other people software running
doing a weird mix of Dev Ops and sis
admin work and comfortably now I've
settled into doing U security Works
which is why I'm talking here um and uh
a little bit about how I like to operate
which will also be reflected in the talk
I like to go first principles up not big
tools down uh really into open source
tools instead of using
black box tools I don't understand and
uh I am not afraid to shamelessly
iterate and be dumb and figure things
out as I go so um some of that will come
across also the slides code and I built
a range to play with this on because uh
even though a lot of the tools we use
adverti being pretty easy to to use are
in practice kind of hard to set up for
the first time so I Tred to build some
VMS with them running so if people
wanted to play with it they could uh
they could do that also there will
definitely be spelling errors um and I
apologize in advance so today's agenda
uh we're going to start with a story
about um my first run in with signals of
compromise and um which informs My
Philosophy for dealing with them then
we're going to talk about the
philosophy of the signals of compromise
all the way to the tactics of how how we
enjoy handling them and then uh some
conclusions so what will you get out of
this um if you're a beginner to
intermediate person or practitioner I'm
hoping to give you a better
understanding of the signals available
from Linux and how they can lead to
improve situational awareness of the
systems you're working with um and if
you're more advanced I'm going to try to
convince you that having the ability to
ask your system very specific questions
without relying on external tools as a
superpower and also tell you about my uh
favorite tools which are all supported
in the main mainstream Linux
kernel um also this is a big topic um as
I was traveling and preparing this talk
um because I do so to the last minute on
all of my stuff uh here's my stack of
books so we're not going to go over
everything in Vivid detail my My Hope Is
that I give you enough details to pique
your interest and then you know kind of
where to go for next steps and I'm happy
to talk about this stuff all day so if
it's interesting please reach out to me
and thanks to the
organizers okay so let's get into the
story of how I ran into signals of
compromise or how to accidentally
discover a zero day so and for the
definition here zero days of
vulnerability that has uh not been
publicly disclosed the important details
here if it if you run into one on your
system that means there's no patch and
no easy way to track um and so to redact
all details of uh clients uh I used AI
to generate this Hyperion Corp a lovely
startup SAS that we will be we will be
hypothetically helping today and so on
this client our goal was to stand up a
firew rall around a microservice
um this was a critical service um this
whole company had been growing fast and
it had one Security Group and no no real
consideration for security and so
finally they decided they needed help uh
securing it so going into this our our
goal was to understand what the service
was doing understand the traffic
patterns of the service and then observe
the the traffic and so a little bit of
detail um we work with a lot of startups
in hypergrowth and this is a phase where
they have enough users that people have
given them truckloads of cash and so
they have users and cash they've been
told to build very fast and um as you
might imagine growth is prior
prioritized over everything uh
specifically or definitely including uh
security and so for whatever reason uh a
lot of companies that do this um choose
to go with microservices which is uh
important for us as security
practitioners to understand I'm not here
to argue for or against them but just on
some quick why um some leadership napkin
math here is if I have a bunch of big
problems and I can divide them into
problems small enough to where I can
hire one team to solve one problem then
I just take the number of problems I
have hire that many teams and we can
build super fast um if you've ever
worked at software you know this is
likely not the case in practice but
anyways um we're using microservices
some keys about them they tend to be one
service that does one thing
or handful of things and um it's
important to understand their traffic so
there's North South traffic meaning some
of our services talk usually through a
load balancer to our users and other
services we have talk more just in
between Services um for example
typically our email service doesn't need
to talk directly to our users but our
other services like our uh whatever is
resetting the password is probably going
to cue the the email service to send
email uh and it's very common for large
compan to have hundreds of these so back
to our story here's just like a napkin
example infrastructure we have a front
end orders API and admin tool and then
those are the more of the customer
facing or client facing Services North
South and then back our back of the
house Services we've got emails uh user
and authentication Service which is the
one we're going to focus on and then
metrics um and so you know we can kind
of generally understand the traffic
patterns here zooming into to this
service we're going to secure together
um we expect a lot of east west traffic
meaning other services are likely
needing to ask like hey can user X do
action Y um we probably have some amount
of Maintenance traffic you know an SRE
or assis admin might log in to to check
something out or troubleshoot an issue
um there's probably deployment traffic
this is something we'd expect if you're
deploying a new image um and then
regular system updates if we don't have
any type of you know internal way to
deliver those they're probably reaching
out to some some p package service to
get
that um and so yeah again to zoom in on
the what we would expect this service to
do is a a user it's it's um so we we're
placing orders to this API if I'm user
one can I delete order 7 this service uh
needs to say yes or no behind the scenes
uh there's a there's a goang service
that was pretty thin and uh we had the
users database powering as a source of
Truth and then a simple reddis cache as
a way to kind of to store um the
permissions um and you would expect you
know some type of network call to this
to to do this um nothing nothing fancy
going on here so as a security expert
thinking through okay what could go
wrong if this is compromised um we could
poison the redis cache so I could make
it so users could do things that they
shouldn't do we could xfill data um if
we can get to redis we can get a lot of
um information about user IDs and roles
um or if we can make it further than
that and get to the database we can get
user data which is uh pretty great uh
it's an attacker not for us as a
Defender um or we can move laterally to
other services
so um got it we got we understand the
service we understand the traffic our
next step is to observe the traffic and
um this is really where we ran into our
first uh like signal of compromise which
was just a signal at this point and so
um in order to do this um we focused on
TCP we didn't expect much UDP or icmp
traffic so care a lot less about that
and we had this idea for how what is the
smallest way or what what is how can we
derive the right configuration for the
service without raising a firewall
config that accidentally kills things
that we don't need to and so our goal
from a high level was to say okay uh we
have the three-way TCP successful
handshake um as and what that looks like
is a source uh sending a a sin or
synchronized packet to destination the
destination sending back a s act packet
or synchronized acknowledgement and then
act and so if we simply got the the
count of the sin and Axe and knew the
hosts and kind of grabbed the counts we
could uh develop a general idea of the
traffic that this service was using and
and and uh deploy this configuration
drive from data that we gathered so what
are tools that people would usually use
to do that you know um wire shark's
pretty common for trying to observe
Network traffic but this wasn't really
appropriate because it was in a a cloud
environment we could use TCP dump um we
could use a TCP dump originally was
built with lid pcap which is a way that
you can um capture and analyze uh low-l
packets um and then eventually we found
what I I at the time I'd never messed
with BPF or ebpf which is uh Berkeley
packet filter um and it's actually a
virtual machine that runs runs in the
Linux kernel that allows you to to write
programs that you can use to um do a lot
of things uh from observability security
um and networking and so mindly in this
case we can look at um low-l system
calls and see see um TCP traffic and
so um and um has has a lot of
interesting guarantees to so even though
it's running in the kernel it's not it's
not dangerous for our users or our
systems Okay so so we ended up doing
this um when we got this
configuration and um we noticed
something weird after monitoring send
and a for a week um there was some
unexpected traffic and so there was an
AR the arbc surface was talking to the
email service a lot and um there's no
reason and it was talking a lot directly
to users and so originally these were
expected traffic patterns somewhat but
it ended up uh leading us to discover
that there is something deeply wrong on
this system which we were able to
address um and so this is moving this is
why why I got why we got really
interested in signals of compromise and
I think have a pretty novel take on them
um and
so uh the the quiet Revelation here is
we were just tracking increments of a
system call and when I historically had
thought about signals of compromise I
was used to a lot of details um but I I
I realized that a small a small counter
revealed enough for us to determine
something was wrong and so the the
typical axis if you will for s for
looking at signals of compromises
there's signature based detections
meaning hey this file name matches a
known bad file this IP address matches a
known bad IP address kind of evolves
into threat intelligence databases um be
Behavior Analysis looking at you know
this system's acting weird and then um
you know database learning so ml AI
throwing data science if you will at
your problems um but the thing that was
missing for me uh with my background
from being an SRE uh there is this
distinction it was Nuance between
monitoring and observability and so
monitoring looks like taking off my logs
from my web server and looking through
them it's a lot of data and you it is uh
intensive to gather it it's intensive to
look through it and drive any
conclusions from it um then
observability we could track a lot less
data but and understand if our systems
were underwater if our users were seeing
things that we shouldn't um if we needed
to provision more more things and and
it's uh you know going back to tracking
simple counters and very simple
measurements led us do uh good service
for our users instead of trying to dig
through a bunch of logs and so that's
where I really started to categorize
this other axis that I found really
interesting in signals of compromise
which is
um trying to take the normal tools we
have then also introduce this idea of
like what is the smallest signal I can
use or most fundamental signal I can use
or the lowest level signal I can use to
drive the to to understand the
conclusion or understand the situation
and so and just um to to to say what I
mean by big and then we'll look at the
small signals a big an example of a big
signal is if I'm going to detect uh like
a C2 like Cobalt strike um there's some
very standard this is just like Red
Canary docks on how to do this um you
can look at uh patterns and named pipes
uh you can look at uh patterns of
process executions and um patterns of in
your command line looking at certain
invocations but the problem I have with
this is it's just a cat and mouse game
so it's like if I have like ah I'll just
change how I name my pipe or I'll change
slightly change this thing and so you're
always chasing a very specific thing
that takes a lot of data to
gather um on the other hand and what if
we took the smallest signal possible and
you know kind decomposed what a what a
C2 is doing um and so trying to find a
low-level signal a granular observable
signal that we can use to detect
something like data
exfiltration um you know they're
encrypting data moving laterally through
our systems privilege escalation they do
a lot a lot of things so let's zoom in
on Data exfiltration this is a concern
we have that's really part of our our
threat threat
model um and note there's very sneaky
ways to move files which is why I think
this is interesting to look at at a very
low level um and get a high signal read
on what's Happening um this is moving
moving files through DNS tunneling which
I thought was uh amusing um what tools
do we have what low-l tools and we have
on one hand we have the file system the
other hand we have the network these are
the two pillars that we can look at um
and how do we interact with those what
is the lowest level way we can do that
is system calls um um I had to go back
to
like junior year of computer science and
kind of remind myself what a system call
was as I was originally getting into
this and really remembering how a
computer how us and userland communicate
with our metal through the kernel using
these functions well these functions are
events that are really interesting and
can can drive a lot of data for us if if
we're observing a system and so uh the
tools we use here um to Snoop the system
calls I talked about ebpf and then I'll
show you an example of collecting them
with Prometheus um what you can use
these are just the open source tools we
use there's lots of uh cognates here um
grafana to visualize things and
Prometheus again to query so we can
actually compare signals across our
systems um and so
again uh just trying to demystify how to
use ebpf and show you some actual
examples here because there's books and
books on this um like I said it's a VM
and like anything that has uh this very
low level there's abstractions built on
top of them the abstractions the two
that I use most are BCC which is BPF
compiler collection which is a toolkit
that lets us um build these use normal
uh languages we're used to dealing with
and build uh collections of this bite
code or BPF tool which um yeah I just
want to let's get into actually doing
them and so for uh Prometheus for
observability if you're not used to that
it's um Prometheus employs a highly
efficient custombuilt time series
database that's optimized for storing
retrieving large amounts this is like
the basic observability tool that um you
would start off with if your Sr so a lot
of our companies already use this um and
then integrating it into grafana just
builds graphs out of our data we're
scraping and U just to show you kind of
the query language here um we can we can
do things like that um and so let's
let's dig to file access um so how could
we detect if uh from a small signal that
we're X filling files so one of
the uh amazing but also frustrating
things is everything is a file on Linux
and so um you know your CD drive is a
file your every process is a file so we
have to be very specific about how we
we're looking at this um and so if
someone wants to move your file uh what
are they going to do how do we know if
someone's reading our files well there's
a whole table of system called that that
are used to interact with files of those
there's maybe four or five that are used
when you're reading a file with a
program that like cat um and so we can
do the same thing we did with syac and
say okay give me a pit and tell me every
time it reads a file and we can we can
find you know if it's our if it's our
web server we would expect maybe one pit
or two pits to be reading files maybe
three pits maybe we have something look
at system Health our web server and uh
some some other tool that we have
install on every host so it gets pretty
easy to see if there's something that's
um reading files that shouldn't be
especially at a large answer and so this
is actually really easy with ebpf
there's something called open Snoop
which if you go into ebpf there's a
whole tool box of scripts that they have
built and so you can run this tool and
very quickly see if something is reading
a lot of files that shouldn't be um and
this is a a post um Brandon Greg writes
a lot about EPF he's one of the original
contributors saw mentioned him a few
times um and we can do we can do this
access for a lot of similar things file
access patterns file modifications file
creation deletion and um even detecting
zero days specifically with ebpf um this
is an example of a Sim leak race attack
and so this attack allowed a user to
potentially gain read and write access
to any path on the host system this was
related to um race condition and Docker
and here is a and I use this in air
quotes one line because it's a very long
line but a one liner um using ebpf um
tracing this very specific system call
that you could use to know on Day Zero
that they came out with to help see if
their systems were um affected um and I
believe this U with Greg he was at
Netflix at the time so they they used
that and so that's files on the other
side of the the other column we have is
network
access um and so networking is not
something we're going to get into how it
works today I mean there's a long long
listen and similarly there's lots of
system calls involved in networking even
though this is a pretty complicated
topic for for understanding uh let's
focus again on the the original problem
of the three-way handshake sin sinak and
act and let's do
sinak um there are
four five system calls for dealing with
this um which I links to the slide you
can read about this in detail if you
want to I'm not going to bore you by
reading every word now and um we want to
so for this we can actually focus on one
call so every sin and sin act uses the
connect system call and so this
initiates the connection to the remote
endpoint by sending a sin packet and
waits for a sin act packet in response
this is exactly what we want and in
order to do that it it's pretty trivial
so this is a clip from again there's
lots of scripts community driven scripts
that can do this and here's um don't
understand the syntax you you don't need
to understand the syntax but it's not
very hard to run this one script that
does exactly this and so the output um
this is a that was a the important bits
from so connect and it spits out exactly
what we want um and so I'll show you an
example of how we um if you're not using
the community based scripts and you end
up building some of this tooling on your
own I'll show you an example of how we
tie this all together into something
that we can instrument monitor an alert
off of um and so it's a I've named this
past ropy which is um tell me every time
that a new uh process is created on my
system so I can see spikes of of
processes being created on a system and
so our goal is to collect the stats on
new prits
created um I want to have complete
control over this data and I want to be
able to visualize and query this data
and so how are we going to do that we're
going to start off this is a ebpf
program the important part here you see
data. pit equals task uh task aop PID
and then there's um this way that uh uh
a BPF map so that I can read this I'm
again I don't write a lot of C but I do
write a lot of go and so this is all the
C I have to write um then I can take
this uh this is the the BPF compiler
collection um package and this is just
in go it's what we write with a lot
there's a python go pretty much any
common language you can use uh uh this
with and so we just
load um sour is the file that uh we
compiled into the bite code and then we
attach a k probe for kernel Probe on um
the system called the um CIS clone is
what what opens a new pit um and we
create a new counter and um we get the
payload so every time this new pit is
created we just in increment that
counter and we can see it running on our
server um and Prometheus is just uh it's
kind of interesting the way it works is
a a text file that has data and then the
prus server scrapes it on an interval so
we can take that after we see it running
um we can see the the counts
incrementing and see them see them in
our our graphing tool this doesn't mean
a lot to us though we had we need some
way to see if this is working and so I
wrote the most complicated script in the
world which just Loops over and does
asleep um which ends up kicking off a
new process so logging into the server
and doing this we can see a massive
spike in um in our Prometheus data well
you can see it in grafana which a lot of
times this is where or uh every time
that grafana and Prometheus is use this
where we cue all of our on call alerts
and so uh even though this is a pretty
uh contrived example for um doing this
this whole process works and scales to a
lot of different types of tools all you
do is customize what type of system call
your tracking and you can throw it into
your alerting uh infrastructure and make
your own tools which can be very
specific to the domain you're working in
um
so a few questions I had uh so system
calls um are a pretty big topic um but
there's tends to be on whatever system
you're working on there's a big
repository of system calls and then also
if there's specific programs that you're
interested in that you're trying to
characterize you can use a tool like
srace and actually look through the
system calls that your normal tools are
generating which can help you lead to
discover what you want to what you want
to Monitor and so the other question I
had is how could we subvert this if we
were an attacker um and the only thing
that I could come up with and I found a
documentation of this is under extreme
loads way BPF works if it gets
overwhelmed as in if I'm a attacker and
I'm trying to hide something that I'm
doing on a system um I could generate a
lot of system calls if you're tracking
them which would overrun uh the buffers
where um all this data is cashed before
we collect it but even on that BPF knows
and can alert that that's happening um
so again all the code
is um there in my GitHub I'm sure
they'll publish the slides and this is
all open source mistakes and all uh this
is something you're interested in please
reach out to me I love working on this
um uh yeah and um so just a few a few
conclusions here um the bigger and more
abstract our tools get the more we lose
touch with our systems I remember when I
started I was copying and pasting files
over FTP on my web server and that was
uh that was it and now I'm running
kubernetes on a hypervisor in multiple
region I mean it's just it's a whole
different world and it feels like uh
after working on on so many layers of
abstraction to me it's really refreshing
to dive back into the nuts and bolts of
how how our tools work and it it even
though there's more overhead today it's
still worth diving in and understanding
the
fundamentals um we're also um I love big
tools I love dropping name brand tools
with big bucks to solve problems but a
lot of times if your problem is specific
you know Big Tool of choice doesn't have
exactly what you need to monitor the
situation and so having the having the
ability to actually ask questions of
your system on a low level comes up
pretty often um and observing small
signals can help us solve big problems
um even though um this was led to zero
day it also helped us configure the
firewall safely so it's uh useful for a
a lot of a lot of tests uh love to ebpf
Prometheus grafana and I think we're
going to continue to radical evolutions
and new new and interesting tools built
with
ebpf um thanks and Happy
[Applause]
Hunting