[Music]
patch we can't patch what do you want to
do turn off the power grid patch we
can't patch it takes three days to file
the safety
paperwork patch we can't patch the only
way to get to the device is by a
helicopter good afternoon my name is
luano Moretti and I'm a senior research
engineer with Verve industrial
protection based out of Madison
Wisconsin I have 20 years of experience
communicating with embedded devices in
medical commercial and industrial space
in my current role I reverse engineer
custom industrial protocols for
interoperability and vulnerability
analysis Verve
helps critical infrastructure companies
secure their devices by identifying them
cross referencing that with known
vulnerabilities and hardening their
systems today we're going to cover what
is OT and
IC how it differs from it
systems the unique challenges of
securing OT and
IC attack
surfaces existing
threats myths regarding mitigation and
what we can do to move towards a better
system
operational technology and Industrial
control systems are the computerized
programmable devices that control
interactions with the physical
environment they're used in control and
monitor processes examples include
controlling manufacturing lines oil and
gas transport and refining power
generation plants Fire Control Systems
and environmental
controls operational technology systems
are commonly made up of many components
devices that are networked
together sorry one second Network
together uh Edge devices are often
custom embedded platforms running on
microcontrollers or application
processors there's diverse as we pick
microcontrollers running bare metal arm
cortex M chips with embedded Opera
realtime operating systems to Windows
embedded or Linux based devices and the
devices are usually paired with one or
more off-the-shelf computer system with
custom software for setup and
monitoring in factories we have plc's
robots valves mixing Chambers and
burners power generation and
distribution has substation controls and
automated
switching Lock and
Dam has power generation Spillway and
slle gate controls I hit the wrong
yes I made a sign oil and gas pipelines
have pressure and temperature monitoring
gas metering and automated
valves and production facilities have
process controls automated Valves and
fire suppression
systems in it
systems sorry okay is there anyone here
who's an IT
administrator Qui couple quick questions
for you what's the oldest version of
Windows you
support Windows
10
okay
okay in an IT
environment
wow um how often do you guys usually
replace an average user's
computer three to four years and are any
be running Linux based
systems what's your oldest kernel
version H not going to mention will not
say
okay well normally they it's fairly Mo
modern but um so most of equipment in an
IT environment are in centralized
locations on fast on Fast connections
equipment usually servers laptops
desktops switches and wireless access
points usually PCS running Windows 10 or
11 Hardware on a as he said 3 to five
year replacement schedule and if
something breaks often a spare is
available overnight via CDW or
Amazon in a contrast in an OT
environment we have all the devices of
it the servers switches desktops plus
tons of specialized Hardware equipment
is scattered across manufacturing floors
and with some
enclosures in isolated locations on
wireless connections including cellular
microwave radio links and
Lowa devices are in place often for 10
plus years we still commonly see XP and
C in the field and Linux devices with 24
or 26
kernels we just recently had to install
deian 6 to support a device we were
installing in our lab
devices usually run a customized OS
without a package management system such
as embedded Linux running build using
build r or yako or microt processor
based systems running rses like free R
TOS mqx QX or VX
works and finally device spares could
cost hundreds of thousands of dollars no
longer be
manufactured or maybe manufactured to
order we had one device recently that we
were looking to to get that and the cost
would have been over a million dollar in
outlay and required a $750 an hour
support
contract to summarize equipment is
scattered across the organization and is
running on older operating systems and
software so why is OT running so much
old
software security doesn't do you any
good if you can't produce your product
so OT systems must remain operational
except for very small scheduled
maintenance Windows why are we this
way factories are often
pipelined not in a batched system so
shutting down means scrapping
product shutting down may lead to
service outages especially in Industries
like P like power industry or may
require redirecting oil and gas to other
facilities and in an extreme cases it
can cause physical damage to the systems
the example I like to hear usually hear
is glass factories where if the glass
cools in the pipes you have to you have
to remove the pipes and replace them
before reflowing the hot
glass I'd like everybody to raise their
hand right
now how many of you H low your hand if
you've rebooted in the last your
computer in the last
day last
week last
month that's we got one guy in the back
or two guys in the very very
back what what's your up
time
sorry three months on your personal what
about
yours about three
months when you're an OT you have to
ask what do you need to do what needs
you to risks do you have in order to
patch
how many of you have had an OS update or
patch crash your
system how many of you have had an OS
update force you to reboot your system
in the middle of
something have there anybody here
flashed a hardware
device if you're lucky it takes
minutes and if you're unlucky it fails
during the middle of the update and you
end up what people generally call a
brick a device that doesn't work and
can't be fixed it's not a huge problem
if it's a $300 router that you can you
can order online or pill one from spares
and replace quickly but when it's a $3
million distributed control system it's
very
different at what point do you shut down
the factory to patch if there's a
security
issue how do you verify that that patch
does what it says it does and doesn't
break something
unrelated it is very with the equipment
as expensive as there is in the OT world
you can't afford customers can't afford
to keep a second spare test Network up
and
running so if you rushed your patches
you can break
production there was a case where a
Windows update to a procedure call
remote procedure call system broke bar
code scanning on production and took
production down for we for several hours
blocking materials moving through produ
through the production pipeline finally
you also you have remote sites which are
boxes in next to a substation a pipeline
or similar we have some devices that we
support where a ping of 1 second is
considered
good how do you push an remote upgrade
to a device over that kind of
connection in the end the common
industry procedure is if it's working
don't touch
it for attack surfaces the cost of
encryption in OT is real devices are
often using older it protocols that
don't require a lot of processing
power many of our systems are either
soft or hard real-time systems where the
communication load is important it's
getting better now that microprocessors
and application processors have built in
cryptography units but it's still rare
and it takes a long time to switch out
devices and
protocols most OT protocols were
designed prior to the industrial
Internet of Things where the expect
expectation is that they were
on on a private or an air gapped
Network one of the scariest things I've
seen is a sales brochure for an IO
controller advocating controlling a
harbor crane
over 802 111b Wireless using weap and
unencrypted modbus TCP
connection now devices are getting
better we're seeing more SSH net comp
with using TLS and other protocols with
encryption support and OT Protocols are
adding optional encryption as well such
as opca and sip
security the other T common tax surface
is
credentials OT has historic operated
where we trust the network and the other
devices on
it often there are no credentials at
all no credentials are needed to talk to
the device or even to configure device
if it's on the wire it's
trusted if there are credentials they're
often way too simple four five-digit
pins and eight character limit passwords
with no rate limiting are
common if the aren't in plain text on
the they're often obis using cheap
operations sometimes often straight
exors with a known number or just
swapping nibbles in the
bites finally many devices have
hardcoded support accounts back doors
for the
manufacturers when a system is down and
it's costing thousands of dollars a
minute for people don't want to have to
hard reset the system because nobody
knows the password
custom
protocols in my business for talking
figuring out how to talk to a device and
get information from it is
great but it's bad for
security often they're just a wrapper
over an older industrial protocol that
they used over over a Serial console
they were designed in the day when you
had to physically plug a Serial cable
into a device now they're wrapped in a
UDP or TCP session and sent over over
the
Internet often there's no login no
encryption no signing no rolling codes
which makes them most of them vulnerable
to replay
attacks malformed Communications often
cause
issues device developers test protocols
with well- behaved Partners usually
software to talk to the device is not
fully specified prior to implementation
and only the green paths are tested Ed
and edge cases often contain serious
bugs unsanitized data causes over and
underflows and undefine behaviors I
can't name names but these are a few
bugs that we found in disclosed to
vendors mform packets taking out the
network interfaces on the device until
reset out of order packets causing the
the device to stop responding and a
packet sent with an empty data section
that caused a hard reset of the system
without user data and the user program
on
it private data is often exposed in
network cap uh Network captures on The
Wire people don't see data or don't
think of data that they don't see as a
security risk but it's often there in
the hidden in the
packet we've seen Wi-Fi SSID and network
keys for private networks private OT
networks sent in plain text in
protocols if a attacker get access to
the device this gives them a second
channel into their
facility we've seen passwords that are
authenticated by the host software on
the computer not by the device
themselves the device sends the password
across over the wire this host software
verifies it that it matches the one the
user typed in and then starts
programming in the
device custom software bypasses the
password
completely and often we see user
password sent with weak xor style
[Music]
encryption as a developer at a previous
company I was once asked to send the
device password across the network using
an unauthenticated unencrypted
protocol and we have found also found a
device that dumped the complete user and
password list via the public SN
smnm sorry um
query
there are a number of existing threats
out there probably the most well-known
OT malware is stuck snut it's thought to
have been a CIA Israeli targeted attack
on the Iranian nuclear program that
caused the center fues to spin
improperly and gradually
fail it's believed to actually have been
introduced to the air gaft facility on a
USB stick by an agent recruited by Dutch
intelligence and escaped the facility by
an unknown vect
but likely a
laptop Havoc was designed to map IC
networks and is believed to have been an
Espionage tool targeting energy oil and
gas and
pharmaceutical IND Destroyer shut down
1th of the Kiev power
grid Trion targeted trionic high and try
safety controlers used in the oil and
gas industry and is believed to have
caused the shutdown of a Saudi Arabian
power station and recently discovered is
pipe dream capable of changing the drive
control parameters on Schneider Electric
and omaran
controllers it even has a
module that uses the update utility
found installed by default with asro
motherboards to load malicious Windows
drivers the hackers of today aren't
Matthew brck wore dialing in from his
bedroom all of these are from
well-resourced government sponsored
hacking
groups
there are a lot of myths regarding
mitigating OT
networks nobody's interested in hacking
our stuff we're not that interesting we
don't sell that many
units it's not worth the hackers time to
hack
us one of my previous employers used
embedded Linux computers in their
devices in their system they had one
system in the field that was hacked and
it turns out was running they were
running a we server off of this Linux
computer they didn't care that it was a
critical device all they saw was a Linux
computer on an internet routable IP
address or the take the Bowman Avenue
Dam it's a tiny Dam in Upstate New York
that was targeted by Iranian hackers who
breached a cellular
modem now the belief is that it was
actually a case of mistaken identity and
they were targeting the muchar larger
Arthur arbman Dam in
Oregon if it's there someone will
try we only share data with customers We
Trust manufacturers only often sell
devices and provide documentation only
under a contract if the hackers can't
get the documentation and can't get the
devices to develop against then they
can't hack
it your
customers distribute that data
to their
Engineers they put that documentation on
shares employees take that documentation
with them when they leave your customers
companies or your
company and that information ends up
published on public PDF PA share sites
and people you've trained as your field
Engineers leave and start and start uh
third party training companies and their
documentation ends up online and unless
you have a strict buyback program
when a customer upgrades or goes out of
business their devices get sold and end
up on eBay through Liquidators who often
don't know exactly what they
have determined actors will get the
data but just don't hook it to the
internet hackers can't hack what they
can't
access how many of you have said in your
job man I wish that data was harder to
get it's just not a viable business
model anymore businesses are demanding
more data about how their systems are
performing in centralized dashboards
they want to monitor the production and
movement of raw materials to finish
product and they want people to be able
to remote and to fix things when things
go
wrong project shine scanned the public
IP space for OT devices and found over
half a million
devices and we're also seeing lots of
equipment that wasn't designed for
ethernet now hooked to it without
additional protections over ser seral
cellular and other protocol
gateways even if you can aops are not
100% effective just ask the Iranian
nuclear
program they're often bypassed
temporarily to make work easier and then
forgotten removable media and laptops
move between your unsecured and your
secured side of your
networks and you have to remember
viruses spread before the
internet and finally you have to worry
about Insider
threats disgruntled employees or plants
from government hacking
groups businesses need the data and a
determined actor will still hack
you a little dance music going
on but we've got a
firewall we have the internet over here
they've got a firewall we've got our it
Network we've got a second set of
firewalls between our it Network and our
OT
Network only people with OT people are
allowed to cross that line this is
probably the most used method in
Industry packets still have to pass
through those firewalls to allow for
data
flows bad firewall rules laptops USB
drives poorly configured remote access
audits often find cables plugged in
where they shouldn't be bypassing your
by passing
protections and finally they also find
many inactive accounts with permissions
to bridge the it and the OT
networks your segmentation is only as
good as your overall security you need
to have tighter segmentation controls
and audit and
audits so what should we do what can be
done comprehensive security takes
cooperation device
manufacturers prioritize security
security partner with researchers to
help them get access to the devices and
information they need to help your
customers developers sanitize your
inputs
please encrypt but don't try and roll
your own there's plenty of well- tested
embedded encrypted libraries out there
good encryption is hard use a well
tested
libraries
oops sorry
okay Buzz test your custom protocols to
prevent under and
overruns and work towards making faster
or in in place upgrades or to or faster
ones to minimize your
downtime integrators know what you have
where and what's it's
running move towards a zero trust
network if Poss possible otherwise
segment closer to devices so a breach of
one otd network doesn't disrupt all of
your
networks encrypt by default turn on the
op the optional encryption on your OT
protocols if you're using a modern
device and disable unused unencrypted
protocols use gateways to wrap your
insecure protocols in Secure
ones and test your
defenses have some try and see if you
can have get from the it Network into
your OT
Network and finally security
researchers file
cves find an OT
device and try and find bugs in it
because there are less people looking at
OT devices than consumer devices but the
risks are higher and the lifespan of a
vulnerability is
longer thank you thank
you I have one minute for questions
yes it's a challenge I mean you have to
as I said
you need to make sure that your the
devices themselves aren't as secure as
they should be um newer devices have a
lot of security optional security
features that need to be
enabled um but you need to segment your
network you need to segment it close to
the device you need to watch that
segmentation
closely and you need to partner with
someone to verify that your that
segmentation is
effective all right thank you everyone
[Music]