[Music]
thank you
[Music]
all right
foreign
real is simply electrical signals
interpreted by your brain
what's up I'm Lance Travis and Ben uh we
all formed a company called holasoft
holasoft is really a company where we
leave the information age has evolved uh
welcome to the age of experiences and we
like building the XR experiences so
we'll come keep coming back to that
so we also uh run and found in the
madison.net user group so if you like
the computer if you like to program
computers uh we have the we have an
online group now we'll go back we'll go
back in first at some point in time
uh but let's see can your perception be
hacked so let's see
the fact that reality is whatever your
mind infers from the nerve impulses sent
by your sensors based on its model of
the world is at the heart of what makes
VR different and more powerful than
anything that has come before pick a
pill and keep your eyes on it
got it let's mask off everything else
interesting you were pretty sure that
out there in the real world one pill is
red and one was blue but they're exactly
the same shade of gray take a good look
at the blue tiles on top of the left
Cube
okay now take a good look at the yellow
tiles on top of the right Cube
you think those are colors you're seeing
now
let's mask off everything but those
particular tiles
they're all exactly the same shade of
gray see the white tile under the desk
and the black tiles just outside the
desk
again let's mask off everything else
and again they're both exactly the same
shade of gray
but if one of them is that is in Shadow
and is that shade of gray it must be
white
and if another one is in bright light
and is that shade of gray it must be
black
your visual system does the inference
for you automatically and what you see
is white and black rather than gray here
we have a nice Square checkerboard let's
add some dots and see what happens
this happens because some line detectors
fire that normally wouldn't be triggered
by a checkerboard rather than looking
orthogonal the corners look bowed
okay that's just weird
here's what's actually happening
real objects especially faces tend to be
convex so your visual system assumes
convexity in the absence of cues to the
contrary
the only way to make that assumption
work in this case is for the head to be
moving in very odd ways so that's
exactly what you see
as the Oracle might say here's what's
really going to bake your noodle later
on now that you know what's happening
try to see the dragon as it really is
I bet you can't do it a little bit of
conscious knowledge isn't going to undo
millions of years of evolution in a
lifetime of looking at faces
obviously she's saying Bar Bar Bar now
let's watch a slightly different video
bar
here we can clearly hear her saying far
far far
except that she isn't
the video shows are saying far but the
audio track is still of her saying bar
exactly if she did in the first video
the visual input overrode the audio
to make it Crystal Clear what's going on
let's look at this one more time
once again we'll have a soundtrack that
says bar but this time we'll have a
split screen with a face saying far on
one side and a face saying bar on the
other
as this plays move your eyes from one
side to the other and observe how what
you hear changes
so when do you guys hear
as you moved your eyes from one side to
the other did you say you're a change on
that weirded the crap out of me
I mean that was pretty wild so uh so
let's let's step back now that's a that
proved to me that you could have had my
perception
uh let's talk about what we can do with
an efforts in that perspective in
regards to Exxon so let's step through
and see what these different
Technologies are
so is this what Dr is can you guys read
that in the back
so heads up this next one's not me
so that's definitely VR for me that's my
favorite game that's what I spend a lot
of time doing my exercise now what
augmented reality
can anyone tell me what augmented
reality is
is it this
is the first thing that we started is
this what got augmented reality into our
vernacular it is for me and it is I
think it actually lent a lot of credence
to what we do on a daily basis that this
was the first thing that got augmented
reality into our vernacular but
augmented reality is also this
this is all about the metadata
that allows us to negotiate all these
different uh scenarios in this
particular case going through a
warehouse I want to know where that data
is I want to know where that outlet is
this is also meta daily
into where we are in the grocery store
and this is also metadata this is
slightly speculative there are a couple
cars that are actually doing this but
this is going to be amazing a couple a
couple cars that I know as of right now
I have some heads up display is and
they're getting more and more and we'll
get to this
what is mixed reality
[Music]
[Applause]
[Music]
Microsoft hololens brings Holograms into
your real world using transparent lenses
spatial sound and an understanding of
your environment Holograms look and
sound like they're actually part of the
world around you that is mixed reality
if you haven't seen the hololens
hololens allows you to basically use
mixed reality to scan your surrounding
with the sensors it basically has a
connect sensor in the band if you've
ever seen a connect sensor remember the
connect sensors back from the Xbox
exactly famous good stuff uh so it has
that entire cut sensor basically built
into the mixer and mixed reality
headsets so that's going to allow us to
do things like this it's a lot of people
just like today we can collaborate in
the scenario of an architectures group I
know I went to college with an
architecture major and the little
freaking toothpicks of wood was burp all
over the room all the time now this is
what they're using
iot heads up displays now we've got all
of these devices that have generated all
this data but they don't have displays
on them we're going to be able to take
that data and turn it into heads up this
way and we'll be able to see in
augmented reality
what is Exxon
XR is literally just extended reality
and it's a combination of b-r-a-r-m-i
how real could xrv
let's look at that
oh no
that is so cool
I'll have to go back I don't want
ah
oh my God oh my God
okay
so how does XR work
so
before we start we're going to talk
about degrees of freedom so degrees of
freedoms are the way that users can move
in XR back when we had the the go uh we
only had three degrees of freedom so you
could basically get your your yaw your
pitch and your roll but now with all the
latest devices we have the up down left
right front back so you have the full
degree so you can move through your
reality and it really becomes real at
that point in time
but your result song Works VR
communicates to your brain in a
different way than looking at a stream
when looking at a screen like our TV or
phone our brains read this as a flat
image the same way it would a picture if
an object on the screen gets bigger or
smaller or the person on TV moves
towards the camera you don't feel the
need to take a step back or move out of
their way but in VR you might want to
because you're not looking at one screen
you're looking at two and those two
screens are literally right in front of
your eyes projecting a slightly
different image into each eye which is
how our Vision Works in the real world
each eye takes in stimuli from a
slightly different Vantage plane you can
test this hold your finger up in front
of your face and wink each of your eyes
your finger should jump back and forth
that jump is the difference between what
your left and right eye are seeing and
the differences between what your IC
conveys depth it conveys three
dimensions this is known as steriopsis
and VR developers have spent a lot of
time I'm perfecting it but what uses
depth if you can't move through it which
brings us to the next most important way
virtual reality tricks your brain you
are the camera and it's fast enough
where your brain starts interpreting it
as your perspective head tracking allows
a person in VR to look and move around a
fake world in the same way that we look
and move around a real one if you look
to the left you'll see more of the world
to your left and if you look down
[Music]
yeah okay maybe don't look down other
subtle effects make virtual spaces feel
more real like 360 audio which is a big
part of the plank experience as you turn
your head the wind will subtly change
the plank creaks and if you listen
closely a heartbeat slowly begins to
speed up our brain takes all this new
virtual stimuli and begins to believe
this is reality
our brain has never really learned
within the last 150 000 years to
actually distinguish between computer
generated content and the real world Dr
Frank Seneca has been studying immersive
Technologies like VR for nearly 20 years
and if all the cues that we perceive
from the virtual environment are so
similar to the cues that we get in the
real world it is makes kinds of sense
that we are unable to clearly
distinguish between both our brains
quickly adapt to Virtual environments
largely because it's wired to trust our
sense of sight there's some research
showing that approximately 80 percent of
all the information that we perceive
from our environment are based on vision
and this allows VR developers to
manipulate our reality even further we
found out about 10 years ago is that we
if we guide users on a circular Arc with
a radius of 20 meters they have no
chance to identify that they actually
walk on a circle in the real world when
they see a straight path in the virtual
environment you can walk an entire
Virtual City without ever leaving a room
once we believe the environment is real
and accept that we're actually in it our
brains then go on to fill in some other
blanks there are interesting findings
that for example if you are in the
virtual world in a very let's say slow
or icy environment people feel cold
although they're in the real world in a
maybe in a hot environment outside of
games VR has actually shown a lot of
Promise in the medical world from
reducing pain in burn victims by
immersing them in a snowy World while
their bandages are being changed and
with exposure therapy that helps people
with phobias like a fear of heights and
body dysmorphia it's also being used for
physical therapy like assisting elderly
people with their balance right now
we're still Tethered to a system with a
headset on our faces the graphics are
nice but not perfect and so we're not
fully immersed when we hover 80 stories
above the ground that is we can still
remember to take the headset off Seneca
cautions that it might not always be
that way we can easily assume that
within the next five to ten years or so
we will not be able to distinguish
visually computer generated content from
from Real World content anymore and then
of course there are a lot of ethical
questions for now VR might not look
exactly like reality but it follows a
lot of the rules our brain has learned
to perceive as real and that's often
enough to make us sweat
so
through XR our brains become become
accurate recording memories why is this
important because this universe in
Maryland studied nine percent more
accurate between desktop computer
learning and virtual area nine percent
that that's pretty that's pretty
significant on the other hand 12 faster
uh traditional training versus VR
training
so those don't seem like big numbers but
nine to twelve percent could be the
difference between winning and losing or
an extreme Circumstance the difference
between light and death we're using
these training techniques in the NFL the
trains to train players we're using
these uh training techniques and
education if you can do 97 12 better on
your test and take you from a b to an A
or maybe even from an F to a c and you
live
uh XR activities our XR activates the
motor cortex and our sensory systems in
a way that height is highly analogous to
real life experiences and we can see
that in Mr exam and basically brain
scans and they can look at all of the
activation centers of your brain and see
it in your brain who uh has had 150 150
000 years of getting to where we are
essentially or cognitively cannot tell
the difference
between a good VR XR simulation and real
life you maybe know at this point in
time I say at this point in time you
know your brain doesn't know so we can
use that to put you into these scenarios
where you learn a lot better in a more
uh interactive with what we're doing
one thing I always get is you know I
don't want to try VR I get the I get
sick
so let's talk about that really briefly
what is VR sickness uh VR which is also
called VR Motions like this
so if you are sick because the cause is
a conflicting signals from your eyes
versus your inner ear so what your eye
sees and your inner ear as long as they
go together you're good but as soon as
your eye see something and your ear goes
ah I don't I don't sense it the leading
theory is that your brain says holy crap
I've been poisoned and the poison is
actually stopping the coordination
between those two things so when you're
poisoned what do you want to do
you want to do it the pumpkin's done so
that is the leading theory thus far and
it to me it has a lot of credibility
um this I mean I don't get sick on
roller coasters I don't get sick on on
you know cars boats but if you put me in
an uncomfortable game it's no good I'm
not going to be in that game for very
long
so as an XR user what can you do
take comfortable games they have a
rating when you go to get the games uh
pick the ones that are comfortable uh
pick a pick box loss for example is a
fantastic game where you're kind of like
a God over this little mouse and kind of
helping him negotiate you through
territory versus epic roller coaster
vomits I mean ever wrote up a roller
coaster coaster so do that now as an XR
creator
what to do well it's really important
to make sure that we get our frames per
second which when you are running a game
uh every time in a game Loop it keeps
running through that frame and you need
to make to be sure make sure that you're
displaying 60 frames minimum 60 to 90
frames or they get too slow and it can
cause uh it can cause VR sickness at
that same point in time anyone know what
teleportation is in a game
cool so what that basically is do
teleportation versus Force movement
teleportation is the you trick your
brain by doing little leaps so instead
of walking through the game like
dragging through slowly you do little
leaps
and that actually tricks your brain and
you can actually figure out what your
threshold is uh how many little leaps it
takes to go through there there are
things that they're working on uh in the
underlying operating systems uh that
will hopefully just make that ubiquitous
for all games you'll be able to set that
on a per individual basis but right now
take teleportation games or at least
pick that option and go through things I
really like fixed play area games like
beatsaber you're in a fixed place you're
kind of moving yourself space pirate
trainers and another excellent one where
you're kind of moving in your space
um so that's great and choose
I like mixed reality
versus versus strictly VR so if you use
Pastor anyone know what pass-through
mode is
so pass-through mode basically shows you
uh in in the quest uh it's really nice
because you basically see the background
you see I would see if I had the quest
set on now I would see a black and white
version of the entire crowd and that
gives me a really good reference to the
world so that I when I move around I've
got a very comfortable scenario I'm not
basically completely immersing unmvr
scenario so those are my tricks
um one thing that a lot that they're
working on now that I've seen some very
progress and the good progress on is
something called Galvin galvanic uh
vestibular stimulate stimulation GDs
that's stimulus there you go thank you I
got a call on call for experts uh so
that they do is they Place little pads
on your skin and send a 3d motion
signals to your inner ear tricking your
brain into feeling what you're seeing so
when they do that and they coordinate it
with what you're seeing visually you can
actually get rid of all those scenarios
but again that right now relies on
little paths attached to your head
so let's get the hacker
uh response so I'm gonna say we have
three categories just like most hackers
we have three categories here we've got
the white hackers in regards to VR and
these are people that are responsible
experimentations uh beneficial
manipulations
um then you have the gray hackers uh
irresponsible experimentations selfless
manipulations and addictions maybe
that's a company that puts uh
did you guys know that uh um if you give
little brief Snippets of cats popping up
that it actually triggers uh endorphins
in your brain that make you happy I
believe that explains the entirety of
the Internet by the way
so
you can have a little pop-up kittens
little pop-up things and that actually
makes it so you pay attention more for
the next 30 seconds scientifically
proven that you pay attention for the
next 30 seconds more than you did
previously if you did not don't know
that those kittens the puppies are
popping up
uh so that's an interesting one and then
obviously we have the black hackers they
can do malicious things and purple heart
purposeful harm and we'll talk through
some of those scenarios what those
purposeful harm can be
[Music]
nice
[Music]
applications
[Music]
no
[Music]
foreign
[Music]
[Applause]
huh
I know coyote
[Music]
tambien
[Music]
Restrepo
por favor
[Music]
[Applause]
thank you
foreign
[Music]
biometrically
[Applause]
[Music]
foreign
[Music]
thank you
okay I'll go
say so
[Music]
thank you
[Music]
[Applause]
[Music]
wow what a future that could potentially
be I mean there were a lot of great
things look at this part remember when
the road clears and it gives you really
clear information about a second because
someone is coming right yeah that
somebody's talking to me that that
safety you're under the road you know
the good you know the road get out of
the way it's going to give you some
information contextual information
there's a lot of great information in
there but however when this happens
that wasn't supposed to happen not the
half of getting in there
and then this
you can't take out your augmented
reality potentially maybe she has
contacts on
she can't take it off this person's
digitally masked themselves
and garbled their face so they can't see
what who they are and they're able to
come in there and then get a bio a bio
scan Trace from her getting her blood
okay theoretical not here yet but I find
that that has got both really good
points and really bad points and when
we're talking about really hacking your
perception that is absolutely perfect
analogy of what we're going to do
so now
this is again this is the part where
we're going to have more interactive
even more than we have so far let's
analyze some vulnerabilities together
what are the vulnerabilities here
first of all what is this
setting up the bathroom absolutely
correct
what are the vulnerabilities that people
think could potentially happen here if
somebody is able to get access to your
headset
you can block them into other dangerous
things you could welcome them very
interesting things uh what if it's near
your stairs and you as somebody could
basically extend that boundary slightly
out past that stance
very fascinating they've actually done a
study with this a lot of games are you
know pretty pretty stationary in the
middle they did something called the the
human joystick effect and you're able to
take that person and just subtly move
them and they don't realize that they're
moving because the environment looks
like it's 99 stable and the person
starts in one location and they take the
headset off and they're in a completely
different location they don't know how
they got there
yeah I mean again you could the the
researchers made some really wild Flames
at this point in time like maybe you're
trying to assassinate that person and
you maneuver them in front of a window
but a little while but still the ability
that you're controlling their particular
environment have you gives you a lot of
control over what they're doing
yeah
who knows what this is
and you're looking at your desktop this
is the way a lot of us are going to work
moving forward at some point in time
probably closely than we expected to be
and this is amazing you're able to have
a full keyboard you have to have all
your monitors outlayed in front of you
how many tabs you can have open metals
tabs all over your house you know what I
always I always say on my laptop I have
a tab addiction uh I have half the
memory for the operating system and half
the memory for Chrome
oh my God
and so but in this particular scenario
now we've got pass-through motor we got
something else like Pastor mode what are
the possibilities of if I'm a hacker and
have access to your system and Facebook
and meta haven't done a nice job of
securing this
you could
just like you can make what a capacitor
looks like because you can control
absolutely if you can take that data
and and be able to see it and then
manipulate it
you can absolutely change it but what
else about what about recording it yeah
ah what if you're going to have the
ability potentially again if it's a nice
job the med has done a really nice job
so far to with my experience of making
so developers don't have access to this
but they're not the only device in town
you can scan an entire entire like areas
of course
data and like have that all mapped up
yep you thought there were companies
that freaked out about Pokemon go yeah
literally freaked out you can't play
Pokemon go in our location because it's
it's whoever recording every bit of data
about our location did you invite giant
copies literally banned him
a game on your phone you're looking out
the window so you can catch landmarks
and stuff and be able to physically yeah
that's exactly right exactly you could
know exactly where the if their device
didn't tell you enough about where they
were you can basically figure out where
they are by by locator yeah so do those
snaps have glasses that they have lots
here today at the same kind of
capability of whether or not like to
triangulate like
so the triangulation happens at a
machine learning level yeah so as long
as they have the ability to capture that
video that's a possibility
so
but on the other hand there's a lot of
huge benefits from these things so the
question is we gotta Leverage The you
know we gotta weigh one versus the other
any anything else anyone thought about
here
let's go to this one now my device can
see my hands
you can see how I'm moving
what can what else what do you think
what's the problem with this
fingerprint somebody based on
they move
what they're actually typing yep that's
exactly right there
you got all their passwords by seeing
where their fingers are every single one
of them
talk about the the the man in the middle
kind of attack
but we'll tidy this into a second with
something else you have the ability to
see every way this person moves
you could build a digital twin of this
person
so the way you like you're doing
that like I know I've heard there to be
away with like the way that you tie them
selves a little ad into that kind of
Security will you be able to recreate
that through like actually
well as long as we can see what keys
they're typing right
I mean you're gonna be able to get
literally the entire Cadence of the way
that they're interacting
so that's something we got to watch out
for
I mean even if you can't see the
keyboard extrapolate enough data yeah
the distance between keys to figure out
exactly it's almost like a crypto a
crypto what is it crypto Cypher key
basically you figure out the bait on the
frequency that they hit X Y and Z key
you can then derive based on the
distance based on the frequency all of
the keys that they're doing
so what about this scenario that we
showed earlier this is the heads up
display
headless to send us this way
so this is the iot scenario there's a
lot of data coming from these devices
and these guys see these green these
these uh scanners and say oh it's green
it's good
what if it's not good yeah the boilers
will make the boiler explode you can
control all of that data at that point
in time
what about this one
foreign
actually 55. or hey
there are so many ways that depending on
the opacity of your windshield
uh I mean if it's able to go ahead this
is actually already in case they mandate
backup cameras on cars and they're thin
cases where the video feed will
momentarily freeze people keep backing
up because it's a clear view
because the the picture froze for just a
second I don't know that's but this of
course is way more you know but that's
cool that's something that exists so
yeah we've been taking people all kinds
of strange ways how about this one
[Music]
[Applause]
[Music]
he looks pretty good I have to put you
down to deep bank and then on top of
that so now we've got deep phase and we
know all the way that you move
you can make a complete digital tweet
himself right now
and take it over take over that
particular way my mom already did that
here
[Music]
nice all right
we're gonna be at the end here here's
something that I thought that was
passing VR diet
food that is virtually enlarged to be 50
bigger leads to 10 percent less
consumption
and while and while food that is
virtually shrunk by 30 makes people eat
15 more
so this this was a Japanese researcher
and the papers online and it's
fascinating and he was able to
manipulate that and over the scan of a
number of people was able to determine
these figures so what I mean if we
started having these nice augmented
reality views and we want to go on a
diet
we have VR start changing our food they
also did a study that uh you could
change the color of the drinks or of the
food and change how appealing they are
wherein you could actually take a pretty
pretty bland piece of food and actually
make the person perceive that it tasted
much better than it was after the fact
so these little things are really
amazing
question when they figure out this size
paper
cookies it does I know I would look like
Robert Downey Jr to my wife
[Music]
all right so we're gonna finish a little
bit early here but we have plenty of
time for questions here's my takeaway as
long as we responsibly approach the
future of XR with our eyes wide open to
the dangers the good that comes from XR
will greatly outweigh the bad enjoy the
XR of the day and look forward to the XR
of tomorrow so I think that's pretty
good I mean I would add to that please
coming from a company that has lots of
social controversial yes XR just like
social media platforms is a tool a tool
like a hammer can be used to build a
house or whatever you use to smash your
kneecaps yeah yeah yes exactly
everything's everything people who who
equates uh only bad that things aren't
looking nearly at the benefits that they
can basically bring with people so uh
if you're interested in uh programming
XR or anything along that line I had a
series of blog posts on there to get 30
requests
um please hop on there this I actually
show you how to go through uh and set up
your quest set up pass-through set up
controllers all kinds of stuff so this
is really
I love this view we love this view we
see the huge potential and we're doing a
lot of work in this particular field but
we know that there are not nearly enough
we can't do it all
and they're not nearly enough smart
people basically working on these
problems and are going to capitalize on
so we're trying to basically while we
learn our path and how to do it spread
that why spread it out to everybody here
and spread that along across the world
um so get involved and basically give
yourself the tool and first of all give
yourself the prop for being here
and watching these talks and engaging
Beyond nine to five
so you should all be very proud of
yourself for basically being here and
taking this time to invest in you
yourself and your future so thank you
very much
[Applause]
right here
[Music]
thank you