So this talk will be a little different. It's an idea that I've been kind of
that I've been milling around my brain for the past six months or so. So this is
the first time I'm testing it out with an audience. You're gonna help me
crowdsource it. So my name is Anita Nikolic. I just finished up a tour at the
National Science Foundation. I was a program officer for cybersecurity. NSF,
National Science Foundation, funds most of the basic research in the U.S. and
they fund almost 90% of computer science research. So security is underneath the
kind of guise of computer science. Specifically, I was a program officer for
networking security, social networking, and anti-censorship. So just to give you
some background. And prior to that, my background is mostly in networking
security. So I've been reading this book about kind of amateur scientists,
professional scientists, and how the twain sometimes meets, sometimes don't. I
saw this cool quote from Arthur C. Clarke. He's the author of 2001, A Space
Odyssey. And I thought this was really interesting. I definitely encountered
this a lot kind of in my time at NSF because there's a lot of distinguished
scientists that that come back and forth through the halls. So I thought this was
very interesting. So here's kind of the proposition. And as you'll see, I kind of
ran this past a bunch of people. And the semantics change depending on what your
preference is. But the proposition is there, you know, the past four years that
I've spent in the government at NSF. And prior to that, I was at a university.
Prior to that, I was just in IT operations. There is a lot of money that's
spent doing academic security work. A lot of this work never gets any place. And
we'll talk a little bit about that. It kind of lingers in papers, that's behind
paywalls. At the same time, there's a lot of cool stuff I hear at these
conferences. That sometimes doesn't make it, you know, the total feedback to
whether it's improving security in a product or getting it out to people who
could use it. That doesn't happen either. Sometimes it does. But the proposition is
how can we get kind of, I'll call them non-professional, you know, us basically
non-professional researchers, operators, and official academic researchers to
work together somehow. So some things to consider. Is this really even a
distinction between these two? Is this becoming meaningless, particularly for
security and privacy? Should it become meaningless or are there roles for each?
Should we have academics who, you know, sometimes can't push the limits as
much as we can, not being in academia? Or is research just research? Before, about
1970, academics could use money very freely. You saw it from World War II,
through the 50s, 60s, they could use money very freely. They were given money. They
could explore things. They didn't have to so much worry about publishing and
tenure as they do now. Modern day, there's something called peer review. And if
you're not familiar, kind of talk a little bit about it. But peer review
means these ideas come to a panel. People like me, we invite distinguished experts
to come and listen and be thoughtful about ranking these ideas. And peer
review is kind of designed to prevent major shifts. Everything is very
incremental in that when you bring up a novel idea or something really
transformative, your fellow peers often say, well, you know, that's not
safe. That may not work. And science has become more like a business. So what
prompted me to think about this? This isn't really anything new. I'll give you
some examples of where it has been happening. So I remember back in high
school, one of my teachers talking about that, and I don't have the
correct name of it, but this great books program in New York City. Basically, it
was kind of ordinary people talking with Columbia professors about, you know, the
great books. So they would bring people in, you know, it's more common now you see
meetups, but this is back in the 60s or 70s. And the insight was, wow, all these
kind of average people who read these classic works have really, you know,
insights that are pretty thoughtful. So that always kind of stuck with me. The
past four years I've been at NSF, NSF spends, and this is just one agency,
there are others, DARPA and others, NSF spends 80 million dollars a year on
basic privacy and security research. Many other agencies spend millions of it on
applied work, on basic work. Why are there still so many security problems?
One of the programs that I started there was this transition of practice program.
I stole the name from Homeland Security, but basically what I wanted to do was,
you know, that with all this money being spent, and this program that I was on was
around for ten years, so that's what, eight hundred million dollars, and some
of the same issues still are there year after year. So what what can we do to
transition some academic work and get it into companies, get it in the hands of people
who can use it? Many academics were very upset about this. They said, we want money
to just do basic security work and that's gonna take away from it. Getting
it more practical, we can't do that. So it's kind of a mixed bag and it evolved
over, you know, the four years I was there to people kind of accepted that we're
not taking away your right to do basic crypto work, but there's a lot of work
that's sitting there that never gets into the hands of people who can use it.
Another thing I noticed as I go to these a lot of these conferences, just I was
lucky for my job, I got to go to a lot of these, if you look at the agendas and
I didn't have time to do this, but kind of in the past I'd say, you know, five to
six years, you look at the agendas for these kind of cons and academic
conferences, they're very similar. The topics are very similar, the approaches
are very similar, but neither kind of wants to be seen at each other's
conferences. So when I'd say, you know, I'm gonna put in my travel, let's go to
DEF CON or any of the, we call them principal investigators, are any of them
going to academics going, they, many just will not go there. Can that be overcome
that, you know, people attend each other's events. Some talks that inspired
me at ThoughtCon last year, maybe the year before, the cyber squirrel, Chris Thomas
talked about, you know, the premise being squirrels are a bigger threat to the ICS
systems and actual hackers. I thought that was really interesting, so I brought
him into NSF and we usually get like Nobel Prize winners and serious talks
and he gave this talk. And it's interesting, half the room walked out and
half the room just thought it was awesome. I thought, you know, there's
something to that, you know, this is a really, this project is kind of
tongue-in-cheek, but there's a lot of kind of quantitative metrics and
approach to it that's very interesting. So that kind of struck a nerve with me,
there's something to it. I'm sure many of you know, you know, Jay Radcliffe, this
hacker researcher, talked about his project a few years ago about hacking his
own insulin pump and going to Johnson & Johnson and getting them to fix it, but
he had a nice talk last year and that kind of really inspired me to
crowdsource this about, you know, can we inspire kind of this community to maybe
use more of a scientific method to get some of our things that are not
academic research to get to get traction with it. Maybe take things a little more
seriously, maybe use a little more methodology and do we want to. So at NSF
we were asked a lot, you know, when an idea came by our desks or something,
what does the community think? What does the community say about this? You
know, and me not being an academic, I kind of landed by circumstance at NSF. I'd say,
well, who is the community? What do you mean by that? I mean, I came from security
operations. When you say the community, who are you talking about? If it's the
same people, you know, writing the same papers for the same conferences, is that
the community? Is that operators? So, you know, what does that really mean? So this
isn't a new problem. It got me to thinking, you know, are there other
disciplines, other areas besides security where this might have occurred? So I'm in
one of these meetups for science books. So one of the books they had us read was
about Mendel. And I'm not a biologist, not familiar with it, but it's a very
interesting book. And I learned about Mendel. This is if you recall your
high school biology, he's the father of Mendel's Law of Inheritance, which
explains heredity and how traits are passed down. Well, he grew up on a
farm. He worked as a gardener and a beekeeper, a quiet guy. He ended up as a
monk in this Abbey in Brno. And what he did for his many years was grow pea
plants. He literally just had pea plants in the garden and he did these
experiments. And he kept very meticulous notes. He wanted to develop new color
variants and examine hybridization. And this had never been done before. So he
did this for many, many years and published a paper which he sent to these
proceedings. He sent them to the Royal Society, the Linnaean Society, the
Smithsonian. And they just, because he wasn't a professional scientist, the
paper wasn't taken seriously. It was kind of put into this very obscure journal
and forgotten about. Well, when this Dutch botanist, a friend of his who was
cleaning out his stuff prior to moving, sent him this paper. He said, this is
amazing. So it took many years. It wasn't until 1909 where all these things were
finally pinned to the discoveries Mendel first described in that paper, which
just by happenstance, this paper happened to be found. So that got me
thinking, you know, there's, it seems like there's a similar thing going on in our
community. Another example, which I love because I live in Chicago, this is Sioux,
the T. rex Sioux at the Field Museum. It's the biggest, best preserved, if you
never seen it, it's this amazing looking T. rex. Well, Sue Hendricks, it was named
after her, she found it. And she was a high school dropout. She moved to Florida
to go diving. She lived with her uncle and she was an explorer, adventurer. She
caught these fish to sell to aquariums, rare ones she would just sell at cost
to museums. And she hooked up with some explorers and said, I'll help you dig up
fossils. And she ended up, their tire was flat so they had to spend the night
there. She said, you know, there's this ridge we haven't explored yet. And it's
kind of on her instinct of doing this for many years that she actually, not the
paleontologist, not the geologist, an untrained explorer had the tenacity and
she found this fossil. So it's another example, you know, I think astronomy is
another great example. Although in astronomy, you know, typically we have the
tools at our disposal in computer science and security versus like
astronomy. But there's a couple interesting examples. I never knew about
the Shoemaker-Levy comet, which of course many of us have heard about. I didn't
realize that Carol Shoemaker, who is one of the discoverers of it, at one point
she had found them the greatest number of these asteroids and comets of any
person. I think that's been superseded. But she was not formally trained. She was
a housewife. She didn't start her observations until she was 51 years old.
She just kind of was a fan. Her husband was an astronomer. She had no
training in this at all, but she kind of made a really big difference in this
field. So I thought this was interesting. I'm sorry my slides are a little all
over the place. But I put together, side by side, I just randomly grabbed a few
years ago, just randomly some talks from DEFCON and some from USENIX security.
USENIX security is a big kind of big event people like to publish in. So I
kind of put them next to each other and, you know, I wonder if people can really
tell which is which. Maybe some of you have been, but if you take a look at it,
kind of look at the topics. I don't know if anybody has a guess or if you maybe know
this already. So the one on the left is USENIX. The one on the right is DEFCON.
You kind of look through the titles and the topics. You could almost switch the
two around and you could be at one or the other conference. So some things
we're not talking about. Citizen science. There's this really good
article, which I know is a little hard to read on the bottom, but I could send you
the link. Really good article on citizen science. Citizen science is where kind of
scientists set up experiments and people help contribute to it. And mostly the
scientists get the benefits. They're the ones that get kind of the fame.
The same with SETI. That's where you donate your kind of compute power to
look for extraterrestrial intelligence. Bug bounties. I think that's a whole
topic in and of itself that has kind of an ecosystem set up already. Same as
these, I'll call them professional hackers, such as this HackerOne, which
helps facilitate the bug bounty programs. I'm also not really talking about kind
of, you know, research being done on your own accord. What I really would like to
get to is how can these communities share better across the boundaries. So a
lot of people call this this valley of death from, you know, a really cool idea to making
something happen or getting into a product. This is kind of gap between
research, whether it's academia or elsewhere, and how is it translated into
either better products, marketable products, something that can
be useful. This isn't new. There's a lot of these incubators. There's these
industry-university collaborations to, you know, generate commercial products. But a
lot of companies are eliminating or seriously scaling back their research
arm. I mean companies like, you know, Microsoft and RSA and Dell and these
kind of places. They're scaling back and what they're doing is just giving
$50,000 grants to academics and saying, you know, we don't want to keep
people full-time on staff. It's a kind of discovery front, which many times was done
by the Bell Labs and these big companies. That's not the case so much anymore. So I
ran some of these ideas just past four different groups. Academics, industry, I'll
say underground, kind of, you know, hacker community, and government. My informal
crowdsourcing, about 70% of people were kind of excited, like that's a good idea,
and about 30% mostly academics were like, no way, just it would ruin my
credibility, you know, I don't see it that I would get anything out of it. But I did
have quite a number of people who thought, and I'll go into some examples
that have been done in the past, that, you know, as long as you can assure there's no
damage done, and as long as you can assure this is done in a pretty ethical
way, that it's a cheap way to get good research, and it's a good way to get
people involved in something official. So if you're, you know, many of us have day
jobs, but we do kind of interesting researchy stuff on the side, it's a great
way if you want to get promoted or get something, you know, on your CV, it's a
great way if you just have altruistic motives, you really want to make a
difference. And it's a great way if you just want, you know, five grand to get
some stuff to tinker around with and help somebody. I mean, I think there's a
lot of different needs that can be satisfied. So just a short, kind of, if you
don't know how academic security and privacy studies are funded, it's just, you
know, I learned a lot about going to NSF, but before, as I mentioned before, about
1970, the funding was just, you know, you went to your university, you're a professor
there, you got money. One example in the UK is, it's called the Royal
Commission for the Exhibition of 1851. It's a granting agency. They used to
allow awardees to just pursue research, they give money and say, pursue whatever
research you want, wherever it goes, that's fine, it comes up, you'll come up
with something useful. Now you need to have a proposal be judged. So say we
go to NSF, you come in, and that's kind of the machine of proposal, right? And you
come in with an idea, you get a committee of people, they evaluate it, they
give you the money. And private foundations also give a lot of grants.
MacArthur, Gates give a lot of grants. And they have very little obligations on the
part of academic researchers. It's almost like a gift, as long as you can get the
money. But the metrics of success, by and large, are papers. I'm gonna skip
this just because it's late in the day. So some obstacles, and I'm sure there
are many, these are some obstacles I have found to engaging kind of the academic
side. A lot of the workshops, academic workshops, are invite-only. You don't hear
about them. Right before I left, I sponsored a really cool one, which is
great. It was great timing on trustworthy algorithms, you know, fake news. And it was
fascinating. But very explicitly, my boss, kind of the government folks, wanted just
academics. And of course, if you do that, you're not going to get necessarily
interesting opinions and ideas. Failure is considered bad. You think in science
that failure is encouraged in experimentation. Failure is considered
bad because then you don't have a paper that says you did something interesting.
Whereas if it's really, really incremental, you know, we call it just a
hack of something, that's praised because it's gone well. Conferences, if
you haven't been to an academic conference, it's mostly people reading papers that
were already put online. There's not a lot of socializing, except for maybe
finding a collaborator you by and large knew at grad school or some other
conference. The incentive in that world, and I think we have to be very
mindful, what's the incentive? It's tenure, if you're an academic, which requires
papers at top conferences, publishing often. At many, you know, MIT, Stanford's
nowadays, in the past about five years, they want you to show that you have some
form of startup or entrepreneurial experience. And of course, getting grants
from funding agencies, it's all about how much money you can get in there. So the
driving factor, you know, if you look at this in a negative way, is not necessarily
making security and privacy better in a tangible way. You know, we can look at
that and say, well, that is what they're doing, but in a very tangible way, the
goal is to get tenure, and that's by doing these things. Some obstacles to
engaging, I'll call non-academic researchers. Cons like this are
culturally a little hard to navigate if you don't know people, or you're
introverted, or you know, maybe you don't have something to do with the village.
So I think for a lot of people, they're culturally hard to kind of get
into. A lot of people don't attend these events, so their ideas or their work is a
little hard to track down. And people attend, many of them don't give speeches,
they don't, you know, broadcast their ideas. Of course, we know not all motives
are altruistic. A lot of the work, however, you know, I think is done by many
people for personal satisfaction, for the challenge. It's not necessarily, you don't
really necessarily want, you know, car companies to fix things, but for your own
challenge, to show that you had an idea and it came to fruition. And I think a
lot of people lack the interest in time and collaborating with any kind of
official entity. I won't read all of this, but so one, you know, one thing here is,
well, can I just read the research and get inspired? And it's interesting, I've
noticed the past four to five years, and I've heard it already today, and I've
only been here since mid-afternoon, is a lot of people do read academic papers or,
you know, want to be able to get to the original source. So academic conferences
might publish the full papers online. Workshops occasionally have the white
papers online. I find those two things, actually, I find them easier to digest
oftentimes because they're shorter. Since 2013, publicly funded grants or agencies
with that grant over 100 million dollars in R&D, research and development, have to
make the data publicly accessible to search, retrieve, and analyze. They're
still figuring out how, but it's via this memo. Journalists, however, are always
behind a paywall, almost always. It's a big problem with a lot of debate. There's
one publishing company that you have to pay them to publish, you have to pay them
to read it. So there's a workaround called Sci-Hub, which has 64 million
articles up there, and they claim to have 85% of the paywalled scholarly articles
up there for you to read. So a lot of people do that as a workaround through
the paywall. And I just found this really interesting quote that people have, you
know, time to become experts on quackery and pop science. Wouldn't it be nice to
start seeing them take up actual science as a hobby and be able to read
these papers? There's a really cool paper in the Atlantic, or article in the
Atlantic, a few weeks ago along this line saying the more kind of
sophisticated science becomes, the harder it is to communicate results. So if you
try to read some of these onerous papers, the concept's often very simple. The
communication of it can tend to be very onerous. So it's a very, it's a
fascinating article. It's kind of a long read, but very interesting. I just noted,
you know, if you're not familiar with these folks, I wasn't, but there's some
really interesting work that's been produced by, just like plucked out a
handful of people with some cool projects that I think would be kind of
interesting takers on this idea. Yoshi Kono, who's University of Washington, he
had this article on the top about encoding malware into a strand of DNA.
Damon McCoy does all this cool stuff on looking at the Silk Road and the dark
web and analyzing Craigslist rental scams and Nigerian gangs and things like
that. Reshipping and mule scams, it's how to use your Visa card to do reshipping
scams. Stefan Savage has this awesome project on measuring cybercrime and
actually how much money is made off the dark web and cybercrime. People talk
about this, you know, but he's actually measuring how much money is made there.
He also did this remote car hacking in 2010. So what if we brought the two sides
together? One of the, you know, we see this time gap kind of between academic stuff
and stuff you see at these cons. So, you know, if you went to DEF CON last year,
this voting village had tons of press attention saying it was a first-ever look
at voting security and they had all these voting machines for the first time.
People have hacked into them, but people have actually been doing this work for
years. It's just not published. It's not publicized very well. There are
tons of, you know, kind of obscure journal stuff and different workshops on how
people have hacked voting machines. But again, you know, were we to get that
out sooner, were we to kind of marry it up with people who could make a splash in
the press, you know, perhaps that can make a difference. This has nothing to do
with security, but I thought this was a super cool project. This guy came to talk
to us. He's a computer scientist at Brandeis. It's a project called Digital
Ahmadi. And he is a computer scientist, a mathematician, and for whatever reason,
even though he doesn't play an instrument, wanted to create the perfect
cello. So he spent like five years tracking down these artists and people
who make cellos. They're more into the art of it, not so much the kind of
equations of it. Well, he's really into the computational thinking. So he thought,
what if he could bring together his computational thinking and Euclidean
geometry and the art of this cello and kind of the science of instrument design.
So he did this thing called a geometry engine, and he wanted to design the
perfect cello to sound like these, you know, ones that were made in the 18th
century. And it took him years before people even... this famous one, he wanted
to do a CT scan to see, you know, what does it really look like? And it took him
like three or four years. He had to fly to Italy in his vacation time and
convince them that he was worthy of what he did. They said, you don't even play
the cello. Why do you want to do this? But it was his fascination with kind of the
perfect shapes and angles. And as it turns out, they have a wonderful
collaboration now of kind of arts and music and in computation. So I thought
that was just kind of an inspiring example. Some prior efforts at DARPA.
When Mudge was at DARPA, there was a cyber fast track where they gave micro
grants to just kind of average people. He noticed that, you know, the cyber
incidents kept increasing from, you know, kind of the five years prior to him
going to DARPA. But also spending on security was increasing by the federal
government. So what can we do? So one of his answers was, you know, kind of give
out these micro grants as an alternative to traditional funding. And
they averaged a week from proposal to giving the funds. One of the projects
that came out of it was the car hacking research that was presented in 2015.
Other efforts, I just wanted to put these kind of down. There's things like
experience tracks at academic conferences where you're not an academic
but you have experience as an operator. Case studies to show, you know, for human
computer interaction and in AI conferences. So there's kind of like
other thinking but nothing really formal. Some more efforts, bug bounties. I'm the
Cavalry is kind of a nascent or I guess not so nascent effort on promoting
public safety research. There's a lot of public private threat sharing, the ISACs
and kind of different places who share threat indicators. I thought this was
cool, this workshop in London, you know, where they were trying to get together
academics and kind of business people. They had some ideas, you know, a number
where you can kind of pair up a business person to bounce ideas off of and an
academic. Hackathons extended beyond just kind of the coding phase but, you know,
how can you, you know, have an opportunity to learn and pitch and talk
to each other. So things like that to kind of make it more sustainable than
just a hackathon weekend. So what can we do with the current model? Some
adventurous faculty have said, you know, they'd be happy to sponsor kind of non
academics to participate, to give them micro grants, encouraging particularly
undergrads who don't have a lot of these boundaries in the way they think. A lot
of the government agencies could easily kind of include this underground
component and I think inviting non academics to workshops. So in this kind
of underground area, what, you know, what can be done? Some of these crazy
academics who are interested in, you know, who have been pushing the boundaries
explicitly involve them. They cannot do offensive research with these grants.
They're banned from doing that. So this is an opportunity, many of them that I
spoke to said, this is a great way for us to not do it, but give you a micro grant
and you can try it and tell us what happened. So it's, you know, from their
thinking this is like a win-win for them. Maybe there's some matchmaking that can
be made at these different cons. It's interesting, many academic, like more than
more than a handful who I've run across who are, and some of them are crypto
experts, say only do they not use PGP. They can't figure out many of the open
source tools that many of us use. You know, simple things like LastPass and
other kind of tools because they're not used to the practical hands-on
experience. So can we, you know, school them on the just the practical design of
usability for some of these tools. So I think a part of the challenge of how we
is kind of non-professional, you know, community is how do we broadcast
what we know and find out. Many of you, many in this world are kind of these
media savvy people. Some are not. They don't want to be broadcasting things. Is
there a way, is there a kind of middle ground to help broadcast some of the
results? Are there things that haven't been tried? You know, a journal. Journals, the
challenges, who would peer review? It takes a lot of time. Who put the time
into organizing it, getting reviews? It's a great place to show off your work. It
takes a lot of work to write, to read, to peer review. Maybe a virtual journal.
Maybe for some of these CTFs, bringing academics and hackers together as equals.
You know, setting up, participating, doing the actual grunt work is something, you
know, that we learn a whole lot. Maybe there's a new joint neutral venue or
conference, physical meetings and virtual meetings, kind of bring people together.
Does the model work elsewhere? I just want to bring this up. So I was
involved in the bodybuilding world and it's very prevalent in sports
performance, at least bodybuilding. I put up there kind of the picture of Mr.
Olympia contenders and supplements. They're called gurus, which
makes you think of kind of the traditional guru. The gurus are like
these guys with the green check. They're like guys who are really good at
figuring out supplements, training, timing, all this stuff. So that was a very
interesting, you know, kind of convergence of you have people trained
in this stuff. They're not necessarily the ones that athletes are using. I think
the biohacking type stuff, quantified self, this guy Larry Smarr, he's an
academic computer science professor, does a lot of this quantified self stuff, but
he's got the tools at hand. He's got a lot of expensive tools to be able to
measure different things about his body. And you have people who go to meetups and
they're doing a lot of the same kind of thing, just kind of on a different level.
Could we get a feedback loop between the two? So that's that's my idea. Thank
you for sticking with me to listen through it. If you have some comments or
thoughts, or think it's stupid or great, or want to make something happen, let me know.