DEF CON - noRecognition: Could a pattern on clothing fool Facial Recognition? - Bill Swearingen
Watch on YouTubeVideo summary
The speaker, Bill Swearingen, opens his presentation by addressing the growing issue of mass surveillance in the United States, highlighting how individuals are often enrolled in facial recognition databases without their consent simply by obtaining a driver's license. He emphasizes that while these systems may perform adequately on lighter-skinned individuals, they generate significantly higher false positive rates for darker-skinned people, leading to unjust criminal accusations within over-policed communities. Swearingen introduces Flock, a private company operating hundreds of thousands of cameras that scan billions of vehicles monthly, noting that while such tools are useful for finding missing children or stolen cars, their current usage often bypasses legal warrants and allows law enforcement to track individuals inappropriately, such as monitoring an ex-partner's new relationship.
To counter this surveillance, Swearingen explains his research into creating adversarial patterns—specific designs intended to fool AI cameras. His initial approach involved digitally imprinting various patterns onto green-screen shirts and testing them against different detection models to see if he could confuse the system regarding the number of people or faces present. Although he initially believed he had broken facial recognition, further investigation revealed that his early work was based on outdated models. He subsequently refined his methodology by building a "gauntlet" to test patterns against modern commercial cameras, including Flock's systems, and enlisted the help of the global hacker community to provide the necessary computing power to run millions of tests.
Through an evolutionary engine and deep reinforcement learning techniques inspired by training a digital block to walk, Swearingen developed patterns that successfully lower the confidence scores of AI detectors without making the wearer invisible. He discovered that attacking the initial object detection stage or the specific person detector is often more effective than trying to obscure the face entirely, as removing the neck area can trick the system into thinking no person is present at all. His research demonstrated that a small percentage of his generated patterns were effective across multiple models simultaneously, proving that it is possible to disrupt these systems through clever design rather than brute force obfuscation.
Despite acknowledging valid objections regarding the potential for companies to retrain their models on these patterns and the fact that the most effective designs have not yet been printed on physical clothing, Swearingen remains committed to advancing this technology. He notes that his system continuously generates superior patterns faster than they can be countered, ensuring ongoing effectiveness against evolving AI defenses. To support the transition from digital simulation to real-world application, he has launched a Kickstarter campaign to fund the printing of "no recognition" t-shirts, aiming to equip individuals with clothing that can actively resist facial recognition surveillance in public spaces.
Read the full video transcript
Yeah. So, I I'm going to have to I'm
going to apologize first. Um I have a a
significant amount of friends in the
crowd here today and u [cheering]
and and if you don't already know my
friends, they they seem to treat me very
poorly.
>> [laughter]
>> Uh well, first of all, thank thank you
everybody for coming to my talk. Um so I
I want to talk about a problem that
that's happening across the United
States right now. Uh we we are emerging
as a mass surveillance state. Uh, did
you know I just from a raise of hands,
did you know when you get your driver's
license that you are mo more likely uh
to be enrolled without your permission
uh for your face to be enrolled into a
facial recognition uh system? Did
anybody has anybody aware of that? So,
this is why I love this crowd, right?
This this is my crowd, right? So, I I
love that several of you already knew
that, but that that's a problem, right?
that was never told to me when I was uh
14 and getting an ID that uh was already
uh enrolled. Now, now I am of a certain
age that probably didn't exist when I
was 14, but
uh so another stat that I want to bring
up uh before we even get into the talk,
uh for facial recognition systems, if
you look like me, uh if you look like a
50-year-old white male, uh the the
facial recognition systems will work
pretty good. I if I'm uh if I'm detected
and someone says that, you know, a
facial recognition system will say that
looks like Bill Swearen, more than
likely that's correct. However, darker
skinned individuals uh see anywhere from
10 to 100x false positives
for facial recognition. So, I want you
to think about it like uh you know, and
and I'm not going to get too political
any up here, but if you think about the
communities that are probably
overpoliced, right? If you think about
the the communities that perhaps are are
u say well this person has been known to
uh to correspond with terrorists. Uh
then you add in the fact that they are
uh 10 to 100 times more likely uh to be
incorrectly
identified as a criminal. I I think
we've got a problem.
Now, you know, it's it's kind of a sweet
spot for me. Uh this has been some
research that I've been working on uh
for well over a year. Um I I has anybody
heard of Flock?
>> So I want I want to introduce my Flock
system here. So this is a a Flock Falcon
uh 2.2. It is turned off. Okay. Uh just
to be clear, it's also wearing
sunglasses, but it might not, you know,
they might not be covering. But but I
just want to say that this camera is
pointed at the crowd. It is not
recording anything. There is a second
camera on stage uh right here and this
one is is pointed at me.
I want to talk about flock a little bit.
Um so I know that I know this crowd
probably already knows all of these
stats. Uh but but before we get into
this, I want to talk about uh some of
the problems that I see. So very early
on, flock was sold to neighborhoods and
homeowner associations, right? Um the
reason why they were they took that
route is traditionally it was very
difficult for law enforcement to uh get
cameras and other detection mechanisms
into our neighborhoods,
right? Uh so we we can all agree uh I I
can understand that uh stoplight cameras
uh those kind of things that you know
just traffic cameras uh you know we
we've we've got some hackers that are in
the audience that have that have done
work on on these types of systems that
it's okay I'm okay with those systems
but now we've got cameras inside of our
neighborhoods
right and then all of a sudden uh we we
start seeing that these these systems
have grown dramatically ally right
they're you you guys have all seen
DFlock I I am not associated with DOC
whatsoever but they're doing some
amazing work and we see the reach has
has has become a mass surveillance
system
I love whatever's going on over there
right now 120,000 cameras are performing
roughly two 20 billion vehicle scans a
month just in the flock system uh so
that 120,000 cameras, 20 billion vehicle
scans a month. Um, and you know, I don't
again, I don't need to go into this, but
in the flock system that allows them it
it gives that private company meta data
on all of those cars. What was the
color? What was uh the condition? Was
there bumper stickers? Was there damage
to a certain panel? uh what was their
license plate that that was that the uh
that the law enforcement teams or
whoever has access to Flock is able to
search.
This is a this is a video pulled
directly off of a Flock camera, a
deployed
Flock camera. Now, I want you uh Yeah.
So, one thing that I I I kind of missed
there on that last slide is they were
sold to be ALPR cameras, automated
license plate re recognition cameras.
These cameras were sold to track stolen
cars.
This video was extracted by Ben Jordan.
Um some of you guys may know that know
that person. Uh but I want you to look
at this. Look at this feed. We see it
tracking people.
We see it tracking a a guy just going
roller skating. We see it zooming in on
a phone at enough resolution that
potentially we could read what was
occurring on that on that phone. Now,
does that sound like an ALPR camera to
you? It it it doesn't to me.
This is where it it started. This is
what kind of caused me to to get
involved with this. Uh the the flock
system is what as I mentioned is uh you
know surveilling our entire country
right now and it's owned by a private
organization that makes this data
available to police forces across the
nation. Um I actually don't mind that at
all. I like if a if a small child is
kidnapped, um I want to use every
resource that is available just to track
down that child. If my car gets stolen,
I want I want the police to use whatever
resources they can to locate my car.
However, I believe that it should
require a warrant to uh to access data,
right? To mass surveillance data. Well,
we've trusted our law enforcement to
access the flock system. And and again,
I know that I know this crowd probably
already knows this, but there so far
there's been over around 20 cases, I
think last time I looked of law
enforcement uh utilizing these systems
in uh what we what we should be
considering illegal mechanisms. There
was a there was a sheriff, a police
chief that it that accessed uh the
camera out of Witchaw, Kansas, which is
not too far from my home, uh queried his
former partner and her new boyfriend
more than 200
times.
Okay, so the the police chief has access
to the flock system and he's checking in
on his ex-wife and her new and her new
boyfriend. Is that okay? No. No, that's
not okay. There's been there's been, as
I mentioned, there's been other cases of
of police uh of of police individuals
utilizing this data in ways that they
shouldn't. Now, I mentioned Ben Jordan a
minute ago. Uh he he asked the his qu he
asked his audience, would you trust
yourself if you had access to the flock
system?
I me either like I you know like we
we've probably all done it like I know
this crowd this is my crowd listen you
guys are my crowd I love you guys right
we're the hackers right we actually a
lot of times we have access to the data
that others don't have access to and and
we may hold ourselves account you know a
little bit more accountable and how we
access that data but if you were had
access to track any person on the face
of the earth or on at least in the
United States um I I I think that we
should uh you know consider how we're
doing that.
Hackers have always been the
counterbalance.
When we look traditionally
independent researchers have repeatedly
established what the true security of
certain environments are. Uh you know
the the vendors will tell you it's
secure. uh the our our governments will
tell you that it's tested, but it's it's
conferences like Defcon uh that have
have shown systems from telephone
signaling, right? Electronic voting
machines. Uh how about the the the
canvas in our vehicles? Uh it was Defcon
and and and other security conferences
is like like this and and and the hacker
community that pushed back and said, "I
took a look and it's bad."
[clears throat]
Oh, come on.
All right. So, this is exactly what
happened at uh at Blackout. We're going
to try it real quick.
All right.
My name's Bill.
>> Come on.
>> Hi. I expected you all to say, "Bill,
you suck."
>> Thanks. That felt better. Uh, I'm a
hacker. A and I'm taking a look at
hacking AI cameras. Now what what I want
you to see here is it should be drawing
a red box around me. Uh but the lighting
I I think is deal is uh causing some
issues. But what at the top I want to I
want to bring your eyes to the top that
shows the confidence that the model is
detecting a person. Um see if I can find
the right spot.
Well, it doesn't matter. So that's a
pretty high confidence. It's not not
found it where it's going to draw a box
around me.
I know.
[clears throat]
you,
>> Bill.
>> Maybe I'm not a person. But what I have
designed is I've designed adversarial
patterns that target the most modern
camera models. The model that's tracking
me right now uh giving me a confidence
of what does that show 50% saying that
I'm a person that is flock model. I want
you to watch what happens when I
introduce my pattern into the into the
image. Just watch the confidence.
[laughter]
So I can't really see. I hope it's I
hope it's uh working. Yeah. Um so this
is an adversarial pattern and this is
what we're going to be talking about
today and I'm going to explain how these
work and and how uh how we can generate
them. Um I've got a live I've got a a a
can demo but I think we all get Oh,
there we go. It's detecting me now. It's
Yeah, you suck. You suck flock.
Maybe it likes my butt.
>> [cheering]
>> Come on, you can do it camera. I love
you little buddy.
So, just the introduction of this
pattern, you can see reduces the
confidence and soon you'll understand
why that's important.
But before we go into anything,
[cheering]
>> I'm a member of SEKC and I'm super proud
to say it. Um, we are having a a party
tonight right here in the LVZC. Um,
kicks off uh tonight around 10:00 and
you guys are all invited. Uh, I I'll be
there if anybody has any questions,
comments, concerns, want to talk about
nerd stuff, I'll be there. Uh, but it's
going to be wild. We've got great
entertainment and those kind of things.
While I'm here, I also want to throw out
some shoutouts to the CDC uh and to uh
the DOPA crew. Um, uh, just they they
really helped out. I want to tell you a
story about how we got here.
[laughter]
So, so it started with a great idea. And
and what I love about this picture is
for all of us in in the audience that
have a partner that may not be in the
hacker community, right? We have to we
have to give them some grace that they
deal with us, right? Like every single
day there's a crazy idea. Every single
week, um we're we've completely changed
the direction. I thought you were doing
that. Now you're doing this. Well, um,
it came I came up with an idea and I I
asked myself,
what if I could create a t-shirt that
when I put like I what if I could create
a pattern that when I placed it on a
t-shirt caused a camera to fail?
I didn't know what I was talking about.
Um, I talked to Kristen and I was like,
I you know, this is something that I'm
I'm really interested and I think it's
something that I could tackle. This was
about a year ago.
Now the interesting thing when we think
about a camera
is a camera is just a parser.
It takes data in and it does something
and it outputs some other things. It
could be uh the detection like we talked
about that I I was attempting to show
you here uh with this camera. But we
also know that cameras do things like
how many people are in a in a uh in in
this image. Is that person Bill? I
didn't know how that worked, but one
thing I did know with a camera, I
control the input.
Now, no parser in the history of
computing has ever properly held the
boundary between data and instruction
once an attacker is able to control the
input. Right?
So, I set off to answer that question.
Can I control the output of a camera
based solely on my wonderful appearance?
So, here was the idea.
The idea was that I would create a
series of six personas
and I would put them in several
different types of fabric. Um, I used
the green screen uh capability. Uh, and
basically what I said was
I'm I'm going to use one uh model and
I'm going to ask it a series of
questions. I'm going to show it these
images one by one. And I want to
establish a baseline. So each one of
these images, I asked it, how many
people are in this image?
How many faces are in this image? And if
and have you ever seen this person
before?
And then I I uh with with some help of
some friends, not to be named. Where are
you? There you are. Not to be named.
um came up with uh 61
uh pattern generators. Uh they were
things like uh QR codes. Um I was more
creative in my pattern generators. Trent
wanted me to make a list of bad words.
Um yeah, and some injection strings,
those kind of things. And then and then
ask the model again once I apply that
pattern to the shirt or to whatever the
fabric. Again, simply ask the model
again, how many people are in this
image? How many faces? And have you ever
seen this face before? I wrote a little
tiny Python script. Um, and what I found
was shocking.
What I found was uh I found that there
were some differences.
So just the different the only thing I
changed in that image was what the
person was wearing digitally. Right.
Okay. So just make sure we understand
I'm still talking about a crappy little
Python program that is that is uh
literally just digitally imprinting a
pattern onto a green screen t-shirt. But
what I found was that I found anomalies.
I found extra people. I found no people.
I found uh just a whole bunch of
different of of different results. Um
and at that point it was running um on a
uh on actually on this M1 MacBook and I
was doing about uh 700 tests per minute.
Okay. But I was pretty proud of myself.
Um, and so I did what everybody does.
You know, you're you've uh you've now
written some code. It kind of works. So
I went and I told my friends all about
it. Um, I bragged and I was like, "Guys,
I'm I've broken facial recognition." And
I don't know if Joshua Marets's in the
audience tonight. Uh, but his literal
response to me is, "I don't think you
know what you're talking about." Right?
And uh it turns out Josh was right.
So on once I got a little bit of push
back um I I started to do uh wanted to
learn I I need you guys to understand
I've never done anything in machine
vision before. Right? This was a brand
new that was a brand new uh like
experience for me. I had never tested
these models but I was on to something.
So, but to make it even worse, other
than Josh telling me that I didn't know
what I was talking about, this work had
been done before.
I I am curious because some of this
research was was uh presented here at
Defcon. Uh has anybody done adversarial
pattern work? Uh if you have, so I' I've
got a couple.
So, I'm gonna tell you, um I after
reading the research that had had come
before me, uh what I found was that um
I'm standing on the shoulders of giants,
right? So, there was a there was an
artist, Adam Harvey, uh that I believe
it was 2010, um came out with CV Dazzle.
How in the world does an artist like uh
and figure out makeup that confuses AI
models, right? Like what a brilliant
person to to be, you know, to
groundbreak. And from there uh there
there's been several other uh other uh
companies and research that that's been
done uh in this space.
So I was a little I I was a little
disheartened quite honestly. Um, and you
know, maybe this had been done. But so
then I started digging into the research
and and quite honestly for the few of
you that that raised your hand, if you
disagree with any of this, please come
talk to me afterwards. I I would love to
hear this. But what I found was tr what
I found was previous research was done
on extremely old models, [snorts]
um, extremely small models, right? uh
that that didn't truly uh represent the
the type of of cameras that we're facing
today. Um but their research was
groundbreaking and gave me a tremendous
foundation to build on.
[clears throat] So real quick, I want to
talk about how AI detection works. Um,
and it's important as we as we move
forward in the talk.
So, there's really three types of
detectors, AI detectors in a camera.
I've [snorts] already mentioned them a
couple times, but
there's one. There's a person detector,
and its only job is to say, is there a
person in this image? And if so, how
many?
Then there's a face detector. Same
question. Is there a face in this image?
And how many? And then there's facial
recognition which says
do we know this face? Now
I applied hacker techniques to this po
this problem. So if I've got a chain,
can I bust the chain early and cause
facial recognition to not work?
Well, it turns out
that the person and the face detector uh
depend in most systems are not chained
together. Uh there is a person and there
I could show a pattern that caused
facial rec or face detector to fail. Um
and a person would still be shown and
vice versa. However, and it's a little
bit uh you know, it's a little bit
hindsight 2020. Uh, I was very shocked
to find that facial recognition didn't
work if a face wasn't present. It makes
sense now if there's no face to see that
we can't determine who this person is.
Uh, but that was a finding in my
research that I quite honestly I wasn't
really prepared for. Uh so just to close
the the the close this slide, if you can
prevent face detection from happening,
you have you will prevent facial
recognition.
So this is what it should have shown.
Look at that beautiful man.
>> So angry.
>> Yeah. So angry. Uh
the purpose of this slide is uh you know
to kind of talk through uh you know the
the different the the different models
but I also want to introduce you to the
the concept of confidence. Um so and
I'll explain this in later but a camera
never really truly sees a person um or a
car or a truck or a license plate. what
it sees it it asks itself several
questions and what what am I seeing what
do I think I'm seeing and then whatever
the highest confidence uh you know the
percentage is uh is is caught is brought
to the surface I think that this is a
person and I am 82%
sure that this is a person
so okay at this point in my life I'm
starting to to kind of understand how
cameras work. I'm starting to understand
where I may have some opportunity for
vulnerability. Right? We've already
proven with the with the Python script
that that I can shift some things, but
now that I understand the pipeline, I
can be a little bit more surgical on
where I attack,
but I don't know what what the models
what what do commercial models or
cameras what do they run? I don't know.
So this this is an eye chart. Um and
this will be my slides are posted. Um
but what I did is I you know again this
was probably October of of last year. So
AI wasn't wasn't quite where where it is
now, but I downloaded as many sbombs as
I could possibly get my hands on. I
downloaded as many uh licensing as I uh
for commercial cameras that I could get
my hands on. I tried to I I scoured the
internet for me to to to to try and make
a an educated guess on what what model
the underlying detection model that
these cameras are running. For example,
this is a Hickvision camera currently
banned in the United States. Thank you.
And uh but you can see up here uh we
have it listed that it's using the a
YOLO model or an SSD mobile net uh
model. From there
um I went and got those models
right
um and and started test. I'll talk about
that here in a sec. Now real quick
[clears throat]
there are cameras everywhere. If you
look up, if you have you have you ever
looked up, right? Uh there there's
cameras everywhere. Um and and so I just
I want to make sure that you understand
that in my testing, I'm I'm trying to
defeat them all. Okay? But every time I
talk about my every time I talk about my
research, I get I I get, "Well, what
about Flock?
Can you defeat Flock? And do you know
Ben Jordan?" Um, and
I don't know, Ben, did you make it?
No, it doesn't look like it. It's okay.
Um, so, so real early on, I wasn't even
testing Flock. Um, but later, I think
around June time frame of this year,
June of this year, I implemented uh,
Flock into my into my testing system.
But one of the things I did is I uh,
built a gauntlet. Okay, same strategy as
what I talked about before. You take a a
person wearing a green screen, you make
a baseline, you apply a pattern, and
then you run it through the the gauntlet
of of the models. So uh you know
starting um you know before ju June July
of 2026 I had 10 models but right now
it's running five person detectors it's
running four face detectors and it's
running two facial recognition systems.
So, um,
what what ended up happening was my 700
tests per minute dropped to something
like four
when when I implemented this giant
gauntlet of running through all of these
all of these models. Um, I truly ran out
of compute space. So, I'm going to need
more power, right? Um, and you're not
you're not going to believe what
happened. I put out I put out a call. I
put out a call to the hacker
[clears throat] community and the hacker
community delivered. Um I had people
from all around the world. Many of them
are are in the audience right now. Uh
that literally gave me their entire
computer right to run my system for
months.
The man in the red hat up there, that's
uh Coupe from Corn Con. If you guys know
Coupe, everybody say, "What's up,
Coupe?"
All right. Cooper literally took his
gaming computer apart and sent me the
GPU
just to help. What a cool guy. Uh the
rest of these guys like Yeah, give it up
for him. That like incredible.
The the rest of the ones uh pictured
here gave just as much. they were giving
me dedicated access to their GPUs and I
mean this for months uh there and you
have to understand I'm learning along
with this so um you know and so those
that participated knew that they'd be
like hey it's doing this I'd be like
okay try it now uh try it now like they
were giving me all of this time but the
hacker community came out and what we
did is we tested 31.7
million patterns that is a huge huge
number of patterns in there. We found
534,000
anomalies.
Earlier I mentioned what an anomaly was,
right? I asked the detector, "How many
people do you see in this picture before
I apply the pattern and how many people
do you see after?" and we found 534,000
anomalies of which 480,000
worked across models.
Uh this pattern is effective against two
or more models of which we found 85
that were effective across three or more
models. Okay, so I did some math. Um,
let's see. That 85 represents
16,000 of 1%
of the patterns that we tested were
effective across
um three or more models.
Now, that's still pretty early on in
life, but what that shows me is that uh
at that time I knew right then and there
um I'm on to something, right? and that
this was uh this was worthwhile. So I
got a little bit creative um with with
the program.
What I decided was, okay, if I am only
finding one 16,000 of a percentage of
successful patterns, I'm going to build
an evolution engine that tries to uh
mate
uh patterns that potentially are
successful in the person detector and
successful in the face detector. Let's
introduce them to each other. maybe
introduce some alcohol to the situation
and hopefully we get better patterns
that that come out. Um I am a genius.
All right. So uh so this this evolution
engine ran for about three months and
what I found was uh we were extremely
successful. the evolution engine uh was
was generating patterns so fast that I
couldn't even look at them. I had no I
had no way to to even see. Okay, so
we're talking about these patterns. What
do they look like? I don't know. They're
generating so fast. I don't know. Um
so I uh I paused the system and I said,
"Let's go. Let's go see what uh what
these patterns look like." And I'm
excited to show you. Uh, they're on the
next slide. I'm looking at them right
now. Uh, [laughter]
that's right. That's right. Thousands
and thousands and thousands of the exact
same image of the exact same guy wearing
a ball of clava, but they're different
because that one's orange, right? Um,
and I also just want to point out like
evolution kind of did its thing, right?
Like it found a s because this pattern
on this piece of clothing is very
successful against both a person
detector and a face detector.
[clears throat] However, perhaps
[snorts] my data set is a little flawed,
right? Like you know, I don't know.
>> [clears throat]
>> Well, I I got I got very lucky, guys. I
got very lucky. Um and just the stars
aligned and um I was I was uh
just basically had a uh an angel
investor say, "Uh, hey, why don't here
you go. Here's two Blackwell cards." Um
and that really changed my tactics.
Okay. So, while I was waiting the I
don't know 40 months it took to for
these Nvidia cards to show up,
um I came across this YouTube video. Has
anybody seen this before? It's Isn't
this amazing? So, I'm not going to show
it to you, but but what this is is this
uh is the best representation in
understanding how reinforcement learning
works. So this person took a block and
he gave that block arms and legs and he
uh introduced gravity and he and he gave
control of the legs and the arms uh for
balance to the block and it and he just
told it learn how to walk. Learn how to
walk. And it took I don't know like
200,000 attempts for this little cute
block to learn to walk. But eventually
through trial and error, he learned to
walk. Well,
that changed my approach in how I'm
going to start looking for patterns.
those 31 million patterns that we had
that the the the hacker community uh and
I had generated I I used as training
data for a for a deep reinforcement
learning system every bit the the
failures are just as important as the
successes what works what doesn't
uh earlier I mentioned we were running
61 pattern generators
throw that out and let's teach the deep
RL to paint. Right? So now I've got more
compute than I know what to do with.
Right? My house is so hot with with
these GPUs because they're running
non-stop. But what all of that became
training data for a deep reinforcement
learning system. I also said, you know,
those those early six probably wasn't
enough data for for me to to truly stand
in front of of an audience like you that
uh to to prove that we we've got some
success. So, I dramatically improve uh
increase the size of my persona images.
I wanted to make sure that um body
types, look at this body.
Thank you. [laughter]
>> I want to make sure that people of all
races, of all sexes, of all body types,
of all skin colors are represented into
my Holy cow. All right. Um are
represented in my data set. I even
included an older lady with a chicken.
>> [clears throat]
>> Um, so th this slide is is a little bit
more dense than my others, but I do want
to talk real quickly about how these
detectors work. Now, again, I'm focusing
truly on uh this flock camera. Um, so
what it does is it it shrinks down the
image into a very small 320x 320 320
pixels x 320 pixels box and it overlays
a grid on that box, right? And then it
in that grid it has um it's just making
guesses, right? So we've shrugged that
down. We've overlaid a grid and in that
grid the the model is asking in this box
do you see anything? Right? Do you see
anything? That would be the object
detector. It's the first gate. Are you
looking at pavement or are you looking
at something? Right? Not what it's
binary. Nothing or something. Then after
that we have we we have uh object
detectors in this flock camera. We have
person, car, truck, bus, trailer,
motorcycle, bicycle, and license plate.
Each one of them
uh makes a guess. So that person
detector says, I think I see a person,
but I'm 1% confident. Right? Or the
motorcycle one says, I think I see a
motorcycle, but I'm 80% confident. So
that becomes our confidence is
objectness. Do I see something? plus the
class probability I see a motorcycle at
u at 80%. Those are mult multiplied
which is gives us our confidence which I
showed earlier uh as as the confidence
number as a hacker I see that as two
opportunities to attack this model. If I
can attack the first one the first
object detector does a object exist uh
the game's over. If I can if I can
attack the person detector, do you see a
person and push that confidence down?
That's a second possibility for me to
attack. And then they get multiplied
together. That's good news for me.
So, um I'm going to have to speed up.
I'm I'm talking too too slow. Um but
what the adversarial pattern is actually
doing is when you when you take a look
at what the model was trained on, right?
So let's take a look at the a car. It
was it was trained on thousands and
thousands and thousands of pictures of
car and it was distilled. This model was
trained on these images and it was
distilled down to the the very smallest
that it can be that it's running on the
circuit board in a camera this size.
So, if you don't think that we can
defeat a model that has been shrunk that
low, you know, to that small, uh, I've
got something else for you, right? Um,
we can't, right? And so, what we're
trying to do is we're trying to use, um,
when you take a look at a human body,
there's a there's a very clear angle
here and a curve on your shoulder,
right? And there's um obviously a very
distinct um contrast difference between
the front of my shoulder and and what's
going on behind me. Um and so when I
inject a pattern, what I'm doing is I'm
putting um a lot of digital noise in a
very specific and repeating way that
causes those angles and those and what
the model was trained on to just lower
its confidence. Right? What I'm not
doing is I'm not disappearing from the
from the camera at all. What I'm saying
is when it has to make a guess at what I
am, it says I I don't know. I don't know
what that is. And it lowers the
confidence. Real early on, I thought for
sure the the best way to to beat all the
models was to drape someone in fabric,
right? I I was was very confident that
the more pattern that I could push into
that into that detector, the more input
I could get into that parser, the
better. And it turns out that's not
necessarily the case. Um, the body
detectors, the person detector, they're
looking right here. They're looking at
your chest. Okay? The face detectors
obviously are looking right here. But
what's interesting is the spot right at
the neck is vulnerable for both. If I
remove the neck that be you're no longer
a person. And that's real and that's
true in real life too.
Um so I'm speeding up. This is the OD
slide. Um this is every single model
I've ever tested uh has been defeated.
Every single one. All 11. Count them.
there's eight.
So, I I've been very careful in every
pattern that I've ever shown publicly uh
to be as uh to be successful
but as mid as I possibly can get. Okay?
Because what I don't want is I don't
want the camera companies retraining
their models on my patterns before
anybody even gets a chance. So these
images represent very successful
patterns, but these are the worst of the
best. Okay, of the three that are
missing, I have so few patterns of those
models. I have defeats, but those are
the best that I've got for those models,
so I'm refusing to show them here today.
Thank you.
[clears throat] All along the way, um I
I I'm running out of time. All along the
way, I've made every hard decision. If
I'm faced with which model do I choose,
do I choose the good one or the one that
might be deployed? I choose the good
model. I ch I choose the hard one. Over
time, I found um that I I needed to fix
my uh fix my research. I found um the
deep RL system that I implemented.
Remember the block guy that learned to
walk? He also learns to cheat. Um, I and
I I found him uh I him I didn't mean to
like give him a personality, but I I
found the D RL specifically targeting a
certain image a and and working back and
forth on on finding a pattern that was
just effective for that image. And as I
moved it across that 600 persona, I
found that it it wasn't.
Long story short, all along the way, I
have been checking my results, I found
cheating, and I have uh resolved that.
Um, you're this is any math majors in
here want to explain this to me. The the
more patterns that I find, the less I
need to adjust. I've the the cheating
happens when I'm only uh so at the very
bottom uh that's P2 is the person
detector hoodie. Uh I've only searched
678. And when I go back and have to
adjust my math, I'm finding the smaller
the sample set, the more I have to
adjust uh the the math major joke. I get
it. I understand.
Um, okay. I know that some of you don't
believe me, and I want to hit the
objections right up front. Um, first of
all, can Flock retrain their model on my
patterns? Yep,
you're right. That's a good That's a
good one. But every minute, every single
minute, my DRL system spits out about a
thousand patterns that were better than
the ones that came before. Every minute
I am testing crossmodel patterns. Can I
find a pattern that works across all
person, all face, all facial
recognition? Every single minute,
they're getting better. Okay, objection
two is a good one. Right now, where it's
at, it's only been digital. is only be
been simulated. It's only exists on
those GPUs. You're right and I need some
help. So, where I stand is um I have
been I have moved into physical testing
uh with patterns uh standing in front of
a camera like what we attempted earlier.
Um that's that's where it sits. It's ne
one's never been printed on a t-shirt
yet. You're right. That's a good uh
objection. Um, and lastly, uh, perhaps
the model weights on certain cameras are
better than the open-source models that
I'm able to get. Um, you're right. All
right, real quick. I'm out of time. Um,
if you want to help, I have a
Kickstarter uh that I that is going to
sell out. It's uh it's already at
something like 80% or something like
that. um where you can purchase um a uh
no recognition t-shirt. What this funds
is me printing on fabric. What that's
that's what all it funds. And with that
um please uh take a picture and or
whatever. And with that, I'll see you
guys at the sec party tonight.
[cheering]
[music]