Video summary
The replication crisis refers to the significant problem in scientific research where initial positive study results cannot be reproduced when the same experiments are repeated. To illustrate this concept, the video uses a demonstration where the speaker claims psychic powers to predict numbers between one and twenty; while some viewers correctly guess the number three, the majority do not. This scenario mirrors real-world medical studies where researchers might obtain a "positive" result simply due to luck rather than because their hypothesis is actually true. Consequently, a positive study does not guarantee that a treatment works in reality, just as guessing the correct number once does not prove one has psychic abilities. Similarly, a negative study does not definitively prove a drug is ineffective; it could merely be the result of bad luck or poor experimental conditions that masked a real effect.
The core issue with the replication crisis lies in the severe consequences of false positives and false negatives. False positive studies are particularly dangerous because they lead researchers to pursue treatments that do not work, potentially harming patients through unnecessary side effects and wasting resources on unproductive directions. Conversely, false negative studies cause scientists to abandon effective therapies prematurely, missing opportunities to save lives. The video highlights that while positive studies often receive extensive media attention and social praise, negative studies are frequently ignored, leading to a skewed perception of scientific progress. This imbalance is exacerbated by the fact that many highly cited studies have been found to be incorrect upon replication; for instance, research from 2005 showed that out of forty-nine widely cited studies, seven were completely reversed, and only eleven had no follow-up attempts at all, despite the immense time and money required to conduct such replications.
Several systemic factors contribute to the lack of replication in science, primarily revolving around time, funding, and academic incentives. Conducting a replication study is expensive and time-consuming, often taking years from design to publication, which discourages researchers who are under pressure to produce new findings. Furthermore, major funding agencies like the National Institutes of Health prioritize innovation over replication because their grant criteria favor novel approaches rather than repeating existing work. While regulatory bodies like the FDA require two independent trials to confirm a drug's efficacy before approval, the broader scientific community lacks similar mandates. This environment encourages scientists to chase novelty and notoriety rather than verifying established knowledge, leading to a situation where many widely accepted medical practices are later proven wrong through better-designed studies, a phenomenon known as medical reversal.
Ultimately, the video concludes that relying on a single study is never sufficient to establish medical truth, as luck and design flaws can easily skew results. The history of medicine is filled with examples where standard practices were overturned by new evidence, such as the discovery that aggressively lowering blood sugar in diabetics could cause heart attacks or the finding that allergen-proof bed covers offer no benefit for asthma patients. These cases underscore the critical importance of skepticism and the iterative process of replication to refine scientific knowledge. The best defense against erroneous medical breakthroughs is a healthy dose of skepticism, recognizing that one study can get lucky while another can go wrong, and that true understanding only emerges when results are consistently verified across multiple independent investigations.
Read the full video transcript
[Music]
we're now going to discuss the
replication crisis
we're now going to discuss the
replication crisis sorry that's a little
replication crisis humor for you so our
goals today are
to define the replication crisis like
what does this
actually mean why is there crisis about
it and
to discuss the difference between a
positive study
and a true study we'll introduce the
concept of
medical reversal and remind you that a
single medical study is
almost never the final answer to
illustrate
all of this though i am going to do a
demonstration for you so
i'm not sure if you knew this but i am
psychic and uh i'm going to project
into your mind a number between
uh 1 and 20. okay so i'm i'm sending it
to you now
and everyone have their number between 1
and 20. okay well
the answer is three
now most of you are very unimpressed
right now but but some of you
guessed three and think that i really do
have some kind of powers or
you you probably don't but maybe you
were mildly impressed for a moment
those of you who got three just
experienced getting kind of a
a positive result when the hypothesis
was not true
i don't actually have psychic powers i
just got lucky
nevertheless you got a positive result
and you might be excited to
to trumpet that and to to post on your
facebook feed
that the professor for this course
you're watching has real psychic powers
i do not the vast majority of people did
not guess
three i'm afraid
so this brings us to the four types of
studies
there is positive study of a true
hypothesis a negative study of a true
hypothesis
a positive study of a false hypothesis
and a negative study of a false
hypothesis so
so what the heck does this actually mean
so let's talk about a positive study
a positive study is a study in which the
data supports
the hypothesis you hypothesized
that your new drug would cure this
disease and the data in the study
supports that
that is a positive study we can measure
how strongly the data supports the
hypothesis using statistical tests and
we'll get to those in subsequent
lectures
but this doesn't mean my hypothesis is
true just like guessing the number
i might show that the data is consistent
with my drug
actually working but in this sort of
platonic ideal world like does the drug
actually work if we were to give it to
the entire world would it
actually work that may be false maybe we
just got lucky
like i did guessing the number three for
some of you so we can have a positive
study of a true hypothesis the drug
really does work
and we showed that it worked or a
positive study of a false hypothesis the
drug doesn't work we just got lucky
what's a negative study just the
opposite the data in the study
doesn't support the hypothesis the data
shows the drug doesn't work and that can
be because
in fact the drug doesn't work but
you can also have a drug that works but
you just had really bad luck when you
ran your study and because of the way
the chips fell
you don't show the effect that it works
and when we get to the statistical
analysis part of this course we'll sort
of show how that can happen
but you do have to realize that just
because a study says
that it is positive does not necessarily
mean
that the drug they're talking about
actually works
now what happens when there's a positive
study this is what happens
there are headlines across multiple
places in
on your social network feed that says
there's a medical breakthrough and
and there were seven apparently in 2017
that are going to
wow the world and and this these
scientists produced
artificial blood i guess and here are
the five most wondrous medical
breakthroughs this year so you know
there are seven here so you might want
to start there
my point is positive studies get a lot
of press and a lot of attention you
rarely see people arguing back and forth
on facebook
about a negative study this is what
happens when a negative study is
published right just kind of nothing
crickets not much going on
what does this have to do with
replication
well replication is really important
because
false negative studies are bad but false
positive studies
are really bad all right so if we have a
false negative study what does that mean
it means
that we had a drug or an intervention
that works it would work if we gave it
to everyone it would work it would save
lives but when we tested it the study
came out negative we didn't show
a difference between the group we gave
the drug in the group we gave the
placebo just because of bad luck
when that happens we miss out on an
important therapy potentially
and we might take the field in kind of a
non-productive direction we should have
stuck with that class of drugs and maybe
we move on to something else
so that's bad we don't want that to
happen
but false positive is is really really
bad because
there we've got a drug that doesn't work
doesn't do anything if we give it to
everyone it doesn't save any lives but
just we got lucky we guessed three and
we got lucky
in our clinical trial and so now we're
going to treat a bunch of people
unnecessarily
right we're going to potentially harm
them because most drugs have side
effects
and we're going to take the field in a
really unproductive direction because
everyone's going to chase after this
drug now and
drugs that are similar to this drug to
see if it works for the disease when we
were wrong
in the first place so the answer to why
replicate replicate is to check
your results for those of you who
guessed three back at the beginning of
this lecture
if i did it again and was right
again you would be even more impressed
right but chances are
that wouldn't be the case chances are i
wouldn't do quite as well
guessing two in a row and of course
three in a row four in a row five in a
row there'd be very few people watching
this
that i would have hit all the numbers
although granted that one guy who
had exactly the same numbers as me would
really
you know would really think i'm cool um
we measure how often replication happens
so this was an article that appeared in
the journal of the american medical
association in 2005
and what they did is they found 49
research studies that had been cited a
thousand times or more so like getting
your study cited a thousand times that
is a
very important study okay so a thousand
times these studies have been
cited of the 49 seven were subsequently
contraindicated the results were
completely reversed seven were
replicated but the effects weren't as
strong so the initial study showed that
the drug worked really well and then the
second study showed
it works but maybe not as well 20
replicated successfully so that's good
drug still works yep everything's on the
up and up here and 11
had no follow-up at all so a study that
was cited a thousand times
and just no one has tried to do it again
kind of crazy
now why aren't people replicating these
studies it seems like a no-brainer well
number one is time because studies take
a lot of time
it can take years from the point where
you start starting to design a study to
where you can publish replication
and many people don't have that kind of
time in their hands especially when
they're working on their own studies
number two money studies cost a ton of
money and very few people are willing to
pay
for replication studies which i think is
a travesty but i don't have billions of
dollars to give out to anyone
number three is money studies are
very expensive i'm deliberately putting
this in twice
because that is the major driver frankly
of
what gets researched it's someone has to
pay for it
and number four impact scientists are
obsessed
with how much notoriety or impact their
studies make and if you just reproduce
someone else's results
you'll get your paper published but no
one's writing a new york times article
about it right no and no
no reporters are calling you you've done
the world in a tremendous public service
and it's completely unrecognized
and so many scientists just choose to do
their own thing
the nih isn't helping much to be honest
and and full disclosure i have funding
from the nih but
um the grant review criteria from the
national institutes of health
have three major components the
significance of what you're researching
okay so
is this important for public health fine
the innovation
and the approach now the approach is how
you're doing the study that's
that's just technical stuff but
innovation is
one of the three major criterias
innovation means how new is this
has anyone done this before are you on
the cutting edge of technology
well replication is not innovative by
definition someone has already done it
before
so even our biggest funding agency does
not put a priority on replication
studies
now the fda on the other hand knows that
replication is important but remember
the fda doesn't fund studies
the fda tells pharmaceutical companies
largely what studies
they need to run out of their own
pockets so it's a bit of a different
when it's not
different when it's not your money on
the line the fda says before
uh before filing a marketing application
a developer must have adequate data from
two large controlled clinical trials
two there are rare circumstances where
they will allow just one trial but it is
for this reason
that the fda makes sure that you have
two
independent studies confirming that your
drug works because anyone can get lucky
once
getting lucky twice is pretty unlikely
now this brings us to the issue of
medical reversal medical reversal is a
term
coined by vinay prasad and adam chifu
these two guys
here defined as when a new trial
superior to predecessors because of
better design contradicts
current medical practice and so these
guys have made a study of these
medical reversals and they looked at
studies appearing in the venerable new
england journal of medicine over 10
years
there were 363 articles that they looked
at that
assessed the standard of care the things
that
like all doctors know you should do
right and
what they found is that forty percent of
them of those studies
reversed the standard of care practice
so forty percent of the time the thing
that we all know that you should do
we were entirely wrong about um
38 of the time affirmed it and the rest
kind of neither affirmed
nor reversed it so one of the major
points is that some of the things that
we
we believe to be true without strong
evidence
once we start replicating or even
improving upon the evidence
we do better um i'll give you an example
of a couple of
of sort of big reversals that happened
intensive glucose lowering type 2
diabetes diabetes is a disease where the
blood sugar the blood glucose is too
high
we all knew prior to the study that if
we got the sugar lower
people would do better right that just
kind of makes sense
no one really tested it or they tested
in limited ways
when it was formally tested we found
that if you drive the sugar too low
actually those patients do quite badly
and had more heart attacks and things
like that
so we still control the sugar but we
don't push it down
as hard as we used to here's a fun one
control of exposure to mite allergen and
allergen impermeable bed covers for
adults with asthma
so at one point in time we thought that
if you're allergic to dust mites
you can put these like might covers on
your on your
uh on your mattresses you know and they
zip up and they like prevent the mites
from being there and that would improve
asthma outcomes well when they actually
tested this
they found no it doesn't make a
difference whether you put a cover on
your mattress
or not your asthma is just as bad so the
importance of verifying results that we
believe to be true
is really critical to advance science
forward
so the take-home points here number one
and this is a huge take-home point for
the entire course
one study is almost never enough one
study can get lucky
one study can just go wrong one study
can be badly designed
a positive study doesn't mean the
hypothesis is true
it means the study is positive but
replication is necessary
a negative study doesn't mean the drug
doesn't work it doesn't mean the
hypothesis is false
it might mean that it often might mean
that but it's not the final say one
study is never enough
replication fixes science it
allows us to iteratively improve our
knowledge but
we are not incentivized to replicate we
are incentivized
to innovate finally the best defense
against medical breakthroughs
is a healthy dose of skepticism
thanks a lot