Lecture 1: Science Based Medicine vs.Evidence Based Medicine

hello and welcome I'm dr. Harriet hall also known as the skeptic and this is the first in a series of ten lectures about science-based medicine in this first lecture I'll explain what science-based medicine is why it's important and how it's different from evidence-based medicine in the second lecture I'll talk about medicine it isn't based on science complementary and alternative medicine orcam in lectures three through eight I'll cover individual topics in cam that claim to be science-based but that are not chiropractic acupuncture homeopathy naturopathy energy medicine and miscellaneous others in lecture nine I'll talk about some of the pitfalls in clinical research why we can't take every published study at face value and how to tell if a study is good science and in the final lecture I'll talk about how science-based medicine is misunderstood in the media and in politics I've prepared a course guide to accompany these lectures it summarizes the important information from each lecture and includes references and suggestions for further reading now the subject of these lectures is science-based medicine not cam so why devote so much time to discussing cam well there's method in my madness there are two reasons to concentrate on cam first I think that understanding what good science isn't is the best way to understand what good science is most of what I've learned about what constitutes a good scientific study I learned by studying the flaws and a lot of very bad studies bad examples can be very educational if you haven't learned to spot what's wrong in flawed science how could you ever be sure that something is good science and doesn't have any of those flaws have you ever read books about neurologic abnormalities like Oliver Sacks book the man who mistook his wife for a hat when part of the brain is damaged by a stroke or a tumor or trauma strange things happen abnormal behavior and loss of certain abilities that we take for granted those abnormalities are clues to understanding how that part of the brain normally works studying brain pathology helps us understand normal brain function and the principle of learning about the good by studying the bad works for almost anything for instance if you wanted to learn how to apply makeup it would be helpful to see examples of people who've used too much makeup or the wrong kind so that's the first reason the second reason is that much of CAM falsely claims to be based on science and we need to understand why mainstream science rejects those claims Jenny McCarthy says vaccines cause autism her autistic son Evan is all the evidence she needs she says my science is Evan and he's at home that's my science hope we can easily dismiss claims like that as Christopher Hitchens famously said assertions made without evidence can be dismissed without evidence but it's not so easy to dismiss a homeopath who shows ups his evidence in the form of a long list of published peer-reviewed scientific studies that he thinks have validated homeopathy we need to understand why his evidence doesn't stand up to scrutiny by the end of this course when you encounter someone who's promoting homeopathy or maybe selling a miracle diet supplement through multi-level marketing if he throws a scientific study at you and says here's proof that it works you'll know why his proof is no such thing and you'll have a rational basis for rejecting his claims now science is a wonderful thing it has improved our lives it's lengthened our lifespans and it's given us reliable tools to understand how the world works but for most of human history we had to make do without science evolution shaped our ancestors brains to help them survive in a world without science a world without newspapers books or the internet without even written language they had only two ways of learning about the world their own experience and the experiences of others if you ate an unfamiliar berry and got sick you'd learn not to eat those berries if you survived or a friend might tell you not to eat them because they made him sick and you might be able to profit from his experience sometimes what your friend told you might be wrong but in general paying attention to the stories of your friends was likely to help you survive to survive as hunter-gatherers on the African savannah our ancestors needed to learn from stories and they also needed to make fast decisions so they evolved to be very good at pattern recognition edit jumping to conclusions have you ever thought you saw an animal and then taking a closer look and realized you'd been fooled by a stump or a rock or something the other day I thought I thought I saw a dead bird out on our deck but it turned out to be just a crumpled brown leaf we look for recognizable patterns but we can be fooled by objects that have a similar profile now let's say you're one of our ancestors on the savanna you see something over there off in the distance and in the bushes and it looks sort of lion shaped but it might just be an illusion created by branches and shadows if you see the pattern as a lion and you're wrong you run away when you didn't need to but there's no real harm done on the other hand if you miss a real lion your mistake could be fatal richard wiseman explained the ability to find patterns is so important to your survival that your brain would rather see a few imaginary patterns than miss genuine instances of cause and effect that's the price you pay for being so amazing the rest of the time we see imaginary patterns everywhere for instance a baseball player notices that he was wearing a certain pair of socks when his team won so he develops a superstition and he keeps wearing his lucky socks his team doesn't always win when he wears those socks but it wins often enough to reinforce his superstitious belief evolution shaped our thinking processes for maximum survival advantage but it was a mixed blessing there were advantages to our skills but there were also downsides we got very good at pattern recognition and that helped us make sense of our environment it helped us identify threats like a lion in the bushes and it helped us learn the effects of poisonous plants or medicinal plants but it gave us a tendency to sometimes see patterns that aren't real and to develop superstitious beliefs about cause and effect we got very good at making quick decisions this allowed us to react in time to run away from the lion now instead of stopping to think about it but it made us more likely to jump to conclusions before we had all the evidence evolution equipped us with emotions that were very handy for survival fear motivated us to run away from the lion fast but emotions can interfere with judgment for instance the fear of death can lead cancer patients to grasp at straws and go to Mexico to try implausible potentially dangerous treatments like laetrile Morgan levy said thinking like a human is not a logical way to think but it is not a stupid way to think either you could say that our thinking is intelligently illogical millions of years of evolution did not result in humans that think like a computer it is precisely because we think in an intelligent way a logical way that our predecessors were able to survive so evolution equipped us with minds that kept us alive in a pre-scientific world by a strategy of paying attention to what others told us of seeing patterns that might not always be real and of jumping to conclusions then we discovered science and what used to be a survival advantage became a handicap the scientific method is a marvelous set of tools that allows us to ask specific questions about how the world works and to get reliable answers and science is a way of correcting for the kinds of mistakes our minds are prone to make but our mind is evolved to help us survive in a world without science and thinking in a scientific way doesn't come naturally our minds still work the old way we prefer stories to studies we prefer anecdotes to analyses if our neighbor had a bad experience with the Toyota we're likely to remember his complaints and not buy a Toyota even if Consumer Reports tells us that their tests found that it was the most reliable brand it isn't logical to value hearsay over data but we're just doing what comes naturally it takes a lot of training and discipline to overcome our natural tendencies how do we decide if a medical treatment works we're impressed by anecdotes and we jump to conclusions based on our observations if your friend says homeopathy worked for me you want to try it when you try it and your symptoms go away it's only natural to assume that it really works but sometimes we get it wrong your symptoms might have gone away for other reasons that had nothing to do with the treatment you have no way of knowing whether your symptoms might have gone away if you hadn't used any treatment at all assumptions can be wrong we make mistakes and no matter how smart we are every one of us can be fooled patients and doctors alike were fooled for centuries by bloodletting everybody knew that it worked the medical textbooks explained how it worked to balance the humors and everyone knew it saved lives because they'd seen it with their own eyes but when someone finally thought to compare patients treated with and without bloodletting they discovered they've been killing more patients than they helped big oops even modern doctors can be fooled not long ago doctors used to do an operation for heart disease where they open the chest and tied off chest wall arteries to divert more blood flow to the heart they had an impressive 90 percent success rate a doctor named Leonard Cobb was skeptical so he did an experiment with a sham surgery control group where he just made the incision in the chest and closed it back up without actually doing anything inside he discovered that just as many patients improved after the fake surgery in this diagram the two bars on the Left showed that chest pain improved almost as much with the sham surgery light blue as with the real surgery dark-blue and the two bars on the right show that the sham surgery light-blue reduced the need for nitroglycerin pills for pain even more than the real surgery the differences were in opposite directions and they weren't statistically significant essentially the fake surgery worked the same as the real surgery so they thought they had this great operation with a 90% success rate but they were wrong when they put it to a scientific test it turned out the operation accomplished nothing at all except maybe to suggest to patients that they would feel better so this was another big oops and doctors stopped doing that operation there are a whole lot of reasons people can come to believe an ineffective treatment works here are some of them don't bother trying to read them all now I'm going to go over each one of them separately one the disease may have run its natural course a lot of diseases are self-limiting the body's natural healing processes restore people to help after a time wounds heal a cold goes away in a week whether you treat it or not a few years ago I developed shoulder pain I was unable to raise my arm without pain and I couldn't reach behind my back to fasten my bra after about a year one day I suddenly realized I was back to full function and he had no pain at all my treatment nothing but tincture of time whatever was wrong had run its natural course and resolved on its own if I'd been using any treatment at the time I would have been fooled into thinking that the treatment had worked even serious life-threatening diseases can sometimes resolve without treatment before we had antibiotics to treat pneumonia it wasn't always fatal some people survived so before we could be sure antibiotics worked we had to compare the survival rate with and without antibiotics to many diseases are cyclic all the symptoms of any disease fluctuate over time fevers go up and down people with arthritis have bad some good days the pain gets worse for a while then it gets better for a while if you use a remedy when the pain is particularly bad it was probably about ready to start getting better anyway so the remedy gets credit it doesn't deserve that's known as regression to the mean here's what happens sarah has osteoarthritis pain in her knees here's the natural course of her pain during a random month the first few days of this particular month she has little or no pain as the middle of the month approaches her pain gets progressively worse and then towards the end of the month it subsides now this is without any treatment she's not likely to take any pain pills at the beginning of the month when the pain is minimal but by the middle of the month she needs relief so she starts taking pills and look what happens as she takes those pills the pain subsides it's only natural for her to assume that the pills worked but she doesn't know that she's only looking at the right side of this graph with the left side cut off her pain did exactly what it would have done if she hadn't taken any pills that she doesn't have any way of knowing that 3 we're all suggestible studies show that we think wine tastes better if it comes from the bottle with a higher price level if we're told something is going to hurt it's more likely to hurt if we're told something is going to make it better it probably will we all know this that's why we just our children zowie's anything that distracts us from thinking about our symptoms is likely to help for the wrong treatment may get the credit if you took mega doses of vitamins along with your chemotherapy you may think it was the vitamins that cured your cancer when it was really the surgery and the chemo sometimes the credit is due to something you don't even realize maybe something changed in your life at the same time one writer call this the spaghetti factor you happen to change brands of spaghetti sauce at about the same time and it was some herb in the new brand that caused the improvement 5 the diagnosis may be wrong you've all heard those claims that at secured my cancer when you track down the details of those miracle cancer cure is quite often it turns out that they never even had a biopsy so we can't know if they ever really had a cancer to cure even if they had a biopsy the results could have been wrong pathologists have been done to make mistakes in reading the slides and there could have been a mix-up in the lab where the sample was actually from another patient the prognosis may be wrong they said I only had six months to live but thanks to treatment X I'm still alive three years later but doctors can only provide an educated guess about the future and they may have guessed wrong all doctors really knew was that the median life expectancy for that condition was six months but that means half the patients died before six months and half of them live longer than six months sometimes much longer six temporary mood improvement may be mistaken for a cure if a practitioner makes you feel optimistic and hopeful you may think you feel better when the disease is really unchanged seven psychological needs affect behavior and perceptions when people want to believe badly enough they can convince themselves that they've been helped if they've invested time and money they don't want to admit that it was wasted we see what we want to see we may not remember accurately what septums we had when we remember things the way we wish they had happened when a doctor is sincerely trying to help a patient the patient feels a sort of social obligation to please the doctor by getting better and he may convince himself that he's better when he really isn't people have been known to deny the facts even to refuse to see that the tumor is still getting visibly bigger in one case a patient believed he was cured his doctor said look at the x-ray the tumor is still there the patient refused to believe it he insisted that that was just a shadow where the tumor used to be eight we confused correlation with causation now this is a big one just because improvement follows a treatment that doesn't necessarily mean that the treatment caused the improvement it's called a post hoc ergo propter hoc fallacy that's Latin for after that therefore because of that the rooster crowed and then the Sun came up therefore the rooster made the Sun come up well it's easy to see what's wrong with that reasoning but look at this I took a pill I got better therefore the pill made me better when we take a pill and then we feel better it seems compellingly obvious to us that it was the pill that made us feel better and maybe it was the pill but maybe it was just a mistake like the rooster we don't stop to think that we might have felt better for some other reason we don't stop to rule out all other possible explanations we jump to conclusions like the man who trained to flee to dance when it heard music and then he cut the fleas legs off one by one until it stopped dancing and then he concluded that the fleas organs of hearing must be at its legs correlation is not causation people notice that the number of autism diagnoses increased about the same time the number of vaccines for infants increased they saw a correlation and they assumed causation vaccines must cause autism but there was also a correlation with a number of pirates science studied vaccines and autism and they didn't find any evidence that vaccines caused autism science didn't study pirates to see if they caused autism I love this graph it shows an almost perfect correlation between autism diagnosis and sales of organic food does anyone think organic food causes autism correlation doesn't equal causation to find out if a correlation really indicates a cause a scientist named Hill came up with this list of criteria for determining causation I'll illustrate with the example of smoking in lung cancer there's no way we could do a randomized control study of smoking you can't divide people into two groups and make one group smoke and the other not you certainly couldn't have a blinded study because they wouldn't know whether they were smoking so we had to approach the question by other routes there was a temporal relationship people smoked first and got cancer later - there was a strong relationship lots more smokers than non-smokers got lung cancer three there was a dose-response relationship the more cigarettes smoked the higher the rate of lung cancer for the results of various kinds of studies were all consistent five the mechanism was plausible we know they're cancer-causing compounds in cigarette smoke six alternate explanations were considered and ruled out seven they did experiments where they exposed lab animals to cigarette smoke and the animals developed lung cancer eight specificity cigarettes produce specific kinds of lung cancers coherence the data from different kinds of epidemiologic and lab studies and from all sources of information held together in a coherent body of evidence richard fineman said science is what we've learned about how to keep from fooling ourselves we fool ourselves because of the way our brains work we prefer stories to studies we have cognitive flaws and biases we have perceptual failings just think of optical illusions and we're influenced by emotions and by psychological motivations biases and cognitive errors lead us to false conclusions that we wish to be true science is the only method that systematically controls for our biases and cognitive errors and allows us to obtain reliable information the author of this book a doctor named ruin birch has called it medicines beautiful idea the idea that no matter how certain we are that a treatment works no matter how reasonable a belief sounds we still have to test it today EB M stands for evidence-based medicine and it has been defined as the conscientious explicit and judicious use of current best evidence in making decisions about the care of individual patients but it wasn't until 1991 that the term evidence-based medicine first appeared in the medical literature before that we had other IBM's first there was expert based medicine where we unquestioningly accepted whatever the expert said Aristotle said men had more teeth than women he was the expert in for centuries everyone took his word for it no one bothered to look in mouths and count teeth and then there was experience based medicine where a doctor could say in my experience this treatment has worked well for my patients for centuries doctors had experience with bloodletting to balance the humors and they were thoroughly convinced that they were saving lives when they were actually killing patients my colleague Mark chrislam says that the three most dangerous words in medicine are in my experience because experience is so very compelling but is so very often wrong so we learned to rely on scientific evidence rather than on expert opinion and experience the first modern clinical trial was done by James Lind of the Royal Navy in 1747 back then the sailing ships went out for years at a time and sailors had no access to fresh foods sailors developed scurvy they became weak unable to work had internal bleeding fever jaundice convulsions bone pain bleeding gums their teeth fell out and they died a painful death Lind had heard reports of successful treatment with various remedies and he decided to test them to find out if any other really worked he divided a group of twelve six sailors into six groups he gave each group a different test remedy a quart of cider 25 drops of elixir of vitriol that's sulfuric acid I sure hope he diluted it in water before he gave it to them 6 spoons of vinegar 1/2 a pint of seawater yuck 2 oranges and a lemon or a spicy paste plus a drink made from barley the winner was tada 2 oranges and a lemon Lynde had shown that citrus fruit work to secure scurvy but he didn't have any idea how it worked today we know that scurvy is due to a deficiency of vitamin C but vitamins wouldn't be discovered for another two centuries Lind thought scurvy was caused by putrefaction and he thought citrus it's work because the acid in the fruit prevented putrefaction he tried sending ships out with bottled juice to save storage space but that didn't work the bottling process heated the juice and destroyed the vitamin C so they had to go back to using fresh fruit anyway the first clinical trial had found an effective treatment for scurvy British sailors became known as limeys and medical science was often running some doctors still think that in my experience is better than evidence here's one J Gordon pediatrician to the stars he caters to pay to parents who want an excuse to delay or omit vaccinations he says my very strong impression is that children with the fewest vaccines or no vaccines at all get sick less frequently and are healthier in general I truly believe they also develop less autism well his personal experience and uncontrolled observations have led him to false beliefs control scientific studies have proved that his beliefs are wrong that he stubbornly prefers his own very strong impression to the scientific evidence evidence-based medicine was a great idea treat patients according to the evidence from scientific studies whenever possible unfortunately the practice wasn't as good as the promise we coined the term science-based medicine to point out the flaws and the way evidence-based medicine was being implemented in 2008 a group of us skeptical doctors started the science-based medicine blog under the leadership of Steven novella if you aren't familiar with the blog I hope you'll check it out it's WW science-based medicine org science-based medicine is all one word here is the published hierarchy for evidence and evidence-based medicine the quality of evidence increases as you climb the pyramid the red bar near the top is the RCT that's the gold standard of clinical science the randomized control trial it uses a placebo control and is double-blind typically they compare an active drug an inactive placebo maybe a sugar pill that's the gold standard of clinical science the randomized control trial ideally it uses a placebo control and is double-blind typically they compare an active drug to an inactive placebo maybe a sugar pill the placebo is disguised so that it looks exactly like the active test drug in a double-blind experiment the patient doesn't know what she's getting and the person giving him the pills doesn't know what she's giving the pills are prepared by a third party who isn't otherwise involved in the research they are quoted and the code isn't broken until the trial is finished and the patients are randomly assigned to two equal groups to minimize any influence from other factors the results are analyzed to determine if there is a statistically significant difference even the best RCTs are not definitive there are a lot of things that can go wrong that could lead to false results I'll be covering that in lecture 9 we should never rely on the results of a single study we should wait for confirmation by other studies the lowest level on the pyramid is in-vitro research in vitro means in glass-like test tubes or petri dishes you can kill cancer cells in the lab with practically anything for instance of bleach or blowtorch but what happens to cells in a petri dish doesn't necessarily happen when they're part of a living body using a blowtorch on a patient wouldn't be a good idea the next level is animal research but the results of animal studies may not be applicable to humans for instance aspirin causes birth defects in mice but it doesn't do that in humans the next level is ideas editorials and opinions which are basically speculation by people who may be more or less qualified to speculate then case reports descriptions of a single patient then case series reports of several patients with the same condition then case control studies for example where patients with lung cancer are compared to patients who don't have lung cancer to see whether there are more smokers in the lung cancer group then the cohort study where groups of people are followed over time for instance groups of smokers and non-smokers to see which group eventually develops more lung cancer the EBM hierarchy puts the gold-standard RCT near the top of the pyramid but there is an even higher level all too often different RCTs get conflicting results to break the tie we can do a systematic review or a meta-analysis to combine information from several trials to reach a more accurate conclusion they are the highest level of evidence but they can only be as good as the individual studies sometimes they get false conclusions because of guy go garbage in garbage out and one really well-designed large study can sometimes trump systematic reviews of lower quality studies evidence-based medicine rigorously limits itself to considering only the evidence but something is missing from this pyramid there is no place in this hierarchy for evidence from basic science showing that a treatment is impossible or highly improbable it doesn't give any consideration to prior plausibility they tend to give equal credence to a study of a new antibiotic in a study of a homeopathic remedy they don't seem to care that homeopathy is incompatible with our knowledge of physics chemistry and biology they don't seem to understand that we would need far more evidence than a clinical study before we could accept that a homeopathic remedy was effective it would require a massive evidence sufficient to prove that all that other scientific knowledge was wrong now we don't reject new ideas out of hand but we do have to take prior plausibility into account we have to require stronger evidence for things that are implausible Carl Sagan said it best extraordinary claims require extraordinary evidence if I tell you I saw a Robin in my backyard this morning that's an ordinary claim you'd probably accept a cell phone photo as evidence or you might just take my word for it but if I tell you I saw fire-breathing dragon in my backyard well that's an extraordinary flame and anyone with a shred of common sense would demand much more proof not just a picture that might have been photoshopped but something more substantial like dragon scales or perhaps some dragon poop that can be analyzed for DNA if a study shows that a slightly different new version of an old antibiotic works better for pneumonia that requires an ordinary level of evidence but if a study shows that standing on your head and whistling Dixie cures cancer that would require a much higher level of evidence evidence based medicine doesn't work well for Cam it works on a one tool fits all approach evidence-based medicine focuses on clinical trial results it simply looks at the published clinical research and accepts the findings and that works pretty well for a conventional medicine but it doesn't work well when you're dealing with things that lie outside the scientific paradigm or when the scientific plausibility ranges from miniscule to non-existent my colleague David Gorski said I suspect that the originators of evidence-based medicine never thought of the possibility of evidence-based medicine being applied to hypotheses as unsparingly implausible as those of CAM it simply never occurred to them they probably assumed that any hypothesis that reaches a clinical trial stage must have good preclinical evidence and basic science evidence to support its efficacy science is a system of inquiry not a collection effects and not a belief system it's not just one single method but a toolkit of methods to figure out how reality works this diagram shows how the scientific process works it starts with observations and then a hypothesis is formed that might explain the observations if the hypothesis is true we should be able to use it to make accurate predictions we test to see if it does if we observe that the predictions are not accurate we have to modify the hypothesis if the hypothesis does allow us to make accurate predictions then evidence eventually accumulates and theories can be formed word theory is very often misunderstood in science a theory doesn't mean a guess or an opinion a scientific theory is an established body of knowledge like germ theory or the theory of evolution science is a discipline process for testing hypotheses it progresses slowly but inexorably as new discoveries built on old ones it doesn't claim to have the truth it reaches provisional conclusions based on the best available evidence and it's always ready to change those conclusions if new better evidence comes along studies are repeated and they're either confirmed or discredited a body of evidence gradually accumulates to the point that experts in the field can reach a consensus science makes mistakes but a self-correcting system is in place it's a collaborative effort studies are peer reviewed and scientists scrutinize each other's work science can be relied on to give us knowledge that works and it can be used to make accurate predictions and to guide medical treatment by the way there is no such thing as Western science there's only one science and it's the same everywhere in the world some people would have us believe that there are other ways of knowing like intuition tradition revelation The Stoned thinking favored Andrew while visions dreams extrapolation speculation and personal anecdotal experience well those other ways of knowing have their place they're useful when it comes to knowing if you're in love or knowing whether you can trust your neighbor or knowing if a picture is beautiful but when it comes to knowing whether a medical treatment is effective there's only one reliable way of knowing control testing using scientific methods those other ways of thinking can lead people to strong beliefs but we can't trust those beliefs to reflect reality only the scientific method can give us reliable knowledge no matter how convincing a claim sounds it must be tested before we can accept it as true no other way of knowing shows the kind of progress that science does with new knowledge building on old no other way of knowing leads everyone to agree and no other way of knowing allows us to make accurate predictions they say science doesn't know everything but as the Irish comedian Darrell Bryan pointed out science knows it doesn't know everything otherwise it would stop and just because science doesn't know everything doesn't mean you can fill in the gaps with whatever fairy tale most appeals to you science works science is probably the best thing humans have ever invented as Bill Nye the Science Guy says science rules in this lecture I've talked about what science based medicine is and why evidence-based medicine is not enough when we set out to evaluate health claims practices and products we want to use the best scientific evidence available in the light of our cumulative scientific knowledge from all relevant disciplines science-based medicine emphasizes that some evidence is better than others it's very aware of all the ways clinical studies can go wrong and can lead to false conclusions especially when improbable treatments are being tested it considers prior plausibility it puts clinical trial evidence into context with preclinical research and it looks for consistency with the rest of the body of scientific knowledge in the next lecture I'll talk about what happens when medicine isn't science-based