How Research Works | Understanding Medical Research

[Music] well here we are in the very first lecture thanks for joining me and before we get into the nitty-gritty of how to conduct a medical study or how to interpret a medical study we're going to talk in a very broad sense about how medical research works in general the 10 000 foot view as we say here are the goals of this lecture we're going to describe the process of medical research how we get from a to z from a hypothesis to a manuscript we're going to talk about the concept of peer review we're going to explain what open data means and what its promises and how that promise sometimes gets broken and finally we're going to talk about some of the perverse incentives that underlie all of medical science but first i want to tell you about the greatest salesman in the world imagine you're in the market for a new food processor you decide to go to the number one food processor store in the world the new england food processor emporium inside a salesman shows you a beautiful box there's a picture on the front a lovely food processor with all the bells and whistles there are multiple testimonials best food processor i've ever used makes other food processors look like a garbage disposal saved my waistline and my marriage very impressive stuff you ask the salesman to open the box to let you see the miraculous device oh no he says i can't do that why not well everything you need to know is on this box it's really a great food processor yes but i'd love to you know plug it in see a demonstration or something sorry all i can show you is this box so what does that have to do with medical studies well medical studies are very much like that box that the salesman had they are nice and flashy on the outside and they tell you all the great things that are inside but very rarely do you actually get to see inside the box you don't get to see the data that generated all the results in the medical study you just get to see what the authors tell you now the vast majority of scientific authors are honest brokers and ethical people that really do want to advance health care in the country and the world but you do have to know that what you see is not always what you get and part of this course is going to be teaching you how to recognize what might be lying inside that box now a lot of times people wonder how can we be spending so much money on research and and it's a great question research costs a ton of money huge sums are invested globally in medical research and development and with good reason right the u.s spends more than 100 billion dollars a year on medical research and the majority of that comes from the federal government in the form of the national institutes of health the nih has an annual budget of 33 billion dollars just a tremendous amount of money going towards medical research so what do you get for that money well you actually get quite a lot and i think mary lasker said it best when she said if you think research is expensive try disease right and so just to give you a few examples of amazing returns on investment the human genome project was a 3.8 billion dollar investment at the federal level just an unfathomable sum of money to do a science experiment and yet the return on that investment is about a trillion dollars worth of economic growth and the explosion of a bunch of new fields of science so definitely worth it on a slightly smaller scale but perhaps no less impressive the women's health initiative clinical trial which was a clinical trial which evaluated hormone replacement after menopause cost a whopping 250 million dollars to perform that's a huge sum of money but resulted in 37 billion dollars of savings and economic output in the long run so medical research costs a lot of money but you make a lot of money too now how does research get from the idea phase to in your hand in a medical study well there are basically two paths one is the path of academia which is where i come from and one is the path of industry which are where a lot of our pharmaceutical companies are lying so let me give you a story here this was a paper we published back in 2016. that's that's me this was one of my papers looking at a clinical trial evaluating automated alerts for a condition called acute kidney injury and this is what we went through to get this paper published so first we had a research question we wondered if these alerts would benefit people we developed a hypothesis we said to ourselves yes we think they will benefit people in terms of reducing the risk of death and dialysis and other bad outcomes that happen to to patients who have their kidneys injured we had an idea to test the hypothesis a randomized trial and then you come to this critical question right how are we going to pay for this so we just said this costs a lot of money so you write a grant grant writing is central to the process of academic medicine it's what most of us spend a lot of our time doing and the way that works is you put together a big proposal and it can be upwards of 150 pages which you submit to a funding agency that is reviewed by a panel of your peers presumably and sent back to you they're often rejected or called for revision and that cycle can repeat over and over and over again until if you are very lucky and you're working on something that they are interested in your grant gets funded then you have the money you can conduct the study you collect the data analyze the data and write up a manuscript but it doesn't stop there that manuscript that you've written the results of your study doesn't just get distributed you know via the new york times or something you submit it to a medical journal and medical journals all want your peers again to read your manuscript and criticize it for them that's a process called peer review the manuscript will go to three or four experts in the field you're talking about and the experts will write back with questions what did you mean by this why didn't you do this don't you think this is another explanation for the results you see here that process can iterate multiple times as well the manuscript continues to get changed finally hopefully it's accepted by that major medical journal and the manuscript is published and that's how you got from a to z it will not surprise you to hear that this process can take a decade between when that idea emerges in your mind and when that final paper is published it is a slow process but it's one that often generates good results what about industry so this is an article that appeared in the new england journal of medicine looking at a drug called pembrolizumab to treat a certain type of lung cancer and broadly speaking the pathway is the same industry identifies a promising agent rather than a question really they identify a compound that they think might be active and help treat a disease they just go ahead and conduct the study they've got their own money in fact they have quite a lot of money so they can skip over that grant writing step altogether which must be nice for them they conduct their study collect the data analyze their data and write the manuscript and then the same process of peer review applies to them until eventually the manuscript is accepted the two major pathways leading from that first idea to that final published manuscript that you're sitting reading when you read a medical study now what is this peer review well we submit a manuscript and our peers read through it they are anonymous we don't know who they are and they are sometimes a bit rough when they respond to us so here are some examples of peer review the first one is is one that i received recently why the authors think an extremely convoluted technique developed in marketing should be taken seriously in clinical research where highly competent statisticians have developed techniques over decades of research is mind-boggling that was a less than constructive piece of criticism but most peer review is actually pretty good here's some other examples that i like since the manuscript is so lacking in all aspects i won't bother going through it in detail not a very productive piece of peer review and finally this paper is fluently written and meticulously researched i do not recommend it for publication so peer review hit or miss but i show you these because they're kind of funny and it's fun to to laugh about a little bit but a lot of peer review is actually really productive and makes the paper much better now publication a lot of people would assume that we researchers make money when we publish papers but it's in fact quite the opposite we are usually charged to publish a paper um we uh in the bigger journals that are more profitable that charge may be waived or in certain cases those journals are free to publish in but they are not paying you for these articles in any way the charges can vary quite a bit depending on the journal and we'll talk about some of those charges especially when we start discussing this concept of open access journals but we don't make our salary from publishing papers at least not directly now you might think if we've paid to have our paper published that you shouldn't have to pay to read it nevertheless when you go to try to read it you may run into any number of pay walls which is of course how these journals make their money and there's quite a bit of criticism that these journals are essentially making money off the backs of the taxpayer who is funding all of this research and i'll tell you how as a tax payer you may be able to get around some of these pay walls but nevertheless the state of play right now is that many of these journals do not give you access to this information for free at least not very easily so let's talk about getting around these payroll paywalls right i'm of the firm belief that information should be free especially information that you paid for through your tax dollars right so how if you see an interesting medical study how can you get it how can you get it in your hands well number one you can ask your local librarian so many libraries even just public libraries municipal libraries across the country have subscriptions to multiple medical journals and so you can just request this and they will print it out for you photocopy it for you not too hard number two pubmed central so there is a law that says if the government funds the research the research has to be publicly available and it's that is published on a place called pubmed central now the catch to pubmed central and the link is right here is that there's about a year up to a year lag between when the paper is published in the journal and when it appears on pubmed central so if that is a hot off the press paper you want to see you may not see it in pubmed central when you go there google scholar is highly useful if you go to scholar.google.com and type in the title of the paper you'll find multiple listings for it and oftentimes some of those are not behind paywalls so take a look and number four this is almost universally effective email the authors the corresponding author which is usually either the first or last author listed in the series of authors in the paper will have their email printed on the abstract of the paper which is usually in front of the paywall you simply email them you say i found your study very interesting would you mind sending me a copy and i have never had an experience where an author declines that so multiple steps to get around pay walls you should not have to pay for this especially for federally funded research that you've paid for once already now we're talking about seeing these papers so i want to talk to you a little bit about this concept of open access so open access sounds great the idea behind open access is we give you the papers for free open access no paywalls read whatever you want sounds great right but it's a bit of a two-faced god so the good side is that it makes this information freely available to everyone the papers get seen by more people broader dissemination more discussion and some journals are entirely open access a famous one is called plos one and you can go online you can read any paper that's there as soon as it gets published you can look at it great but there's a bit of a downside right someone has to pay for this so the charges to the researchers who are publishing the paper can be quite high ranging between two thousand and five thousand dollars to have their paper appear open access now once you introduce that kind of money they're gonna be people who take advantage of that and so there have um evolved what we call predatory open access journals these are essentially fake scientific journals that if you pay enough money as a researcher they will publish your paper now why would you want to do that as a researcher well it gets into some of the perverse incentives in medical research that i wanted to address before we get too deep into this whole course now predatory journals reach out to you all the time and ask you to submit your articles and uh i receive maybe 10 emails a day from various journals like this asking to submit an article and to show you that despite the fact that they say that your article will be rigorously peer reviewed that is not true this is a uh an article that appeared in the journal international journal of advanced computer technology called get me off your mailing list and the entire article just consisted of the author writing get me off your mailing list over and over and over again it was accepted after peer review and published for a fee of course now this tells you something right the international journal of advanced computer technology sounds legitimate it sounds legitimate to me how do you know that the journal you're reading is a good journal or a predatory open access journal there are a number of ways to tell how good a journal is or how reputable a journal is one way is if you know someone in the field you just ask them have you ever heard of the international journal of advanced computer technology and if they tell you no i've never heard of it and i'm an advanced computer technologist then it's probably not a great journal but you can look at other ways there's a concept called the impact factor and you should know what that is the impact factor is the average number of times that a paper that appears in a journal gets cited so if i publish a paper in a journal and it gets cited over the course of the next year or so you know 50 times that's a high impact paper a lot of people are citing that paper it's an important paper and so if you take the average of all those citations across the entire journal you can say the average article in this journal gets cited how many times and here's just a ranking of the top impact factors to give you a sense of where we are so you can see the cancer journal for clinicians is very high up there that's a journal that publishes mostly review articles actually but they do tend to get cited quite a bit but number two is the new england journal of medicine many of you will have heard of that a very venerable journal one of the top journals in in medicine with an impact factor of 72 which tells you that the average article in the new england journal gets cited by 72 other articles over some defined period of time which is which is pretty high these predatory journals might have impact factors of less than one which is to say that the average article doesn't even get cited once subsequent to publication so you can always google the the journal's impact factor to get a sense of is this article being published in a highly reputable journal now i told you that this is all sort of corrupted by perverse incentives for me as an academic why am i paying money to publish my research well one is that i you know i want to get it out there i want to tell people what we've done but my job is dependent on how much i publish if i don't publish papers and obtain grants then i don't advance in my career i don't get promoted and if you don't get promoted over a certain period of time you're kindly asked to leave and so there's an incentive to me to publish as much as possible there's an incentive to me to publish things that make my lab look good that make my hypotheses look good because i will reference those publications in future grants and say oh look what great ideas we have and that's really bad because we want everything to be published if i do an experiment everyone needs to see the results even if the experiment fails or else someone else is going to do the same experiment and fail again now pharmaceutical companies have a different perverse incentive and it's almost entirely based on dollars so they want to publish papers that make their drug look very good so that people will prescribe their drug now there are some rules in place to prevent gaming the system which we'll talk about in a subsequent lecture but you need to be aware of the fact that the science the medical science enterprise is not this pure uh uh activity you know from sitting lofty on high in the ivory tower that there are forces at play that make some of the research you're going to read not as reliable as you may think so our take-home points for today the medical manuscript is the end point of a very long process it takes a long time to get published you don't get to see much of that process you just get to see the box at the end the peer reviewers don't see much of that either right they just are reviewing the manuscript they don't get to see the data that went into the manuscript or very rarely do they systems are in place that make open data less attractive to researchers in particular the very high cost and finally the medical research system has many perverse incentives that encourage some bad behavior so knowing that bad behavior is out there you need to know how to identify it and we'll cover how to identify that those behaviors through the rest of this course