Most Research in Deep Learning is a Total Waste of Time - Jeremy Howard | AI Podcast Clips

Duration: 4:39 736 transcript words EN summary

Video summary

Jeremy Howard opens his discussion with a stark critique of the current state of deep learning research, asserting that most academic efforts in this field are essentially a waste of time. He argues that the scientific publishing system forces researchers to focus on topics their peers already understand and can easily recognize, leading to minor advances rather than practical breakthroughs. This structural pressure discourages scientists from pursuing work with significant real-world impact, resulting instead in an accumulation of studies that lack utility outside narrow academic circles. Howard emphasizes that true innovation requires solving actual problems quickly for students and researchers who need pragmatic solutions, yet the prevailing culture prioritizes familiarity over novelty or usefulness. The speaker highlights two specific areas where practical necessity has been ignored by academia: transfer learning and active learning. Transfer learning represents a world-changing capability that allows more people to perform high-quality work with fewer resources and less data, yet it remains under-studied in research papers because it is not trendy at the moment. Similarly, active learning—which involves optimizing how humans interact within machine learning loops—is rarely published on despite its obvious value. Howard illustrates this disconnect by noting that practitioners inside companies inevitably reinvent these concepts when they face real problems; for instance, engineers manually realizing that labeling only difficult classes saves time and money are essentially implementing an intuitive form of active learning without ever publishing it as such. Howard shares a personal anecdote to demonstrate the potential impact of practical research versus theoretical trends. He recounts writing his first paper on Generalized Low-Rank Matrix Factorization (GLM-IF), which introduced successful transfer learning techniques to Natural Language Processing (NLP). Although he initially created the algorithm for an educational course because no existing examples were available, a simple prototype developed over just a few days immediately shattered state-of-the-art results on major datasets in a field where he had little prior knowledge. His colleague Sebastian Ruder subsequently wrote up and published these findings at ACL, the top conference in computational linguistics. This experience shocked Howard with how quickly practical solutions could dominate once implemented, yet it also highlighted why junior researchers might avoid such paths due to career pressures regarding citations and publications. Ultimately, Howard concludes that while people do care about practical results when they are demonstrated effectively, the academic environment makes it difficult for early-career scientists to pursue them without risking their professional standing. Researchers feel compelled to choose "safe options" by making slight improvements on existing trends rather than tackling under-researched but vital problems like active learning or transfer learning in new domains. He admits his own lack of concern for citations or papers, noting that nothing in his life makes those metrics important enough to compromise practical utility. The core takeaway is a call to shift focus away from incremental academic exercises toward solving real-world challenges that can genuinely change how people utilize artificial intelligence with limited data and resources.

Read the full video transcript

[Music] so much fast they had students and researchers and the things you teach are pragmatically minded I practically minded freaking figuring out ways how to solve real problems and fast right so from your experience what's the difference between theory and practice of deep learning well most of the research in the deep mining world is a total waste of time all right that I was getting it yeah it's it's a problem in science in general scientists need to be published which means they need to work on things that their peers are extremely familiar with and can recognize in advance in that area so that means that they all need to work on the same thing and so it really Inc and and the thing they work on there's nothing to encourage them to work on things that are practically useful so you get just a whole lot of research which is minor advances and stuff that's been very highly studied and has no significant practical impact whereas the things that really make a difference like I mentioned transfer learning like if we can do better at transfer learning then it's this like world-changing thing we're suddenly like lots more people can do world-class work with less resources and less data and but almost nobody works on that or another example active learning which is the study of like how do we get more out of the human beings in the loop where's my favorite homage yeah so active learning is great but it's almost nobody working on it because it's just not a trendy thing right now you know what somebody's suicide interrupt you're saying that nobody is publishing an active learning but there's people inside companies anybody who actually has to solve a problem they're going to innovate an active learning yeah everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think gosh this is taking a long time and it's very expensive and then they start thinking well why am i labeling everything I'm own the machines only making mistakes on those two classes they're the hard ones maybe you ought to start labeling those two classes and then you start thinking well why did I do that manually why can't I just get the system to tell me which things are going to be hardest it's an obvious thing to do but yeah it's it's just like like transplant learning it's it's under studied and the academic world just has no reason to care about practical results the funny thing is like I've only really ever written one paper I hate writing papers and I didn't even write it it was my colleague sebastian ruder who actually wrote it I just did the research for it but it was basically introducing transfer learning successful transfer learning to NLP for the first time the algorithm is called GLM fit and it actually I actually wrote it for the course for the first day of course I wanted to teach people in LP and I thought I only want to teach people practical stuff and I think the only practical stuff is transfer learning and I couldn't find any examples of transfer learning and NLP so I just did it and I was shocked to find that as soon as I did it was you know the basic prototype took a couple of days smashed the state-of-the-art on one of the most important data sets in a field that I knew nothing about and I just thought well this is ridiculous and so I spoke to the best unit and he kindly offered to write it up the results and so it ended up being published in a CL which is the top link with computational linguistics conference so like people do actually care once you do it but I guess it's difficult for maybe like junior researchers or like like I don't care whether I get citations or papers whatever I was right there's nothing in my life that makes that important which is why I've never actually bothered to write a pic of myself now for people who do I guess they have to pick the kind of safe option which is like yeah make a slight improvement on something that everybody is already working on you