Andrew Ng: Advice on Getting Started in Deep Learning | AI Podcast Clips

so let's perhaps talk about each of these areas first deep learning that AI how the basic question how does a person interested in deep learning get started in the field the Atlanta AI is working to create courses to help people break into AI so my machine learning course that I taught through Stanford means one of the most popular causes on Coursera to this day it's probably one of the courses sort of if I ask somebody how did you get into machine learning or how did you fall in love with machine learning or will get you interested it always goes back to rain and you rang at some point you were employing the amount of people you influence is Ruchika so for that I'm sure I speak for a lot of people say big thank you know yeah thank you you know I was once reading a news article I think it was tech review and I'm gonna mess up the statistic but I remember reading article that said um something like one-third of all programmers are self-taught I may have the number one third Romney was two-thirds but I read that article I thought this doesn't make sense everyone is self-taught because you teach yourself I don't teach people and it's no good huh oh yeah so how does one get started in deep learning and word is deep learning that AI fit into that so the define specialization offered by today is is this I think one it was called service specialization it might still be so it's very popular way for people to take that specialization to learn about everything from new networks to how to tune in your network so what is it confident to what is a RNA nor sequence model or what is an attention model and so the design specialization um steps everyone's through those algorithms so you deeply understand it and can implement it and use it you know for whatever from the very beginning so what would you say the prerequisites for somebody to take the deep learning specialization in terms of maybe math or programming background you know need to understand basic programming since there are Pro exercises in Python and the map prereq is quite basic so no calculus is needed if you know calculus is great you get better intuitions but deliberately try to teach that specialization without requirement calculus so I think high school math would be sufficient if you know how to Mouse by two matrices I think I think that that deaths that desperates so little basically in your algebra it's great basically algebra even very very basically the algebra and some programming I think that people that done the machine learning also finds a deep learning specialization that bit easier but is also possible to jump into the divine specialization directly but it'll be a little bit harder since we tend to you know go over faster concepts like how does gradient descent work and what is an objective function which which is covered most lowly in the machine learning course could you briefly mention some of the key concepts in deep learning that students should learn that you envision them learning in the first few months in the first year or so so if you take the d-line specialization you learned foundations of what is in your network how do you build up in your network from you know single it's just a unit stack of layers to different activation functions you don't have a trained in your networks one thing I'm very proud of in that specialization is we go through a lot of practical know-how of how to actually make these things work so one of the differences between different optimization algorithms so what do you do the algorithm overfit so how do you tell the algorithm is overfitting when you collect more data when should you not bother to collect more data I find that some even today unfortunately there are your engineers that will spend six months trying to pursue a particular direction such as collect more data because we heard more data is valuable but sometimes you could run some tests and could have figured out six months earlier therefore this problem collecting more data isn't gonna cut it so just don't spend seconds collecting more data spend your time modifying the architecture or trying something also go through a lot of the practical know-how also that when when when when someone when you take the deviant specialization you have those skills to be very efficient in how you build is net so dive right in to play with the network to train it to do the inference on a particular data set to build an intuition about it without without building it up too big to where you spend like you said six months learning building up your big project without building any intuition of a small small aspect of the data that could already tell you everything needs you know about that date yes and also the systematic frameworks of thinking for how to go about building practical machine learning maybe to make an analogy um when we learn to code we have to learn the syntax of some very language right be a Python or C++ or octave or whatever but that equally important or maybe even more important part of coding is to understand how to string together these lines of code into coherent things so you know when should you put something in the function call and why should you not how do you think about abstraction so those frameworks are what makes a programmer efficient even more than understanding two syntax I remember when I was an undergrad at Carnegie Mellon um one of my friends would debug their codes by first trying to compile it and then it was sleepless code and then every line in the syntax error they want to give her the syntax errors as quickly as possible so how do you do that well they would delete every single line of code with a syntax so really efficient for general syntax errors were horrible service so I think so we learned how the debug and I think in machine learning the way you debug a machine learning program is very different than the way you you know like do binary search or whatever use a debugger I traced through the code in traditional software engineering so isn't evolving discipline but I find that the people that are really good at debugging machine learning algorithms are easily 10x maybe 100x faster at getting something to work so in a basic process the debugging is so the the bug in this case why is in this thing learning learning improving sort of going into the questions of overfitting and all those kinds of things that's that's the logical space that the debugging is happening in with neural network yeah the often question is why doesn't it work yet well can I expect it eventually work and what are the things I could try change the architecture malteaser more regularization different optimization algorithm you know different types of data are so to answer those questions systematically so that you don't heading down the so you don't spend six months hitting down a blind alley before someone comes and says why should you spend six months doing this what concepts in deep learning do you think students struggle the most with or sort of this is the biggest challenge for them was to get over that hill it's it hooks them and it inspires them and they really get it similar to learning mathematics I think one of the challenges of deep learning is that there are lot of concepts that build on top of each other if you ask me what's hard about mathematics I have a hard time pinpointing one thing is it addition subtraction is it carry is it multiplication law this is law stuff I think when the challenges of learning math and of learning certain technical fields is that a lot of concepts and you miss a concept then you're kind of missing the prerequisite for something that comes later so in the deep learning specialization try to break down the concepts to maximize the odds of you know each component being understandable so when you move on to the more advanced thing we learn your confidence hopefully you have enough intuitions from the earlier sections didn't understand why we structure confidence in a certain certain way and then eventually why we build you know our n ends and L STM's or attention more than a certain way building on top of the earlier concepts I'm curious you you you do a lot of teaching as well do you have a do you have a favorite this is the hard concept moment in your teaching well I don't think anyone's ever turned the interview on me I think that's a really good question yeah it's it's really hard to capture the moment when they struggle I think you put a really eloquently I do think there's moments that are like aha moments that really inspire people I think for some reason reinforcement learning especially deep reinforcement learning is a really great way to really inspire people and get what the use of neural networks can do even though you know networks really are just a part of the deep RL framework but it's a really nice way to the to paint the entirety of the picture of a neural network being able to learn from scratch knowing nothing and explore the world and pick up lessons I find that a lot of the aha moments happen when you use deep RL to teach people about neural networks which is counterintuitive I find like a lot of the inspired sort of fire and people's passion people's eyes is comes from the RL world do you find I mean first of all learning to be a useful part of the teaching process or not I still teach me forceful learning and one of my Stanford classes and my PhD thesis will endure for some writings thank you I find it if I'm trying to teach students the most useful techniques for them to use today I end up shrinking the amount of time and talk about reinforcement learning it's not what's working today now our world changes so fast maybe it does be totally different in a couple years I think we need a couple more things for reinforcement learning to get there if you get there yeah one of my teams is looking to reinforce the learning for some robotic control tasks so I see the applications but if you look at it as a percentage of all of the impact of you know the types of things we do is it's at least today outside of you know playing video games right in a few of the games the the scope I said numerous the bunch of us was standing around saying hey what's your best example of an actual deploy reinforcement learning application and you know among your like scene in machine learning researchers right and again there are some emerging but there are there are not that many great examples but I think you're absolutely right the sad thing is there hasn't been a big application impactful real-world application reinforcement learning I think its biggest impact to me has been in the toy domain in the game domain in a small example that's what I mean for educational purpose it seems to be a fun thing to explore new networks with but I think from your perspective and I think that might be the best perspective is if you're trying to educate with a simple example in order to illustrate how this can actually be grown to scale and have a real world impact then perhaps focusing on the fundamentals of supervised learning in the context of you know a simple data set even like an eminence data set is the right way is the right path to take I just the amount of fun I've seen people have with reinforcement learning it's been great but not in the applied impact on the real-world setting so it's a it's a trade-off how much impact you want to have versus how much fun you want to have yeah that's really cool and I feel like you know the world actually needs also even within machine learning I feel like deep learning is so exciting but the AI team shouldn't just use deep learning I find that my team's use a portfolio of tools and maybe that's not the exciting thing to say but some days we use internet some days we use a you know the PC a the other day are sitting down with my team looking at PC residuals trying to figure out what's going on with PC applied to manufacturing problem and sometimes we use the promising graphical models sometimes to use a knowledge trough where some of the things that has tremendous industry impact but the amount of chat about knowledge drops in academia has really thin compared to the actual rower impact so so I think reinforcement learning should be in that portfolio and then it's about balancing how much we teach all of these things and the world the world should have diverse skills if he said if you know everyone just learn one one narrow thing yeah the diverse skill help you discover the right tool for the job so if we could return to maybe talk quickly about the specifics of deep learning that AI the deep learning specialization perhaps how long does it take to complete the course would you say the official length of the divine specialization is I think 16 weeks so about 4 months but is go at your own pace so if you subscribe to the divine specialization there are people that finish that in less than a month by working more intensely and study more intensely so it really depends on on the individual who created the divine specialization we wanted to make it very accessible and very affordable and with you know Coursera and even higher education mission one of the things that's really important to me is that if there's someone for whom paying anything is a is a financial hardship then just apply for financial aid and get it for free if you were to recommend a daily schedule for people in learning whether it's through the deep learning that a a specialization or just learning in the world of deep learning what would you recommend how do they go about day 2 days or a specific advice about learning about their journey in the world of deep learning machine learning I think I'm getting the habit of learning is key and that means regularity so for example we send out our weekly newsletter the batch every Wednesday so people know it's coming Wednesday you can spend a little time on Wednesday catching up on the latest news through the batch on the on on on Wednesday and for myself I've picked up a habit of spending some time every Saturday and every Sunday reading or studying and so I don't wake up on a Saturday and have to make a decision do I feel like reading or studying today or not it's just it's just what I do and the fact is a habit makes it easier so I think if someone can get in that habit it's like you know just like we brush our teeth every morning I don't think about it if I thought about this a little bit annoying to have to spend two minutes doing that but it's a habit that it takes no cognitive loads but there's be so much harder if we have to make a decision every morning so and actually that's the reason why we're the same thing every day as well it's just one less decision I just get out in there where I'm sure so if I think you can get that habit that consistency of steady then that she feels easier so yeah it's kind of amazing in my own life like I play guitar every day for life forced myself to at least for five minutes play guitar it's it's a ridiculously short period of time but because I've gotten into that habit it's incredible what you can accomplish in a period of a year or two years you could become you know exceptionally good at certain aspects of a thing by just doing it everyday for a very short period of time it's kind of a miracle that that is how it works it's adds up over time yeah and I think is often not about the births of sustained efforts and the all-nighters because you can only do that in a limited number of times it's the sustained effort over a long time I think you know reading two research papers there's a nice thing to do but the power is not reading two research papers this meeting through research papers a week for a year then you've read a hundred papers and you actually learn a lot we read a hundred papers so regularity and making learning a habit give do you have general other study tips for particularly deep learning that people should in in their process of learning is there some kind of recommendations or tips you have as they learn one thing I still do when I'm trying to study something really deeply is take handwritten notes it varies I know there are a lot of people that take the deep learning courses during a commutes or something where may be more awkward to take notes so I know it's may not work for everyone but when I'm taking courses on Coursera you know and that still takes someone every now and then the most recent I took was a was a course on clinical trials because interest about that I call up my little moleskin notebook and I was sitting at my desk is just taking down notes so what the instructor was saying and that act we know that that act of taking notes preferably handwritten notes increases retention so as you're sort of watching the video just kind of pausing maybe and then taking the basic insights down on paper yeah so I should have been a few studies if you know search online you find some of these studies that taking handwritten notes because handwriting is slower as we're saying just now it causes you to recode the knowledge in your own words more and that process of recoding promotes long-term attention this is as opposed to typing which is fine again typing is better than nothing or in taking a class and not the canals is better than nothing any cause law but comparing handwritten notes and typing you can usually type faster for a lot of people you can hand write notes and so when people type they're more likely to transcribe verbatim what they heard and that reduces the amount of recoding and that actually results in less long-term retention I don't know what the psychological effect there is but so true there's something fundamentally different about writing hand handwriting I wonder what that is I wonder if it is as simple as just the time it takes to write is slower yeah and and and because because you can't write as many words you have to take whatever they said and summarize it into fewer words and that summarization process requires deeper processing of the meaning which then results in better attention that's fascinating oh and then I spent I think yeah because the Coursera spent so much time studying in pedagogy is that human passions I really love learning how to more efficiently help others learn yeah one of the things I do both in creating videos or when we write the batch is I try to think is one minute spent of us going to be a more efficient learning experience than one minute spent anywhere else and we really try to you know make a time efficient for the learning it's good to know everyone's busy so when when we're editing them I often tell my teams everywhere it needs to fight for his life and if can delete awareness easily to then not wait let's not waste than during this time wow that's so it's so amazing that you think that way because there is millions of people that are impacted by your teaching and sort of that one minute spent has a ripple effect right through years of time which is just fascinating talk about how does one make a career out of an interest in deep learning give advice for people we just talked about sort of the beginning early steps but if you want to make it at entire life's journey or at least a journey of a decade or two how did it how do you do it so most important thing is to get started right and ever I think in the early parts of a career coursework like the deviant specialization or it's a very efficient way to master this material so because you know instructors be it me or someone else or you know Laurence Moroney teaches on tensor field specialization and other things were working on spend effort to try to make a time efficient for you to learn new concepts so coursework is actually a very efficient way for people that learn concepts and the becoming parts of break into new fields in fact one thing I see at Stanford some of my PhD students want to jump in the research right away and actually tend to say look when you first come for us the peer student spend time taking courses because it lays the foundation it's fine if you're less productive in your first couple of years you'd be better off in the long term um beyond a certain point there's materials that doesn't exist in courses because it's too cutting edge the courses we created yeah there's some practical experience that we're not yet that good as teaching in a in a course the thing after exhausting the efficient course were then most people need to go on to either ideally work on projects and then maybe also continue their learning by reading blog polls and research papers and things like that um doing practice is really important and again I think is important to start spawns just do something today you read about deep learning if you say all these people doing such exciting things what about not building a neural network they change the world and what's the point well the point is sometimes building that time in your network you know be it m-miss or upgrade to a fashion em this to whatever doing your own fun hobby project that's how you gain the skills to let you do bigger and bigger projects I find this to be true at the individual level and also at the organizational level for a company to become clear that machine learning sometimes the right thing to do is not tackle the giant project is instead to do the small project that lets the organization learn and then build up from there but just true both for individuals and and for and for companies to taking the first step and then taking small steps is the key should students pursue a PhD do you think you can do so much that's the one of the fascinating things in machine learning you can have so much impact without ever getting a PhD so what are your thoughts should people go to grad school should people get a PhD I think that there are multiple good options of which doing a PhD could be one of them I think that if someone's admitted to a top ph.d program you know that MIT Stanford top schools I think that's a very good experience or someone gets a job at a top organization at the top of AI team I think that's also very good experience there are some things you still need a PhD to do if someone's aspiration is to be a professor here at the top academic University you just need a piece you do that but if it goes to you know start a completely build a company to create technical work I think a PhD is a good experience but I would look at the different options available to someone you know where the places where you can get a job where the place isn't getting a ph.d program and kind of weigh the pros and cons of those so just to linger on that for a little bit longer what final dreams and goals do you think people should have so the what options for they explore so you can work in industry so for a large company like Google Facebook buy do all these large companies already have huge teams of machine learning engineers you can also do with an industry sort of more research groups that kind of like Google research Google brain then you can also do like we say the professor neck as in academia and what else oh you can still build your own company you can do a start-up is there anything that stands out between those options or are they all beautiful different journeys that people should consider I think the thing that affects your experience more is less are you in discomfort versus that company your academia versus industry I think the thing that affects to experience Moses who are the people you're interacting with you know in a daily basis so even if you look at some of the large companies the experience of individuals and and he was very different and what matters most is not the logo above the door when you walk into the giant building every day what matters and Moses who are the 10 people who are the 30 people you interact with every day so I should tend to advise people if you get a job from from a company also who is your manager who are your peers who actually talk to you we're all social creatures we tend to you know become more like the people around us and if you're working with great people you will learn faster or if you get admitted if you get a job and a great company or a great university maybe the logo you walking know is great but you're actually stuck on some team doing really worth it doesn't excite you and then that's actually really bad experience so this is true both for universities and for large companies for small companies you can kind of figure out who you work quite quickly and I tend to advise people if a company refuses to tell you who you work with sometimes say Oh join us the rotation system will figure out I think that that that's a worrying answer because it because it means you may not get sense to you mean not actually get to team with with great peers and great people to work with it's actually really profound advice that we kind of sometimes sweep we don't consider to rigorously or carefully the people around you are really often this especially when you accomplish great things it seems the great things are accomplished because of the people around you so that that's a it's not about the the worry whether you learn this thing or that thing or like you said the logo that hangs up top it's the people that's a fascinating and it's such a hard search process of finding just like finding the right friends and somebody to get married with and that kind of thing it's a very hard search process of people search problem yeah but I think when someone interviews you know at a university or the research lab at a large corporation it's good to insist on just asking who are the people who is my manager and if you refuse to tell me I'm gonna think well maybe that's because you don't have a good answer it may not be someone I like and if you don't particularly connect if something feels off for the people then don't - you know that's a really important signal to consider yeah and actually I am in my standard cause cs2 30s was an ACN talk I think I gave like a hour long talk on career advice including on the job search process and then some of these cells of yours if you can find those videos on the oscillator pointers I'll point people to them beautiful you