RCE 116: Jupyter

[Music] welcome to another edition of rce again this is brock palin you can find us online at rce dashcast.com where you can find links to our twitters our blogs all that fun stuff once again i have jeff squires from cisco systems and one of the authors of openmpi jeff thanks again for your time hey brock how are you it's getting to be fall here which means it's getting to be the ramp up time for super computing right so um once again i believe we both have booths this year uh yeah i am not directly participating anything but a number of us will be computing uh computing participating in uh panels on arm for scientific computing machine learning gpu computing a number of other different you know little things that we've dabbled in over the last couple of years so this is probably the most involved year we've ever been with supercomputing for my group yeah that uh that spans a wide gambit there and i'll be having the usual open mpi birds of a feather session um with dr george basilica from the university of tennessee given the state of the union of where we are in openmpi and where it's going all these things so love to see uh all of you there but enough about super computing let's talk about today brock who do we have okay so our topic today um we have dr brian granger to talk to us about jupiter so uh brian let me take a moment to introduce yourself yeah thanks so much for having me here today uh chapman brock and i'm a professor of physics at cal poly state university in san luis obispo and my background is theoretical atomic physics but for the last i don't know decade or decade and a half i've gotten very involved in open source software development originally in the scientific computing space sort of before data science was a thing and then as the entire sort of universe has shifted around this new idea of data science a lot of the open source projects i'm working on are right in the middle of data science and but also continue to be relevant in traditional scientific computing as well that includes project jupiter i'm one of the leaders and core developers of jupiter and ipython and then uh also in the last year i've been working with jake vanderplus on a new data visualization library called altair so that that's kind of funny because actually my degrees in nuclear engineering got into scientific computing started off with the classic you know scientific computing in the last couple years with the rise of data science i'm doing a lot more data science infrastructure for people in social science health iot and engineering and everything else so it's funny kind of parallel tracks yeah i think there's a lot of us who've taken very parallel tracks in that respect this is amusing i'm the odd man out on this podcast here where i am the only uh pure computer engineer we just consume what you do yes apparently so yeah so uh brian um can you give us a little bit of detail we're here to talk about jupiter what is jupiter yeah so uh jupiter is a sort of offspring of ipython and so i can start with a brief history of ipython it started in 2001 fernando perez a classmate of mine at the university of colorado in graduate school started ipython originally as an enhanced interactive shell for python this is sort of in the very early days of python starting to be used in scientific computing and uh fernando had long been a user of mathematica as i was and really missed a lot of the sort of niceties that mathematica offers for working interactively with code and so he started ipython in 2001 to bring that to python and for the first roughly decade it remained a terminal based interactive shell for python and then in 2001 we built a web-based notebook it was part of ipython at the time and uh over the following years we abstracted out an architecture that allowed other languages to plug into that web-based notebook and jupiter was born as sort of the the language independent part of the overall effort and so today we refer to that notebook as the jupiter notebook a python continues to exist as one of the uh language extensions or as we call a kernel for jupiter which provides basically the python language support for jupiter so where did the name jupiter come from and particularly it's got a a slightly odd spelling there there must be a story behind that yeah it it it was a long time for us sort of hunting and trying to find a name that met the basic constraints of you know something that was open on domain names open on github twitter etc and so that once you impose all of those constraints there's not much left we also wanted to name that sort of nodded in the direction of our scientific computing heritage and so you've got jupiter galileo sort of built into the name and then also the fact that even though jupiter is language independent and support we support many different languages our our python heritage as well and so the sort of changing the i in the name jupiter to a y is where we ended up um [Music] at times we've said the name also sort of has fragments of some of the main languages we support namely julia python and r so you could think of jupiter i think that's a little bit tongue-in-cheek but if your listeners want to take that seriously that's also completely fine some i think some people in our community that's the folklore that's emerged around the name and i'm i'm completely fine with that so uh for those who haven't seen this before uh why would you not just use a python julia or r ide yeah that's a great question and it's a question that is becoming more difficult for us to answer and so let me first give the answer looking back the the main thing that the notebook the jupiter notebook provides is uh interactive computing so when someone is running a computation and while it's still running writing more code submitting it looking at output and so it's really optimized around that type of workflow and that in combination with a more of a narrative format so the jupiter notebook is actually a document that a user would create that mixes live code with narrative text lotec equations uh visualizations and so at the end of it not only have you sort of been able to do your interactive computing yourself you end up with an artifact that is can be used to reproduce the work share the work with other people etc and so traditional ides don't offer those two aspects of sort of reproducible interactive computing and then what we refer to as a computational narrative that could be shared with other people this is the concept of literate programming right the idea that you can have all of your documentation everything embedded but uh it's more for learning rather than actual documentation yeah we we like to use the the word compu uh computational narrative rather than literate computing the original vision of literature computing was slightly different in that it wasn't designed to be an interactive experience it was more that you would have a single file that contained the source code and the documentation and then later run various post-processing steps to generate documentation or source code whereas what we're talking about here is actual an actual live interactive computation in the middle of a document that obviously looks more like a you know a google drive document or a microsoft word document than a traditional ide so yeah this actually kind of strikes me and i'm going to date myself a little bit by by making this reference but when i was doing research for this interview and looking into what jupiter is and how it works and whatnot it kind of struck me that this seemed like a google wave for science particularly with the timing back when you started this in 2001 or so that um you know google wave was the pre predecessor of google docs and google sheets and things like that where they kind of introduced a lot of this technology where you could even have multiple authors working in a single workspace at the same time and everybody gets a simulcast of exactly what's happening all throughout and stuff like that but you've taken that and run with it even further um to actually put computation in there is this was was there any relation to the google wave ideas at all or were you before that or did it happen at the same time or is that even an apt analogy or am i just speaking nonsense here that's a great question i i don't remember the exact year of google wave and to be able to sort of put it in the context of our our thinking but sometime i got involved in with ipython development around late 2005 and originally i had started to do work in more in parallel distributed computing in python and part of what fernando and i started to realize was that the the interactive computing experience was really something that we loved and other people loved but companies and efforts such as google and gmail we're starting to come out and show that the web could be used for more than just static content and so the idea of a web-based notebook was something that we started to talk about very early on in 2005 and it was it wasn't a single thing but it was really a constellation of new collaborative web applications that people were starting to build google docs is one of those gmail the the social media side of it was probably less of an influence i think for us at the time social media was more entertainment um than sort of a a productive work tool um but but that sort of broad new direction that many people and companies were taking of building rich collaborative interactive web applications was definitely something that informed our thinking and planning and direction of ipython and jupiter so while you were speaking there forgive me i did go off in research google wave looks like it came around in 2009 so you were several years ahead of that so quite the the pioneer but let's flash forward back to today um so from your definition here it sounds like you're you're not even intending to be a competitor to an ide this is an entirely different paradigm of doing work is that an accurate assessment increasingly those lines are being blurred and let me sort of describe what we found over the years so you know we we've really focused on interactive computing and what we've seen from ourselves and our users is that eventually those interactive computations start to look more like traditional software engineering so eventually you need to pull a function out of a cell in a notebook and put it in a standalone file and start to write documentation and a test suite and package it and it's been pretty painful to go through that transition from an interactive notebook to more full-blown software engineering and in response to that we're actually building a new user interface for jupiter called jupiter lab and honestly when most people see it they say hey wait that's an ide and and so it's uh definitely something that is becoming much blurrier in terms of traditional ides and then interactive computing and i think the way we're casting it now is that uh if if by ide you mean an interactive development environment then yes we're willing to commit to that the traditional notion of an integrated development environment that's focused on more software engineering type of workflows that's not there there's many other ides that are much better than that we're really focused on the interactive portion of the development process so if it's being less used for interactive um and people are really kind of using it for everything uh does running a kernel inside of jupiter introduce any performance overhead no not really so uh for your listeners the the word jupiter kernel is what we use to refer to the the separate process that we send network messages to and that process runs code the user's code in a particular language so there's a python kernel a julia kernel an r kernel and uh the the only thing we're sending to the kernel is a string of source code that that uh process can then interpret and run the there is a small amount of overhead to just sending that small string of source code over but once it starts running that code there's essentially zero overhead um in it you know we have for example a c plus plus kernel and uh once it gets that network message that has a source code it compiles it and runs it and it's you know full-blown native uh c-plus plus performance so how would you say the most common way people kind of you know start with jupiter and end with jupiter you know if they're using it in their project yeah sort of the i think the the sweet spot for jupiter right now is people that are doing uh a wide range of tasks in scientific computing and data science where you know maybe they're running a simulation or they're loading a data set cleaning the data set uh processing the results of a simulation doing statistical analysis uh doing machine learning based on uh data sets uh and then doing visualization uh in an interactive context and then at the end of it wanting to have something that they can share with other people and communicate their results that's sort of the the core use case for jupiter even as we support more traditional software engineering type workflows so you kind of already talked about the history between jupiter and ipython are you prime is jupiter still primarily used by python or is it kind of evolved into one of these other kernels becoming more popular yeah that's a great question and uh there's a a research group at ucsd that has recently scraped all of the notebooks off of all the public notebooks off of github i think there's 1.2 million jupiter notebooks on github and they're starting to look at these notebooks to help learn about how people are doing interactive computing and i'm pretty sure they were the ones that mentioned that of the existing notebooks on github i think it was something like ninety seven percent mid ninety percent were still python now it's very possible there's some sampling bias there that other communities are not putting their notebooks on github like the python community is but based on our observations it there's still a very large fraction of our user base is python that's also helped by the overall popularity of python in this space is another use case because you mentioned latex in there as well do have you seen anybody write a paper uh specifically in jupiter and like have their graphs and charts and what not be active computations so that they could produce say a pdf that actually represents um you know an integrated set of work rather than oh i got to have my my scripts over here that generate my pdfs of graphs that then get slurped in the latex and blah blah blah blah that whole kind of thing has it been used to create publishable results like this yes and no so the the narrative text in a jupiter notebook is marked down and uh even though we support latex in the markdown cells markdown is a little bit too limited to author sort of full-blown publication content it lacks a lot of features that you need for that and because of that the main way we're seeing people use this in a publication context is as uh sort of accompanying material for a formal academic publication that's being done quite often so a great example of that and it's a perfect day to mention it is the ligo collaboration which uh discovered or observed gravitational waves and actually won the nobel prize for that just today that was announced they any time they have a an observational event they actually publish a jupiter notebook that reproduces all of their analysis that goes into the associated peer-reviewed publication and that type of usage pattern is something that we're seeing quite often and that a lot of academic publishers are quite interested in so let's talk about the guts and the way this works a little bit you mentioned a couple languages but out of the box if i install jupiter what language is slash kernels does it support so the jupiter itself i actually that's a really good question today i think if you for example installed jupiter with pip or conda the only kernel we will install is the python kernel that we build that the ipython kernel and then any other kernels that you would install beyond that you would have to install separately and the reason we've done this is that we ourselves uh the sort of the core team core jupiter python team uh only maintain a very small number of kernels most of the kernels built for jupiter are developed by third parties and so it's completely up to those third parties how you would install those uh also the many of those other kernels are written other in other languages that have completely different packaging systems and so it wouldn't make sense to pip install an r kernel r has its own packaging system and the r kernel is shipped using that packaging system so what are some of the common kernels that are out there yeah so the the julia kernel is quite popular uh julia was actually the the second language to have support for jupiter other than python so that kernel has been around a long time it's fairly mature and the the core julia team has sort of been using and promoting jupiter for quite a long time and uh other popular kernels the r kernel uh there's an open source r kernel that is uh people are definitely using other sort of broad areas that we're seeing is a movement towards uh jvm based languages that a lot of people are interested in for tools like spark and there's a number of scala kernels and then also actually javascript kernels there's a couple of different javascript kernels for jupiter and it's javascript is a great language for working in the web and we give users being a web application we offer users a lot of the niceties of being in a web-based environment so you can use libraries such as d3js to do a visualization if you want has a anybody come up with uh or any of the commercial languages supporting julia this would be things like matlab sas uh s plus anybody like that uh so do any of those have jupiter kernels is that more of the question you're asking yeah yes actually uh so i'm pretty sure that idl which is sort of an old-school uh interactive computing environment used a lot in the astronomy community they as far as i know ship a jupiter kernel and then sas as well has a jupiter kernel that they they're shipping there is an open source matlab kernel that is available i've not used that myself i've had some students that have tried it and said it's okay like you can you can use it but it's definitely not uh sort of a first class kernel uh we would love to see uh mathworks take on that and build a really nice robust jupiter kernel and that is something we're hearing from our users that uh a lot of people are still using matlab and want to keep using matlab but they want to integrate with jupiter and get the jupiter notebook format and a lot of the other benefits of the overall ecosystem so for creating these kernels you said they're a separate process what's what's the mechanics how do you get from the web front end to the kernel and back yeah that that's a great question the uh a kernel is defined by the network protocol that it speaks and the the transport layer that we use for kernels is called xeromq it's xeromq is a message oriented layer on top of tcpip that we use and uh the way that kernels talk over xeromq is basically through json messages and we have a formal specification for the types of messages that the front end would send to a kernel and then also for the types of messages that a kernel would send back to the front end and as long as a process uses 0mq in that way and speaks uh sends and receives the right json messages it can be a valid jupiter kernel and it like there's a lot of flexibility within that that exists and but that's sort of the the minimal notion of what a kernel is so then a kernel is it's not even a plug-in it's just a stand-alone entity and as long as it it listens and speaks in the right way and you just tell the jupiter core uh like what tcp address and port it's listening on you're good to go is that correct pretty much yeah we so the kernels are registered with the notebook server by dropping a small json file in one of a couple different configuration directories and that json file essentially has the command line program to run to start that kernel and so it's a very there's no sort of you know language to language calling we literally just you know you tell us what process to start and we will start that process and assume that it speaks the right network protocols there is a way for kernels and the notebook server to agree upon which ports are being used as part of that but it's all a fairly simple uh system so then let me ask my my own bias here being an hpc mpi kind of guy um has anybody written kernels that you know front a back end hpc cluster using you know mpi or some other parallel technology so that you actually have a jupiter notebook uh either launching or controlling or directing some larger computation that's running either a small or large size uh hbc job yes definitely and there's a couple different ways you can architect that one is that there's no constraint over the type of code you run in a kernel and so for example if you're running the c plus interactive kernel and you want to [Music] start to just use mpi in that context you could do that and it should work fine now with that said uh jeff i'm sure you know there could be a lot of subtleties about how mpi processes uh get started and so if you wanted a kernel that really did that well you'd need to think about that sort of bootstrapping phase that mpi does but there are examples of that um one other project that exists within the ipython organization is something we call ipython parallel or ipi parallel and it actually exposes the python api for talking to basically mpi clusters that are separate from the kernel so imagine that you might you know a typical use case we see is someone running a jupiter notebook on the head node of a large supercompute cluster the kernel would be running interactively on that head node and so it's not doing anything computationally demanding but then the user might start a large parallel job with python and mpi and then be able to steer that interactively from that python notebook that so that that is one of the very early use cases that we had in mind for this what about something using one of these new like web stack orchestration engines like kubernetes or rancher or mesos where you could actually say start my big thing over here it listens on this port and i can run jupiter locally is anybody kind of doing that almost like i run jupiter locally and when i'm ready fire up this thing in the cloud or something like that to do the heavy lifting yes definitely that uh the biggest place we're seeing that is in the spark community so the uh there's a pi spark client library and that library knows how to communicate to spark clusters and there's a couple different ways of doing it either the kernel can be started as part of the spark cluster there's a new rest protocol for talking to spark called livy l-i-v-y i think it is but that's definitely one of the usage cases that we see in that's happening a lot in the large companies that are offering sort of turnkey spark deployment is there they're sort of packaging that around the jupiter notebook-based front end uh for their users so uh what's jupiter hub then how is that coming to play with all of this yeah so the original jupiter notebook is a single user web application so it's something that users tended to start just on their local machine so they would type jupiter space notebook at the terminal on their local laptop and that starts uh the jupiter notebook server and then they use the the software through their web application but it's just talking to this local server jupiter hub is a organizational multi-user version of this that basically takes care of spawning single user notebook servers on behalf of different users it handles authentication and then there's a proxy layer that routes the traffic to the appropriate uh single user process so it's a think of jupiter hub as a multi-user organizational implementation of jupiter jupiter is it possible for does jupiter hub understand batch systems or cloud orchestration apis or anything like that so it can kind of spin these things dynamically because i know there's other tools already do this like the tac visualization hub at the texas advanced computing center allows you to submit a job that spawns jupiter for you and reverse proxies it back and we actually support that at michigan 2 on our cluster so people don't have to make their script or anything so does jupiter hub have that built in yeah they're really how i look at jupiter hub is a set of building blocks that you can assemble in different ways for particular types of deployments and for example one of those building blocks handles authentication and it's an extensible api so if you uh you know want to authenticate with oauth you can plug in whatever oauth system you have at that point another building block takes care of spawning individual single user notebook servers that's also extensible and so the the simple default one just starts a local sub process on the server but there's uh people different people have written spawners for different batch systems for kubernetes for uh docker and so on and so that that's something that a lot of work has been put into and i think the largest scale deployments that i know of these days of jupiter hub support many thousands of concurrent users and i think the largest ones right now are using kubernetes to manage the sort of spawning and load balancing and auto scaling of the system now something you mentioned and alluded to earlier in the conversation here was about the efforts you guys have encountered upon for developing a community around jupiter i mean what can you tell us about that for example you just had jupiter khan in in august can you tell us a little bit about that yeah so over the last few years we had started to experiment with different uh events uh to bring together jupiter users and we had had a number of jupiter day events as we were calling them all over uh the world really and we were had started to observe that there were a lot of users and our users have really amazing things to share about how they're using jupiter and part of what's fun about it is the the really diverse ways that people are using jupiter ranging from social sciences and humanities to traditional physical sciences to data journalism and jupiter khan was our first sort of uh larger conference to bring as many jupiter users together as we could uh so yeah this that was just this past uh august about i guess just over a month ago now in new york city and we had around 700 attendees and it was organized co-organized with o'reilly media and also the non-profit organization for jupiter which is the num focus foundation so what's the easiest way for someone to get started with jupiter yeah most of our users probably install jupiter through the anaconda python distribution that's really the easiest way to get a working jupiter installation that includes all the other dependencies that you will likely want to use along with it different visualization libraries scientific computing libraries machine learning and so the the anaconda python distribution is probably the most common way that people get started another increasingly common way is uh organizational deployments where someone within an organization deploys it on behalf of the rest of the organization and at that point you're typically a a user is pointed towards the the url for that deployment and they can log on with whatever credentials are set up for the deployment and what's the strangest thing uh that you've seen jupiter used for yeah the strangest thing um let me think about that a little that's a we usually like to ask this question uh for most of our guests to kind of emphasize the way in which software and even science itself escapes out into the world and then gets used for these sometimes wacky or crazy imaginative ways that uh the authors and developers just didn't intend at all yeah i mean i think i don't know if strange is quite the right word but one usage case that i don't think we had in mind back in the mid 2000s when we got going on this journey was its usage in data journalism um i think like i was ignorant of any work in data journalism happening at that time there may have been again it could be just that i wasn't aware of it and so that the idea that that uh journalism teams would have computational folks involved who are using a tool like jupiter and doing machine learning and data science and data visualization uh is something that we've been extremely happy to see but it's also something that i think has has surprised us uh in in the best possible way of being surprised and so that uh and i honestly i think part of the fun of it is that the organization that has done this most successfully is buzzfeed which is not usually pictured by folks as being a serious news organization but there's a fantastic data journalism team at buzzfeed's buzzfeed news uh jeremy finger vine is one of the folks there we've interacted with a lot and at this point as far as i know any time they publish an article that has data behind it they share their analysis and the data set on github and they're they're publishing that as jupiter notebooks and so they they're really setting a very high bar for openness and reproducibility in data journalism so that actually raises a fascinating question what license do you distribute jupiter on and under and does that carry through to the work that is published by jupiter notebooks yeah we use the three clause revised bsd license and it's a very liberal license and that's a choice that we made very early on we wanted people to be able to use jupiter in pretty much any way they wanted whether it's for non-profit work or academic research or even for building for-profit companies and products around it and so there's no constraints on how people license jupiter notebooks themselves they can license those notebooks essentially using any open source license or not even an open source you could write completely proprietary jupyter notebooks and that's completely fine okay so uh what about the jupiter organization as a whole you said you started as ipython and you kind of made it abstract how are you guys organized yeah so we are now part of the num focus non-profit foundation num focus is a 501 c 3 non-profit that's home to a number of open source projects in the python r and julia communities so a lot of the other open source projects that users are using when they're using the jupyter notebook are also part of gnome focus and we have a fantastic development team working on jupiter our our project is sort of led by a steering council as we call it of 12 individuals that have made long-term significant contributions to the project and then fernando perez continues to be the bdfl for the project um but there's it's a very large and significant effort by a lot of different people um i think we have somewhere on the order of 25 full-time people plus hundreds of other part-time and occasional contributors to the project so it's really a large community effort at this point and many many countless people making contributions to the project and we're really grateful grateful for all the work that everyone's doing so uh thanks very much for your time where can people find out more about jupiter and get started yeah we have a website at jupiter.org and that's probably the best place to start there's links there to installation instructions as well as our documentation the other place that would be great to go to learn more about the project and how it's being used would be to go to the jupiter con youtube channel we have videos of all the keynotes and all the sessions there and i'm not sure all of the sessions are uploaded yet but uh they were in the process of finishing those uploads over the last week and there's many really good uh talks uh that are on youtube for free that anyone can watch and learn more about the project okay brian thank you very much for your time thanks so much for having me on brock and jeff and uh yeah thanks for what you do all right thank you you