Similarities and differences of Spark, Dask, and Ray by Holden Karau

[Music] Holden welcome hey thanks thanks for having me uh I'm Holden this is Professor Timbit he's helping me this morning to make sure don't fall asleep uh hello Professor Timber hello Professor can I call him professor yeah yeah he goes by Professor um okay his his research area is you know is he's not he's not ready to talk about it just yet he doesn't want to get scooped um but he has some very exciting papers coming out I'm sure okay looking forward to listening to you Holden one thing uh just remember remind our audience that you can start sending questions for holding as as now as as as from now because otherwise we won't have time apologies for starting a bit on with delay Holden so any questions for Holden please uh where did she get those fantastic um bed sheets is also allowed so all yours Holden thank you thanks so yeah I'm going to talk about some of the similarities and differences between spark dask and Ray um and in doing that they're they're all distributed system so we're going to talk a little bit about some of the principles of distributed systems um so yeah my pronouns are she or her um that intro was actually already amazing uh only thing that I I want to add to that is I also do these code review live streams um and I also do some live programming now and uh live streamed writing of techbooks um and so if you're interested in in those things you you know definitely check out my my YouTube um there's there's a bunch of streams there um I'm also trans queer Canadian I in America um I got my green card uh this year which is very exciting it means it's harder for them to get rid of me um and also part of the Lea community and that's not directly related to these things but I think it is important for those of us who are building uh data or ml tools especially those of us working uh in open source or in large companies which can have a really large impact in the world to look around at our teams and you know if everyone's from the same background as us uh it's time to try and expand the the pool of people that we're working with and part of that is talking about where we're all from as well our background so I'm hoping you're all nice people uh you are probably interested in distributed systems if you're here and if not that's that's okay too you know I'll I'll try and have some pictures of Timbit to distract you um if this isn't your cup of tea so I'm going to talk about distributed systems I'm going to talk about data parallel distributed systems and we're going to look at these three systems we're going to talk about how they're different um and then we're also going to talk about some common parts some of the parts where they're very similar and we're also going to talk about some of the mistakes that we've made in building these systems uh over time so for those of you who aren't familiar with the distributed systems your life is probably much happier um there's this wonderful quote from Leslie lampor uh a distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable um and I think as as someone who works on on spark you know we we do a lot of work to try and make it so that that's not the case but at the end of the day there are still times when the failure maybe not of a computer anymore but at the very least the failure of a rack could very easily uh cause our our system to become unusable um so why do people use to serated systems scale um generally speaking if your data fits in memory on a computer that's a lot better it's it's much less work uh you can solve problems by throwing money at it relatedly the follow-up one is it turns out buying a lot of memory in a single computer um like yeah you can buy huge huge amounts of memory in a single computer but then computer starts to get really really expensive um and the last one is a bit of a joke but not completely um distributed systems also make traditionally simple problems really challenging to solve um and I know for me that that's part of the appeal and from a business point of view like this is terrible um but from a like engineering point of view it means that all of these problems that are kind of boring become interesting again um and I think I think that's neat so what are what are the core building blocks that all of these different distributed systems uh build on top of so there's distributed locks distributed clocks distributed counters um and pretty much you know every sort of fundamental computer science thing a distributed version of X for all in X but locks clocks and counters are are sort of some of the key building blocks that they all depend on and while I'm just talking about data parallel systems today um like dask uh Ray and Spark it's important to know that there are other kinds of distributed system problems file storage systems if you've used hdfs or S3 um those things tend to be distributed systems minio can be uh can be not um if you've run folding at home or the distributed. net rc5 challenge um those things tend to be embarrassingly parallel uh with minimal or no coordination between the nodes those problems are really fun um and they're really nice because they don't involve a lot of communication between computers and communication between computers just like with humans is where things break um databases not all databases are distributed but Cassandra is a good example of a distributed database API servers are also you know distributed systems frequently these days we we tend to have multiple API mpoints and put them behind a load balancer um and for the most part data parallel systems which we show where I work let us ignore a whole bunch of problems that we have to deal with in all of these other ones right file storage systems super super painful to write um but data parallel systems not not as bad so to a degree um in data parallel systems we get to ignore time and ordering of events and this is pretty awesome um the notable exception is when people insist on making streaming systems which is unfortunately increasingly popular uh to a degree we get to avoid Network partitions um not because they don't happen just because we dedicate the the winning partition is whichever partition happens to have the head note on and that's very very easy uh multiple clients generally speaking we don't allow multiple clients so it's a lot easier when you have a single client uh leader elections we generally a statically assign a leader so there's no election um and we tend to ignore this failure of a key Noe thing and this lets us get away with all kinds of problems um because we can essentially just take a distributed system problem and say you know what we're just going to solve it on just one computer we're GNA make that computer responsible for it um and you know that's really cool um the downside is of course if that node fails everything breaks but you know it's it's not too bad we get to skip a whole bunch of problems but there's there's some downsides to this um we'll we'll get back to that so what's what's left when we when we skip all of those problems dividing and coordinating the work reliability on machine failure besides the key Noe um and the times we allow state so while we get to ignore state to a large degree um training machine learning models uh tends to involve like building up a bunch of State you're you're building up this uh collection of parameters that represent your model um transactions sort of matter and this comes from even without streaming we tend to need to do things like speculative execution um and the last one the last one is the really important one uh bottlenecks on the reliable Noe right so once we've designated this keynote the problem is Engineers are lazy and we tend to put a lot of things on that one key Noe but then it turns out that this starts to get really slow and all kinds of sad so how hard can dividing work be um um so if you if you're an IC you might be like you know what that doesn't seem like that much work but if you have a manager or a PM you can go and ask them how hard it is to divide the work of your team and they might have some opinions um but even in Computer World things are really difficult right uh key skew um falls into this problem of dividing work because frequently we try and Partition by keys and so the key skew here uh really gets us non-uniform processing time so stragglers um which if you if you've been using spark you're probably well aware of and pretty much any variant of trying to coordinate uh and split up work is actually really hard you know it sounds really easy until you try and do it and then life just gets all kind of painful then there's the fault tolerance like how are we going to handle losing a node um and different people have different approaches um so Hadoop map reduce um solves this reliability Problem by just saying you know what none of my workers are reliable I'm just going to save the data out to the Hadoop file system and that's going to replicate it across a bunch of computers and then it doesn't matter if my computer fails because there's a replication of it somewhere else and I can just go and read it from there uh recompute on failure is the approach taken by spark uh and dask and to a limited degree Ray and we'll we'll talk about that more later um but it requires that we keep track of how to recompute data um and it also really breaks down when we're updating State because when we go to recompute the data we might update the state more than once um and also it gets really annoying if our failures become correlated uh historically recompute on failure was really good because failures of computers they weren't not like dependent they they were semi-independent I would say uh but with more and more people moving to the cloud they've become a lot more correlated as people run on things like spot instances or preemptable instances um the other one you know math and extra computers like that's that's the paxos approach um and this is really hard it tends to be the most reliable approach and we tend to not do it very much because it's also really slow and the last one is ignore it um you would be Sur surprised how often that's the approach that people implicitly end up choosing um and we'll we'll talk about that a little more um actually I'll talk about it now so and Ray uh in its in its early versions actually took the ignor it approach um to failure for anything involving State um and that's not great that means that if you're if your actor which is how Ray represents state was scheduled on a node that failed it would just fail and your application would just stop working and you are responsible for managing that and recovering from that and in the newer versions of Ray they've added um framework to to use actually it it doesn't have a strong opinion on which one of these techniques it it uses to recover from failure it's up to you to pick which one but they have a framework that that lets you implement the recovery logic a little bit less painfully so why do we have to care about State um so even if we're doing stateless Transformations there is some State like how far have we come along which records have reprocessed um generally at the end of the day right like as much as people love functional programming we want to do something with our data we want to write it out and that's kind of kind of state um we can think of this as has Timbit had a bath this month I think he has um but keeping track of that is you know that state the state of Timbit has as we had a bath once we add state things start to go to hell um using specialized systems is often how we deal with it um in spark we mostly deal with it by shoving all of our state onto that one reliable node but at the cost of just being really really slow um so there's some options here to handle that that failure of the key Noe um and generally speaking I would say I don't see this done very successfully most of the time uh normally what people do is they just restart the job on the failure of the keyn Noe um normally people use like zookeeper um to keep track of of everything and that's the spark High availability mode uh but restarting the jobs is is non-trivial and so this is like the magic handwave but it's about as easy as convincing Professor timit to shake your paw without any treats sorry shake his paw with your hand um so what about bottlenecks um so spark and dask both fall into this situation of having a central scheduler and that's really great in that it lets us make all kinds of smart decisions because all of the scheduling logic is happening in one place and we can do all kinds of things like caching and stuff like that inside of our scheduler uh the downside is that if we have thousands and thousands of nodes uh and we're trying to schedule so many tasks that scheduler can get get overwhelmed um the other one is in spark we put all of the state on that one node and so that that one Noe is just very very busy um and that's that's not great right and the distribut system you really don't want one node to be busier than the rest that's that's the sign that you aren't doing a good job of splitting up your work um and then the transactions one uh so this one's important for speculative execution even if we're just considering uh traditional data parallel systems uh like non-streaming um this is because we generally have multiple writers but we have one committer uh who is responsible for deciding like hey am I done processing this move it like Market is done so that the next job can know that this data is done and ready although it turns out that the approach that we we take uh which came from hdfs uh which involves renaming files to indicate that everything was ready uh not all file systems support Atomic renames and uh it's really important that these operations be Atomic otherwise you know you don't really have transactions if it's not Atomic uh you can get like these partial views and that's that's really bad um and the solution to this is to put another system on top of the non-atomic system that then gives you an atomic view over top of it which is the approach taken was s3a uh it's kind of weird but it's okay so we've talked a lot about sort of the core building blocks um of these systems and a little bit about their differences but what are what are some more of the differences I think a really important one is the apis Expos um and this is really important spark just exposes highlevel apis um it really doesn't let you schedule raw tasks it's very much focused on data parallel systems only um another one is this unit of scheduling work and this task overhead right so essentially we can think of this as like your manager talking to you um if it takes them five minutes to tell you about a task versus it takes them 10 seconds to tell you about a task um they're probably going to be comfortable delegating different things to you um and so that that task scheduling overhead applies to computers as well and that approach to node loss right we could think about like how we handle it when our coworker quits um and similarly in spark it's and askk and right it's how do we handle it when one of my computers die um and another one that's really important that I think we often Overlook uh because we're technologists is what is the around these tools like um so more concretely uh Ray has probably the best approach to the distributed state of of these three um strangely enough it doesn't support the standard example that we're all used to of word count um because it doesn't it doesn't have Shuffle of course there's an asterisk there you can make word count work but it's just really really painful um you you normally end up running dask on top of Ray at that point uh and it's built in C++ um and it has Python and Java apis and by default Ray is less tolerant less fault tolerant that is um and that's that's okay uh you can change these configurations to make Ray behave more like Das or spark from regards to fault tolerance handling the state is more than just a configuration change though so you'll you'll have to write some code to handle your actor recovery um dask is notable for having really kickass pandas apis it probably has some of the best python Integrations out there and it also has these really wonderful low-level python apis Ray also has lowlevel apis but they tend to be implemented in C++ and that's that's great for performance but not as great for getting people to use them um because they can be a little bit more complicated for people to figure out what's going on um spark is sort of the one that we're all used to I would say or at least it's the one that I'm most used to it has really only highlevel apis um and that's that's not a bad thing right these high level apis mean that spark is able to take a much more aggressive approach to uh fall tolerance uh it's able to do a lot of really cool things with optimization but it does mean that you can't schedule raw tasks in the same way it's built in Java and it has apis for I would say probably the most languages um python is built in R is built in but then there's also a whole bunch of apis for different languages that come from the broader Community uh like C um it does have a new pandas like API it's not as feature complete as dasks um it does also have more overhead uh Ray probably has the lowest per task overhead dask is in the middle and Spark is at the high end and what that means is like in spark for it to make sense for us to be using spark we need to be able to split up our work into sort of like moderate size chunks uh and in Ray we could use much smaller size chunks and in Ray somewhere in between um and the last one is of course the Hadoop ecosystem um if you are working at a place which has you know a big data stack spark is a much better job of integrating with the rest of the tools like Impala um and things like that in the Hado system um your catalog all those things that Ry and ask you know they can they can talk to Hive of course but they don't understand the hi catalog in the same way that spark does okay and we are still running a little over time so I'm very sorry about that um but one of the things I want to talk about and this is because it's it's come up recently is there's there's a lot of conversation around benchmarks um and this is because well one of the vendors um is let's just say they're they're trying to illustrate that there's they're still relevant uh by benchmarks um and I think benchmarks definitely have their place right um I think it's very very reasonable to do benchmarking on the other hand I think that dask and Ray and Spark all perform pretty well at the mediumsized scale of data um and if you're at the like petabyte plus scale of data I think it's really important to not just take one of the like industry benchmarks like tpcs um you should probably make your own benchmarks uh that are related to your use case um because like tpcs is a lovely synthetic Benchmark but it may not represent very well what it is that you're trying to do um and I think I think really for most of us probably the thing to do is to pick the one which is best suited to our domain and our team right like if you got a mixture of java and Scala and python programmers you know spark looks pretty appealing because they can all work together um on the other hand if you've got some amazing kickass data scientists who just like came here to chew bubble gum and use pandas and they're all out of bubble gum uh you know dask has probably the best distributed pandas API of any of your options right um of course you know if you want to look at benchmarks that's cool I'm not going to go into benchmarks because I think that um well pretty much you can always make a benchmark say what you want to say and it's just it's not worth it um to me so we're we're goingon to skip this but for preemptive so on that note um I am three minutes over I'm very sorry about that I am working on some new books um namely scaling python with Ray in Python with dask and distributed computing for kids that one's in spark um and if you're interested in being an early reader or your kids are interested in being an early reader for any of those books please DM me on Twitter it's just my name Holden Caro or email me it's just holden. car@ gmail.com and let me know that you're interested in seeing uh early drafts of this stuff and I would love to share it with you and uh get your feedback another one is I really like all of these things are open source I think uh the community is of course a little bit different um I've contributed to to all of these projects um and if you're interested in getting involved with any of the projects definitely please feel free to reach out or just try getting involved you know I think they're they're great projects and I think that one of the ways that we can make sure that our voices are heard is by contributing to our our open source tools that we're using um and I'll be doing more open source live streams if anyone you know wants to come and and watch and get an idea of what it's like to contribute to these projects in the open source space um so I'm hoping that we might have enough time for for a question I know I know we are five minutes over though uh so feel free to shoot me an email with your questions um and I will I will do my best to answer them okay thank you thank you so much Holdem uh that was fantastic uh first of all what does uh Mr Professor Timber uh think about all this where is he so Professor Timber he's he's got a mixed view I would say um he does really like uh the dask people the most I would say um I think that's mostly because they talk to him when we're talking on video together um the other people don't talk to Professor Timbit as much he's he's very engaged in in the research of course okay well there's a question for him just one so just one for him and we don't have time for more but if you could answer on his behalf because we don't see him so uh oh yeah he went back to sleep it's it's a very busy day transmit it to him we say this is clearly not a case of one size fit all so in order to choose the right framework which you obviously explain a few of the difference somebody says is the option having a data science INF infrastructure flexible enough to allow for a mix and match mix and match approach yeah so I I think what professor tempit would say here is that that's definitely an option um I think with with kubernetes um it's it's quite possible to have a mix and match approach and I think it's it's very solid uh the downside is um it's it's a little bit more painful to to maintain from a systems point of view right um if you can convince people to pick two of the three uh your life will probably be a bit easier um than trying to support all three of them um okay yeah all right in case of Doubt uh people can email you you said the best way to contact is through Twitter DM or to uh or through your well obviously watch your um your YouTube uh streams but in order to ask any questions about which frame to to use or which two to use in case of only we can only choose two to DM you on Twitter or um if we have any questions or your new books coming out uh yes that that true any books coming out we email you uh well congratulations on that green card by the way excellent thank you m professor timberos as well does he go in the package with you yes he's he's included ined that's great he's a lucky he's a lucky man or he's a man or a girl or boy or Professor he's a lucky Professor yes yes holding thank you so much for your talk uh I'm sure people will contact you directly we hope to see you again you know you're very much loved in the big things conference so uh I really appreciate it and and thank you thank you for the well wishes the the one year that I got in the in the motorcycle crash it was it was very very kind you're are very very much loved Holden so we hope to see you on Professor Timber next year if not before that so in the meantime we DM you for whatever we may need lots of love and uh see you very soon thank you so much see you see [Music]