Interactive Spark at Your Fingertips: Integrating SparkConnect into Kubeflow Noteboo... Vikas Saxena

Hey guys, welcome to my talk on interactive Spark at your fingertips. Now, the talk as the title mentions is about how to run Spark work workload interactively on Kubeflow, primarily in Kubeflow notebooks. And both Kubeflow notebooks and Apache Spark, they've been around for long, so they need no introduction. But what probably needs introduction is me. So, about me, who I am. I'm one of those data scientists who used to make who who used to have his life building models, but unfortunately not all of them made to production, and not because the models were crap, but because the engineering around them was never there. >> [snorts] >> And over the years, I moved over to the other side, and now I build platforms so that it doesn't happens to people like me, or doesn't happens to others. Wherever I Wherever the tooling is missing, I try to contribute to it through community through open source contributions. Kubeflow is one of them. I also maintain a Spark Scala archetype which gives you a template of a Spark project to get started with your with your Scala Scala based coding for Spark. And my company Brex AI also provides ML consulting in Australia and New Zealand area. So, moving ahead, why interactive Spark? Many of you who come from data science background would probably already be familiar with exploratory data analysis or EDA for short, which primarily requires interactive interactive connection to data. And in case where you have actually gotten hundreds of GBs or probably 8 terabytes of data, that's where things like Pandas or R, they don't actually are not very helpful and you need something scalable. And because you are constantly digging into data, trying to find out the, uh, what actual what information or what valuable insights data uh, contains and that's where you need interactivity. >> [snorts] >> So, this is where as, um, one of the service providers, uh, in the data science domain, we primarily uh, pitch Spark to our users and for us having the ability to run Spark interactively was is pretty much necessary. In the past, what we have done is basically used Jupiter Enterprise Gateway. We primarily Most of our clients are actually based on AWS, so we primarily went with what is actually recommended in, uh, AWS's uh, EKS guide and there is actually a complete section which talks about, uh, integrating, uh, Jupiter Enterprise Gateway using virtual clusters, uh, into your self-managed notebooks where we replace our self-managed notebooks by Qubole notebooks. It works well, but the majority The problem is most of the other clients which we have are in government sector and they are pretty much very concerned about the security and that's where, uh, this solution is very difficult to, uh, work because many of the components over there, especially a, for Scala users, the Tony kernel is not very well maintained. I've seen some customers running PySpark directly through a notebook, but that's kind of running Spark in local mode. The solution doesn't scale. It's basically one container running everything, and that's not good for either. Now, what the approach which we took after stumbling across multiple solutions including Apache Livy, uh which again is not very well well maintained, is basically Spark Connect. And uh Spark Connect uh was introduced as part of Spark 3.x series. Primarily it came around Spark 3.2.x, and it actually gave us gave everyone a new way uh to connect uh to submit their Spark jobs. And in terms of interactive jobs, this is basically the high-level kind of architecture. You have got your Kubeflow notebook pod which kind of acts as a thin client. It could be a a Python-based client, or if you use something like Almond, which is much better maintained, it uh it could act as a kernel, especially if you are using the Scala API. The Spark Connect server can be deployed as uh as a separate service, and that's what we I'll be referring as Spark Connect server um in my uh in my talk. And once you submit a job, it basically what it does is it's actually spawns up the driver, and the executors come in as and when required. If you've got cluster auto scaler in, the system will auto scale based on your workloads. And uh as compared to Jupiter Enterprise Gateway, this is where this solution has uh is more flexible, is you can actually easily control the users can easily control their own uh driver and CPU size based on nodes, especially if you're doing something evil like calling collect uh on a very large data frame. Previously, if with uh Jupiter Enterprise Gateway, because it's kind of deployed by the platform team, so it is uh it is a a ticket to IT service desk, which could probably take hours causing valuable times and productive productivity lost. But what we actually with this approach what we we wish is to try to make users self-sufficient as in like they can start their own Spark Connect server and that's assuming that the underlying platform administration team has given them sufficient privileges. And once they have those privileges, with a Kubeflow SDK with the help of Kubeflow SDK, they could actually be pretty much be able to do that. >> [snorts] >> Now, Spark One important thing before I move to this slide is Spark Connect CRD comes specially out of Spark Connect, so there is no Unlike Jupiter Enterprise Gateway, you don't have to manage and with extra services. It's pretty much something which is already exists in the system and what we're trying to do over here in the session is just connect the two pipes together. Pretty much most of the advantage of this approach I've already explained, so I would just skip to the next slide. So, now how to get the Spark Connect servers? There's two different way. One One is the good old way of using a Spark Connect YAML defining it in a YAML file and using kubectl apply -f and do it. And the other one is through Kubeflow SDK. Um for my demo, I've actually chosen to go with Kubeflow SDK because with Spark Connect resource with a YAML, it again it would be >> [snorts] >> the scenario where someone from the platform team has to deploy it. It is good in some of some of the customers have gone for this choice because in their case they actually don't want basically instead of having one Spark Connect server running in each user profile they want to actually have a common long life session which is basically deployed in the team name space. And everybody and the connectivity in the background is set that all the users are spawning their own notebook in their profile but the I am roles under underlying I am roles allow them to basically make connection to this resource by assuming the role of the team name space and but for for scenarios where a lot of flexibility is needed, users need to scale up and scale down their driver or executor sites. There's another way using Kubeflow SDK. And the functionality kind of existing in Kubeflow SDK but there is a MR pending from my side which in Kubeflow SDK the MR sorry the PR number is 719 which basically just makes the error messages more user-friendly. And in my demo I'm going to use my version of SDK so that the error messages are visible. >> [snorts] >> Now let's actually go directly to the demo. Quickly before we start I've actually got two notebooks running. One for the first one is basically my Python and the Scala. We'll see if we have time to demo both of them. >> [snorts] >> And this is my local version of uh This is my This is my local version of uh Kubeflow notebooks deployed. Apologies, it's actually on a cheap connection because it's all local and it's backed by a client uh by a Kubernetes cluster on uh client uh client. I've actually pushed my a precompiled my version of the my version of Kubeflow SDK. Just don't go with the version number. I just uh picked up a random uh version version 1.0.0. Uh that's not actually the real version, but uh that's what I prefer to use because my chain was not merged before uh uh before I could do my talk. And mostly one of the things uh tricky things over here is although I'm using the default uh Jupyter uh image, but uh there's a problem with the typing extension package and that's why I had to do all this. The version which is uh which uh the Kubeflow SDK needs uh the one with Sentinel, it doesn't uh come uh by default and there was a problem in me uh it probably it all uh it's I was unable to remove it and that one was taking priority on the uh on the path. So, this is the hard way which I've gone to actually do it. Um the my presentation which would probably shared at uh at the end already has the links to all these notebooks. So, you don't have to worry about it. Uh it's already there on GitHub uh uh and I'll uh paste the link to the GitHub repository after this uh uh after the session as well. Now, So, it's a good idea to check whether you're using the correct version. Anything above uh for typing extension above version 4.13 should do and as long as you can see if there is Sentinel, then you're good to go. Typically, this is how you spot uh start Spark But in this case, it will fail because the SDK by default points to the default namespace. So, you actually got to kind of tell it to basically use the correct namespace. And when you actually do that, there is there's some another caveat as well. Uh firstly, the Istio or the service mesh which comes with Kubeflow, it actually blocks a lot of traffic. And and the way to actually bypass that is to basically bypass the Istio sidecar for your pods. And if you're using the Kubeflow community distribution, this is already done for you. But if you're like me where you've installed Kubeflow locally and then you have you have installed Spark operator separately using Helm Helm chart, you're definitely going to run into it. And the other uh the other thing is basically the service account which the SDK doesn't pick up correctly by default. And it by default it goes for the default service account. And that service account doesn't have the right privileges. For my self when I set it up, I created this service account service account. And this is one which is which we're going to use. And another thing to note, if you're interested in knowing what how do you actually scale up and scale down your driver pods, basically is through this section where you actually pass these arguments. And you can actually manage resources pretty pretty well as per your need. And going into this, it takes a while to actually come up. And if we look at the terminal, so because I started with three executors, so you can see this is a this is the this is the three executors, and the server which actually kind of act as the driver as well. And anything after this would actually work fine. >> [snorts] >> So, I've just included some basic operations over there, including the favorite the pie. I had to actually modify the pie code a little bit because Spark Connect doesn't it doesn't expose the Spark the Spark context class. So, the code is a little bit modified, but actually does the same thing. >> [snorts] >> And the SDK also actually has some helpful functions that what are the to list your sessions, and if needed to basically delete delete the session as well. >> [snorts] >> And pretty much this comes out of the box. All you need is to install the Kubeflow Spark API and the and PySpark locally. And one of the beauty about this architecture is depending upon which version of PySpark you install, that's basically that's a version of Spark that is going to run that is going to be used to run your code. So, in a way you can easily uh scale up or not scale up, basically change your uh Spark version as well. And because the Kubeflow SDK is actually in Python, so the same code can actually sit in the same notebook. But, if you're using Scala, you you're probably stuck with creating two notebooks and then getting this uh this URL. And over here I've just defined this as a variable which needs to be changed every time. And once you do that, yeah, it took it. It should be because the Spark Connect server actually is in the same namespace. So, the connectivity the connectivity is pretty much simple even if you're using the Scala the Scala API. And speaking of Scala, as I as I mentioned that there is a little bit of work to be done. This is basically the repo which I'm going to share with you, but the only thing is because >> [snorts] >> because Scala connectivity Scala already always needs a always needs a kernel to talk to. So, that's where you actually got to got to install a element. And for that actually I've actually installed cord share which I'm using to basically install install the kernel. >> [snorts] >> And these are basically the default if you're using Java 17, you'll probably end up doing this because by security due to enhanced security, some of the classes are actually not open. So, you have to actually do this. This is basically just some extra Java arcs which you need to pass. And in this image I've actually hardcoded this version because I needed some sort of because I needed Python over there. But again, with Python one you can actually it's pretty much easier to install them on the fly as well. Or you can actually make a create a new image and install all those packages which I was talking about including the QFlow SDK over here as well. And pretty much coming back to the slide deck. So, the what actually while the demo actually looks easier, but there were a lot of our backs related issue. But, and again that's two related issues the last two. You as I mentioned you probably don't need to worry about if you're using the community distribution, but if you are installing Spark operator separately, you'll run into them. And the easier phase to basically for your for your especially for your executors just bypass bypass the Istio sidecar and it should work fine. And do not forget to actually add service account to your driver when you are actually in initializing your Spark client session. One thing which actually I forgot to show in the end was actually deleting it and the and the SDK is a function called uh delete session which actually does that. And in case if you're deploying if you're using the same approach, it's a good the Spark connect section doesn't actually have a TTL or a timeout parameter, so it will exist. So, if you're the platform administrator, you'll probably have to rely on something like this to basically clean any Spark connect servers which are not being used at the end of the day. And by executing by calling the delete session on the the API, this session could actually be deleted and if I go and see my pod, it's connecting terminate currently terminating the the server or the main the driver and after a few seconds it will actually go away. >> [snorts] >> Now, pretty much why the reason I one of the question which I usually get asked is why Scala why I want because it's actually more more actively maintained and over the story and almond are the the only two reliable things which I could find. >> [snorts] >> Now, this is pretty much the end of my sessions. I welcome all the questions along with the review on my PR. So, I'm sorry, I can't see the chat but if you've got any questions, more than happy to take them. >> [snorts] >> Awesome, great. Thank you so much. Because we have a question. Will Kubeflow notebooks support interacting with Spark Connect via chat in the near future? >> Sorry, would they support Kubeflow? >> Yeah, the the Spark Connect via chat. So, will will Kubeflow notebook support interacting with Spark Connect via chat in the near future? I think that's >> Sorry, I didn't get like when when you say chat, can someone whoever asked the question, can you throw light to it? When you say chat, what do you mean by that? Or Sorry, I'm not able to understand the question. That's the whole thing. Is it actually in the Slack channel or somewhere? Can I >> Uh no, it's it's in the event Q&A. I also didn't get the the question fully. So, if you go to Q&A >> Okay. Yeah. Oh, where exactly? Oh, yeah, Q&A tab. Sorry. >> It will be at the top. >> Okay. Sorry. Jupiter and QFlow Oh, sorry Ben. I didn't understand what do you mean by chat, but pretty much if you're talking about interactively the you've got Jupiter cells. You can actually create uh as many as cells and put the code in there and it should work. Uh but I'm not sure what do you mean by chat over there. But pretty much I'm very well convinced that the integration actually within Jupiter notebook they can be made to talk to each other. Uh of course, you have to take care of the R backs as well as the traffic flowing through STO. And Sham, thanks for your comment. I 100% agree with it. And I'll put the link to my repository in a couple of minutes in the chat. If uh Do you actually have couple of more minutes guys? >> Yeah, actually you still have uh 10 minutes. >> Cool. Actually So, one of the things which actually I didn't actually talk, but that's also but that's another option is basically it's again there in the repo, is basically deploying it through a Spark Connects server. And the only what it changes is that the way if you deploy it, the name is going to be basically your Spark Connects service name. And sorry. Which is actually defined over here. And the rest of the URL remains the same. And the connectivity like this would would also actually work. But I find the SDK to be a more user-friendly. And it is and it's a way to basically make your users self-sufficient. So if you don't if you if you as a platform engineer, you don't want too many service specially with data science requesting the change the change in driver size or the executor size, this would be the this is basically I would say the better way to go. But again, both options actually exist for you to connect. >> [snorts] >> I think that's it for pretty much from my side if there are no questions. Uh I'll give you 10 minutes back.