Breaking the trilemma: engineering serverless Spark that runs cheap and stays in your account

Hello all. All right, my topic today is about serverless data platforms. Whenever you have any kind of a data platform, typically you can either you can only get two of the three things, either ease ease, performance, or cost, just like your CAP theorem. So, this is going to be about that. All right, before I go deep into the topic, maybe let me just introduce myself a little bit and then my history with this problem. So, I'm Nilesh. I've been the founding engineer and head of infrastructure at Onehouse. If you don't know about Onehouse, Onehouse is data platform as a service. We work on the lakehouse platform. I have before that for a few years, about five five and a half years, I was at Uber, before that at eBay. So, all these last 10 plus years, I've been working on building something that's really really high scale, managing, operating data platforms. So, yeah, all this comes all this talk comes from my real experience in building data platforms. All right, so in the data SaaS market, typically you get a lot of serverless offerings, and those serverless offering typically have some trade-off. That is basically out of the three things, ease, performance, and cost, you can only get two of them. So, I'll explain that more with an example, but when you say ease, it ease means you have a easier way of running your jobs, you have easier way of managing CICD upgrades, everything. When you say performance, obviously you want all of your resources being utilized properly, auto scale to demand, and like they can give a really high throughput. And the third is cost. That's always the main factor where you want to make sure that you are running it at the cheapest way and whatever discounts you get from your cloud you kind of can claim them. So to give an example let's let's talk about let's say serverless by vendor the first row EMR you have data proc or data brick serverless right? So if you kind of high level evaluate them they are much easy to use but then there is like they are quite expensive and they give you the top notch performance. So they meet the two of the three. If you use EMR data proc like with a BYOC model where they run in your account it's like it meets your cost because you can use your cloud savings. You can get good performance but it's kind of tedious to manage them. If you do your in-house stack then you can have like you can build it in the way that it is easier for you to manage. You can make sure that you are optimizing it for cost but then it's very difficult to get that top performance through any house stack. Same way in the world of like Spark you have warehouse bundle compute also. Again they are very easy to use but then they are neither like cost efficient nor they are like highly performant for lakehouse workloads. So the talk is about that last line that how you can get all three. That's that's what we are going to focus on. So before I go into it I want to make some real data and like I want to make sure that we are not like we are grounded with whatever we are talking. So when I say cost I want to make a case first. Let's say that you run a workload which is daily 1 TB ETL. Which runs on maybe an 11 node cluster with one driver and executor. Let's for the sake of like analysis, let's assume that the speed of Spark is the same. You have two options. You run Databricks serverless or you run Databricks classic Photon which runs in your account. The first row tells you that if you run in a serverless mode for that same one daily 1 TB ETL cost, your monthly cost comes out about $230. Uh you don't get to use your EC2 cost or discounts. You So you pay directly to the vendor. If you do Databricks classic Photon, your DBU cost will come out about $98. On top of that, you will pay the cloud cost of $17. Let's say you have a good cloud optimization. You've got discounts. You can use reserved instances. You can save about $10, 60% of your EC2 cost. The final cost will come around like $105. So for the same job that runs on let's assume Databricks Photon is same like speed-wise, performance-wise in serverless and classic mode, but then the cost differentiation is huge. You You end up paying like almost double the cost in this manner. So where you run your compute makes a huge difference. Whether that engine runs in the customer like in the in the vendor account or in your account, that will determine cost by large. Now, let's talk about performance. So here the context is we are more like talking about Apache Spark. So a lot of these slides content is geared towards Spark, but we are generally talking about all data platforms. Here let's talk about let's say Spark performance. Now, if you have OSS Spark on the TPC-DS 10 TB, it it does about like 12,000 seconds. And then you can see that the These are the performance comparison of the like the four five engines. The top three being Spark, Comet, Gluten, which are the like the open ones. And then Photon and Quanton. Obviously, the lower is the better. So, the Quanton is the fastest engine here. Now, if you run the daily that Let's say you run the same daily 1 TB ETL job on let's say Quanton versus OSS Spark. And Quanton runs in your account. And same way, Spark OSS Spark you run in your account. What happens is if you run Quanton in your account like even after like paying the vendor fee of some $1, your monthly cost comes out about $44. Okay? But if you do it in OSS Spark, even in your own account, even with discounts, everything, you will probably get up to $91. So, the point here we are trying to make is even when you are paying a vendor, uh OSS Spark is quite expensive because it is not as fast as the other engine. So, free doesn't mean cheap. That That's the point we want to make that every time we try to go for something that is free, open, but that end end up being much slower. And then even after paying a vendor cost, it it basically comes out quite expensive compared to uh other fast engine. Now, third point is ease. Like a lot of the uh like managed offerings, like the they solve the problem of ease pretty well really well. But if you have to build something of your own and at that point these like if you operate a data platform with more than like five data planes, let's say, where you have staging production maybe different environments, you have different internal teams and customer, it becomes very complex to run all these by yourself. Your your multiple data planes, multiple cloud accounts, they need peering networking setup for it to be able to talk with each other. You need access control across all these different data sets that you have. Um if you are managing internal platform, you want to have cost attribution. And then the last is like, yeah, you have to maintain it, you have to do patches, upgrades, everything. So, ease is definitely like a factor that we have to look into. All right. So, we talked about the three buckets of problem. We made a case that your running the data platform in your account is cheaper. If you get a faster Spark build, it's actually even more cheaper for you. So, let's see if you can build a serverless data platform in your account all by yourself. So, before I go, I'm going to share some uh like a high-level architecture so that the terminology becomes clear. By the way, this is all based on how we built one out. So, we are sharing all the learnings that we have. We are just telling that hey, using open source, you can all build by yourself. So, our high-level architecture is, let's say, the main thing you care is data planes. You have a lot of like you can have multiple data planes and our jobs, pipelines, monitoring stack running, and they are talking with uh your data sources, catalogs, everything. Now, typically, if you have so many data planes, you want something to manage them. So, there is a shared control plane, which is like the management layer, where you take care of like billing, access, security, everything. And you ideally want your CI/CD, everything to interact with the control plane, and then pass on things to data plane. All right. So, the first problem I want to talk about is networking. You have so many data plane running in different accounts. You have one control plane. You have CI/CD that you want to deploy jobs and everything at with ease. The most important thing is you need good connectivity between all these different uh uh components. How do you build that? So, there is a OSS software called FRP, fast reverse proxy. It is something that we use. It's very reliable, and that's what our experience has been. So, how it works is you can run a server in your control plane, you can run a client in your data plane, and the client talks with server and establishes a tunnel. The tunnel can go through private link, internal network. It can go through like you can put up an MTLS in between to make it even more secure. The point here is in your data plane, if you do this architecture, you are not even opening a single port to the internet. That's what makes it a bit like makes it secure for you to be able to operate them without having to do any kind of uh like security, firewall kind of complications. And then your control your CI/CD and UI will just talk with the FRP server whenever it wants to interact with the any of the data plane. This will make it easy for you in terms of networking stack. Once you have that substrate, everything becomes like all all the new things, all the observability integrations can be built on top of this. Trade off? Yeah, you got to own this. You're You're made it yourself, right? But, it's not difficult to maintain. It's just like few parts. And like I think the hard part would be if you do MTLS, you have to manage the certificates. All right. Second problem is I am security model. Right? You have multiple data planes, control plane, your CI/CD operating something, administrative task. You have your support engineers trying to debug pipelines, access data. You want to make sure that everything is kind of secure. You give the least amount of privileges to everything so that you can scale this architecture pretty well and make your data and everything secure. So, again, this is like a high-level I am model that we built. So, you basically on on control plane, you can have some kind of a bridge roles called admin role and support role. Your CI/CD will use the admin role, your support will use the support role. Through these role, they can connect to a specific data plane. They can assume the role on the data plane with a similar mapping as the admin role, support role. And the last thing is you can have something like a workload role, which is which allows your different clusters to only talk with specific subset, subsystem, buckets, access secrets. So, you have like a well-managed isolation of roles and layering so that all use cases are properly handled. How do you do that? You can write some infrastructure as a code. You just have to templatize it that for every data plane, you will follow a consistent pattern of building roles and security. This is the most tricky part. So, you are talking about running multiple data planes. You also want them multi-tenant. You want the resource utilization to be high. You want a really good isolation between different tenants that are running in your data plane. You probably want to leverage uh spot on demand kind of thing to like optimize the cost. Maybe even you want to do Intel arm kind of a fallback. So, all these things are very difficult. But, there are There's already a lot of open-source tech out there to help you build this. First is If you know a lot of you probably know carpenter. Uh it now comes pre-installed in EKS actually. It's very good, but it it all it does is node management. It can't give you the most optimized compute. It can't give you isolation. It just is node management layer. Now, you have unicorn. I don't know how many of you know something called Apache unicorn. It's a very good resource scheduler. It allows you to build queues on top of your resources. But then, obviously, it's a virtual resource scheduler. It doesn't talk about underlying nodes or resources. So, that's why it's like one layer. The third is Spark operator. If you're running Spark job, the operator is basically If you're familiar, you can submit a job as a YAML, and that makes it easy for you to run a Spark job in any Kubernetes environment. So, if now we'll go a little bit more into like how you can use all three to do a like like a real optimized resource management. So, let's say you are a user. The first step is you submit a Spark job. It goes to It's It's a CRD. It goes to the operator. Operator spins your driver and executor. Behind the scene, what you will do is you can have like a unicorn queue. You can have one queue or you can have multiple queue. You can put driver in a driver queue, executor in a in its own queues. Now, all these pods now when they like then you map the queue to particular carpenter node pool. So, what happens is the driver can get scheduled on an on-demand instance versus the executors can get scheduled on spot instances. So, what did this what this do did is basically give you a nice multi-tenant system. You can have different queues for different customers. There can be a different setup at the carpenter level. This instances with NVMe, non-NVMe, uh EBS, whatever you configure, you can make sure that the allocation is right that. It's a much easier layer to operate if you if you stack them well. To give an example like this is how it is today in one house. We have a UI. So, user can go just say that they want to create a cluster. So, what we do is we created abstraction going to a virtual cluster. They specify the type is Spark. They can specify min-max resources on how many instances they want. They can specify what kind of worker and drivers they want like workers are maybe executors are larger, driver is smaller, whatever depending upon your workload. And then you can have a some setting to say is it on-demand or spot. So, this that's it. This thing goes behind the scene, sets up some queues, some carpenter node pools properly. Now, from there on, the user experience is very simple. They submit a Spark job. The job goes to the the via the CRD and it just runs on the optimized infrastructure. And you will get a full isolation of resources. Uh you can even do priority management through it. So, it runs pretty well. The challenge in implementing this is yeah, we got to own the integration. All of these things you have to build yourself, but they're pretty stable and mature technologies at this point. We've been running them for really long time now. So, I can tell you that yeah, wiring them is hard work, but the the benefits are plenty. The last problem I want to talk about is chargeback. So, obviously you are running a multi-tenant multi-team setup and you want to have to make sure that you know, you have chargeback to the team that is utilizing the resources. So, challenges there is you want obviously something that is trustable. You want something that is like has good observability for analysis and capacity planning as well. And it needs to be quite reliable, right? For you to make decisions on. So, it has to be like a single source of truth. It should reconcile with your cloud bills, right? Because ultimately you want to make sure that everything is correct here. So, how did we do this? So, let me So, we use two standard open source technology pieces. Cube state metrics, which is which exposes a lot of Kubernetes metrics very easily accessible to you. And then there is open telemetry which can collect metrics. So, the implementation is simple. In each data plane, you run an Otel scrapper. It scraps the metrics from like from the jobs, the state, unicorn, everything, and then puts them into some kind of a storage. You can use cloud storage cloud metrics or you can use Prometheus. I will suggest you prefer cloud metrics instead of Prometheus because that's more accurate. And then you use those metrics to consume in your control plane, store them in billing for like long-term DB capacity prediction analysis, all of all of sorts. Trade-off, yeah, you got to you have to build it again, same, but it's it's pretty mature at this point. All right. So, coming back to the trilemma, the like we said, if you build a data platform in your account with the steps that we mentioned, you can achieve all three. And you can have ease of use, a nice networking stack that works with CI/CD, everything. You can have great performance by choosing either you use Comet or Gluten if you want open source, or if you want like you're okay with like a vendor, you go for Quanton, and you get like a most fast Spark in your account, in your settings. And then you can save like you can claim your cloud discounts that you have with committed usage. So, I'll just close this on like a last point that you know, till now all the data platforms were talking about managed offering. But with AI, this thing is changing now. Managed just mean that, you know, I'm managed like some basic layer of management, software patching, everything. But the even after using a managed platform, you still have to do the hard work of like tuning the Spark jobs, right sizing the cluster, capacity planning, handling on-call pages, right? So, the hard work is here, right? That's the operated part. That's the operational work that you have to do. And we believe that now with AI, the value or the the value of the label managed is diminishing. There's there's not much there. The future is going to be more like operated SAS, not a managed SAS, fully operated SAS. And agents are going to play a huge role in this operational work. All right. So, what can you do? Using all this information, you can go build it yourself. It's definitely worth the effort. If you're evaluating some other vendor, then you should ask these questions that where is my computer running? What is vendor's incentive to speed up? Right? It needs to align, right? The skin in the game. That they they need to align so that, you know, both benefit from this. And then third is who benefits from the cloud provider discount. Are you Are you getting your the discount for your workloads? Uh you want more information on this, please check Onehouse. We have We like this is the working example of everything we talked here. Check onehouse.dev. This is specifically about Spark and running Spark and cost analysis and everything. There's a lot of information there on the site. And uh yeah, Onehouse uh lot of the founding team at Onehouse is contributor to Apache Hudi, Apache Xtable. So, we are do check out our open source projects. Thank you. Questions? >> Hey English, thanks for the talk. Now, can you elaborate on what kind of complexities does AI [clears throat] bring to the table? Because you mentioned that there's a control block and data layer, right? Now, uh when it comes to deploying something like agentic and control, it means that I'm literally expecting anything to happen in data. It can access any data. So, there I feel like control block would would be the easiest way to lose control if I start introducing agents there. >> Uh we Okay. Two different parts. Uh help me understand the question. Are you talking about running agents in control plane or data plane? >> Uh in control plane. >> Okay. Uh to do what? What kind of agents are these? >> I I wanted some access some data, some some user-specific data. >> Generally, the the principle that you should follow is data should never leave data plane. Okay? Only the metadata or some sort of orchestration should happen between control and data plane. If you want to build agents that accesses data, put them in the data plane. The And then you can expose them to your internal customers through internal networks only. So, your data is never exposed outside. So, let don't never let your data go outside your data plane. That that's fundamental. >> Got it. Thanks.