Building a logs engine faster and cheaper than ClickHouse

Hey. Yeah, thanks for the intro. Um, uh, thanks for staying up so late in the afternoon. Um, as is already mentioned in this talk, I'm going over, uh, how we built a logs engine using uh, S3 and Lambda. Quick background, in 2015, when I was still in college, uh, I just walked across the street and joined a company that looked cool. Uh, we were around 30 people at that time. uh rubric. Then in 2014 wanted to relive that experience. So joined UDL as a founding engineer. Um and 2006 I it just occurred to me that last time I did any public speaking was 20 years ago. So thanks so much in advance for putting up with me. Um so before we go into building a logs engine and its architecture we need to also think about why do we need a new architecture right and if I were to sum up that answer in one picture this would be that picture which is u the current state-of-the-art or current most prevalent most well- reggarded logs architecture is basically uh elastic search the elk stack back and this is what the numbers look like. Also mind you that this is not apples to apples comparison because for elastic search what you are seeing is the cost for UD what you are seeing is a price and price is basically cost plus margins and since my bosses told me to not talk about margins this is the number that you are going to see but in your mind you can adjust the number downwards. Then the next question is wait don't I read it everywhere on Twitter on hacker news everywhere that click house is the new brand new architecture columner storage so on and so forth we thought so too but the one one sentence answer is not no enough and these are the numbers we ran the numbers this is how it looks like for one specific query that we benchmarked click house takes 17 seconds what we built takes 2 seconds and the 17-second query also cost 4.5 cents. Uh we serve the same query in under 3 cents. Um and these are the numbers. [snorts] What we are going to go over in this talk is what exactly powers these numbers. How we get from those red numbers to those blue numbers. Again taking a step back at a fundamental level logs workload is two different kind of workloads. Lots of data coming in like a torrent of data coming in but it's pretty stable high volume but stable but when it comes to querying sometimes queries appear when they appear they appear in bursts and they consume a lot of uh resources. So queries are spiky, queries are bursty. And now when we throw agents in that it becomes even more interesting because agents are producing code now. They are producing more and more code. Now more code produces more logs. Also agents are just not producing codes. They're also debugging your code. And for agents to debug code effectively, they need more comprehensive logs. So it's not just that more logs are being produced, it's also that more logs are needed. And when agents debug, they're also not as prudent as humans many times. So the spikes are taller, the burst, they're wider. So both of these problems that we just saw, think of the same problems, think of it worse. That's what we want to solve for when you think about. So now let's get into the architecture of a logs engine. Right? There are six properties that we think are very important for a logs engine to be performant for the users to like the logs engine and for the company that runs the engine to be profitable. Those six are it needs to support high volume injection. It needs to support the separation of injection and query. It needs to have for querying it needs to have low latency. Querying infrastructure needs to scale up rapidly. It needs to scale down when it is not in use and especially for multi-tenant systems which is what we are building. Queries should be protected from noisy neighbors given. So we are set out to build a system that satisfies all these six and more if we want. So first prototype, right? The new kid on the block or the popular kid on the block is Clickhouse. So since we are prototyping, we'll start we we did start with Clickhouse and we started with Clickhouse Cloud because it was a hosted offering. So we you don't need to manage the clusters, babysit the cluster. It supported the S3 as a underlying storage and so many of our competitors are actually built around Click House. So that seemed like definitely prototype worthy at the very least. How does click house perform on high volume injection? The biggest problem that we saw with click house is right amplification. If you want to serve the queries efficiently and under the time budget that you have, you need to have clever not super clever but somewhat clever things like materialized views. But those materialized views come at the cost of right amplification at injection time. So when vector is sending those logs to click house, vector is not hearing back till click house says hey I updated the materialized views as well. Then separation of concerns we could not solve it. We just ended up using two clusters for our prototype and for low latency. Now we come to the part where the high volume injection was being impacted. You need to have materialized views so that you can do various kind of pruning. You can prune the time range. You can prune uh you can short circuit some of the queries because you are maintaining some aggregates in materialized columns. You and since it's a logs product, you definitely have to have pre-ext queries that return fast. That basically means you can't really avoid a full text search index. And even after doing all this, you do need a beefy cluster. And how beefy? You're going to see it in uh second from the last slide. So stay tuned. Rapid scaleup single answer is no. 5 minutes is not rapid. It is scale up. It's not rapid. So we you need to provision your click house query cluster for your peak and no one is good at predicting the peaks. So that basically means you provision whatever and then queries fail and then you feel sad. Scaling down to zero it is theoretically possible but once it scales down to zero it doesn't come up. That basically means again you can't use it for this use case. Protecting noisy neighbors is slightly partially solvable using some things like hey for this query don't assign these many threads maintain some thread budget but then thread budget is capped by your CPU budget. So again, same problem. Few noisy neighbors can be protected against, but that's not really protecting against noisy neighbors. I was working on this slide and uh my son, he's 2 years old. He's like barural. He speaks Marati, right? He is basically so much red because that's what he saw on the screen. And it's not just my 2-year-old son because when we were prototyping Click House, this is what we thought. So much red. Like we want all of this to be green. We don't have any of it to be green. So next question is not click. What do we do? Do we build our own? So few things that you need to evaluate when you decide that okay let's build something on your own is like do you have the right primitives? We believed that we had object storage and serverless compute. Why did we have it? Because we had already built a matrix engine from ground up using these. And we know that and we know that they were uh cost efficient, cost effective and efficient. So we had the expertise, we had the right primitives. But is it going to produce the sensible numbers? Well, you already saw the slide number one. So it does produce sensible numbers. But to know it ahead of time basically prototype fast benchmark quickly and see whether it works. Now we are past that we know that it works. This is what the highle design looks like on the injection side. All the logs are coming in through vector vector pods on kubernetes that are horizontally scalable using kada and those vector pods are directly writing park files to s3 and those fi those paret files are asynchronously compacted indexed by um a by a compactor that we have written. And when it comes to querying, every query spawns a lambda uh serverless function which first queries the metadata based on the metadata does some pruning and decides okay these are the files from which I actually need to query the data. Here for each of the file or for a set of files I am going to spawn serverless functions which will query the actual data. Each serverless function uh goes over a bounded number of files and once it has results it replies back to coordinator which will aggregate which aggregates and returns the results. So at a high level this is the entire architecture and the devil of course lies in the details. So one of those devils that we are going to take a little bit closer look is what's on S3. So on S3 as I said it's park files everywhere. All the data resides in park files. Also metadata about those park files resides in a separate park file which has entries per data file. In addition the metadata also stores um aggregations for few um low cardinality columns and we also store a full text search index. Kant TV is a rest library that we use which produces kind of loose incompatible indexes and then we put it in a puffin container because we want to optimize the S3 puts we don't want to put too many small objects on S3 so we put it in a puffin container it becomes one object um and that's what we put on S3 coming back to so now we have stored the data right how do we query the data When a user sends a query, they are actually using the open search interface. So everyone who knows how to use Kibbana or how to use open search, they see exactly the same UI in the product and the query open search query comes to UD which we translate to SQL and then that SQL is executed using uh data fusion again uh in Rust. in the lambda functions uh to get the data from the park files. So this engine this architecture how does it perform on uh the six criteria that we think are important. Remember the right amplification problem here. There is no right amplification because the vector vector is directly putting the files in S3 and as soon as the files are in S3 vector is good to go. The compaction the indexing which was previously synchronous now is async. Um, I love this slide because just like in math, right? A lot of answers you can just say by definition. I like the slides where the answers are by design. And this is one of those like how do you separate injection and query by design because nothing in the infrastructure is shared except for S3. The injection path is completely independent of the query path. for low latency. Again, this is where uh some of the interesting stuff happens. First thing is the metadata file. Some of the queries, especially uh statistics queries, aggregation queries for visualizations, they can be shortcircuited because a lot of aggregation data can a lot of aggregation queries can be answered just by reading the metadata file. Second, when the coordinator needs to decide which park files to actually query, um it uses the per file statistics and uh uh metadata about specific files to decide which files to actually query. There again quite a bit of pruning happens. Second, another uh avenue where pruning happens is the full text search index. like if the index says uh you don't need to query something then you don't need to query that thing. Um and the final and most important thing is like literally tens of thousands of lambda function you don't really need it. I'm just saying we haven't found a need for it but thousands of lambda functions you can fire concurrently and uh as soon as as long as each of them are going to finish in your time budget you are good. Uh rapid scale up again. Sorry but I forgot a bullet again. Here this is rapid scaleup is by design because you can spawn lambda function just like that. There is no 5 minute waiting time. There is no scale up time. You just spawn a lambda function as as soon as as long as each one of them is finishing within your time budget. Again you're good. Uh what about scaling down to zero? Do you want to guess the first bullet point on this slide? It's by design. When there are no queries, there are no lambdas. So, it's just uh scale down to zero is by design given to us by the benevolent people at AWS and uh noisy neighbor protection. Huh, now you I think you're tired, but I am going to test your patience. It's by design. Each query spins up its own lambdas. Each query is spinning up a coordinator lambda. Each coordinator lambda is spinning up its worker lambdas. So no one lambda doesn't even know about the existence of other lambdas. And what is the resource? What are the resources that they are actually sharing? Effectively nothing except for the s except for S3. Uh so practically speaking no contention. So, noisy neighbors like turn on your uh noise. No need for your noise cancellation earphones. So, yeah, it's a bunch of green, but uh the primary brand color is blue. So, it's a bunch of blue. Uh here is the numbers. To recap them again, one query that we ran took 17 seconds on Clickhouse costed us 4 1/2 cents. Same query in our engine runs under two seconds, runs in under two seconds, costs under 3 cents. And I think the curious among you or the skeptics among you will be like, wait, what do these numbers even mean? Like what is the configuration? What are the queries? Like are you just looking up numbers? So that's why here it's a quick look at the configuration. We uh at the time of benchmarking 25.12 was the latest uh version of click house that was available on click house cloud. Uh the region pretty much doesn't matter. It's the same numbers everywhere. And look at remember the beef beefy cluster that I talked about. Look at it. It has 59 CPUs. That's the biggest cluster you can have in uh click house cloud scale tier. And what kind of we ran uh two two representative queries to benchmark whatever we were whatever we were prototyping and whatever we're prototyping with click house. One was uh keyword query. So like lot of logs will have words like error. So the user is just entering error and they want to see all the error logs. So that's one kind of query. And another kind of query is you got a UID and then you're searching for that UU ID in the logs. So needle in the haststack and keyword based queries. And for uh what kind of workload were we testing it against? Uh not our largest customer but one of the customers I can publicly talk about uh sends us around 25 uh 24 u terabytes uh 24 terabytes of data per day. We tested one/3 we tested at around uh one/ird of their scale. So u 8 terabytes of data per day. We queried for around 3 hours of data. It has around uh slightly less than a billion uh uh log lines comes around to 1 tab. And those were the numbers and uh these are just two numbers right? If you want to look at all the numbers, the methodology and so on, um, scan away and you'll get a full um, detailed benchmark at this link. And these are just the references for the few things that we use. Uh Apache data fusion, tant full text search indexes and uh putting it in a puffin container and uh basically we are done with the talk because I want to leave quite a bit of time for questions. I also have bonus slides. If there are no questions, we are going to do a quick poll and we'll go over one of our bonus slides. But I'll give here a pause for any questions that you have. So in the choose your own adventure style few of the details that I deliberately left out from the architecture diagrams. One is the um one is the compaction and uh indexing part of it and the second one is uh the metadata part of it and uh so compaction and indexing metadata. If you want uh if you would like me to dive deeper into either of those, now is the time to shout or I can do both. Okay. Yeah. So, few more details on the metadata, right? Uh as I told you, Tantv is basically like a Rust implement. Think of it as a Rust implementation of Lucine compatible indexes. for this talk. It's uh good enough to think of it that way. And uh so what exactly are we indexing like it's not uh in open search each document is basically uh each log is basically a document. Here a row group in the park file is a docu using we use a row group in a park file to construct a document and that's what tantive is indexing. Also we have hourly indexes not that anyone asked but we do and um in the metadata file some of the as as I told you some of the short circuiting like statis stats queries that can be answered without querying the data files for every column in the logs which is low cardality column think of it as cluster uh deployment it's going to be dev canary blah blah blah some five six names right that's low cardality so for low cardinality columns each low cardality column becomes a dimension in the aggregation cube and that cube resides in the metadata file and in the metadata file other details are of course per file details like trivial details like what is the latest event in this file what is the time stamp of the earliest event in this file the size of the file the size of the park a footer um and I can get into details on why we need to even store this in metadata why can't we just query it and other bonus slide that I have is uh compaction so main goal like a lot of times main goal of compaction is like you have too many files small files you want one bigger file because whatever specific uh data access pattern that you want to serve. But here for compaction our main goal is to given any time stamp our main goal is to minimize the number of files that contains this time stamp. So imagine there are 200 MB files. Both of them are spanning 12:00 a.m. to 1:00 a.m. comp. After compression, you're still going to have two files, but now they're going to be 12:30 12:00 a.m. to 12:30 a.m. and 12:30 a.m. to 1:00 a.m. So now when a query comes, like one of those files, you don't even need to uh query anymore. Uh and yeah, basically the where does the size cap come from? Our park files are size capped. Compaction makes sure that like they don't exceed like soft limit. they don't exceed a certain size because remember one of the things that we want to be really careful with lambdas is there is already some cold start price that you are paying. So when you have a subsecond query budget you need to subtract that cold start and whatever like remaining 600 millconds or 700 millconds that you have you need to be able to back calculate the size of the file that can be queried using that time. So uh that's why the files are size capped and goal of the compaction is not reduce the number not necessarily reduce the number of files but reduce the number of intersecting files. Uh that's it. Those were the maybe it answered some questions already. So, and we are also at time.