Introduction to Floe (Kurt Westerfeld)

Um, and I'm going to give an introduction [snorts] to Flow. Um, as Andy said, we're an offshoot, um, fully funded by Yellowbrick, um, and we're, you know, operating in startup mode and focusing on a different part of the data warehouse and, um, query service, uh, market. So, let's let me let me talk a little bit about what, um, what Yellowbrick has done in the past, so I can relate it to what we're going to try to do for the lakehouse. Um, we built a petabyte scale MPP database, um, focusing on predictable performance, um, a lot of systems engineering. We we first delivered it as a as a hardware appliance with dedicated flash, um, uh, compute, and, um, moved that to cloud, um, when I say around uh, 2020. Um, and have have operated both for for different, uh, kinds of customers, different kinds of workloads, customers doing both at the uh, you know, at the same time, etc. Um, and what we've realized is that as the market has has sort of started to move towards open table formats and open catalogs, that really achieving the the the degree of success we had with the high end of the OLAP, um, space with Yellowbrick is incredibly challenging. And I'm going to talk about a number of those challenges today. Um, we've we've got I, you know, really clear plans on how we're going to solve a lot of these challenges, but some of them we're still, you know, in the process of solving. Um, and I think that's kind of an exciting place to be. Um, certainly building stuff is fun. Um, solving tough problems is is what we we love. Um, and and so what we did was we started to to to take a look at, you know, on this lakehouse architecture, um, we're no longer, um, sort of the owner of, uh, the right path. And we still need to achieve, you know, enterprise scale concurrency predictability, auditable results for financial workloads, something that is maybe non-obvious but related to, you know, getting accurate statistics for planning, um, in in kind of subtle ways. Um, have the same level of ecosystem interoperability, try to do better actually. Um, so we're Postgres-based, um, we haven't changed that. Um, and then and really we know what's happening in the industry, you know, in the last year, especially kind of speeding up in the last 6 months, is we have these agentic workloads. Um, we see what they do to a data warehouse, um, writing, you know, really kind of crap code, um, for SQL at times, sometimes, um, you know, massive joins that, you know, aren't easy to plan. Um, and so we know that what our customers want is they they're moving data to kind of a central place with open table formats and and, um, there's many writers, many readers, but, um, there's a lot of untamed sort of feel to this, right? There's there's a a fair amount of, um, you know, ad hoc aspects to it that make it incredibly hard. And and so this is the the the place that we knew as we succeeded with Yellowbrick with these ad hoc workloads, um, with, you know, humans, that we know that we can kind of apply some of the same, um, technologies to solve this problem. Um, and then and so looking at, you know, what what does it mean a closed system, an open system? Um, obviously databases are open, anybody can connect to them, but, um, when you have a a multiple writer kind of dynamic going on, um, it's it's incredibly challenging in ways that I don't think are very comfortable for database engineers. Um, it's it's really about instead of, uh, you know, optimizing, you know, this particular, uh, you know, part of the database, it's it's really kind of looking at, well, this thing is essentially an integration problem. Um, it it of course has other aspects to it, but if you don't embrace the idea that that integration is key to your success, you won't you won't achieve much. Um, execution engines are are all over the place, um, but query planning on the lakehouse is an incredibly hard problem. Um, it turns out that, you know, the Parquet spec, for example, has mixed up the word should and must, um, in terms of what statistics are written. Um, and so because you don't control who's writing the data, you really can't know, okay, there's a few things you can know, the the number of rows, some basic statistics are are knowable by looking at Parquet files, but in order to do this at the at the scale that we want to achieve, we have to sort of re sort of engaged at the level of statistics that we need to build, so, um, and and and maintain, um, for data skipping. Um, and this really kind of drives back to our our customers' demand of us, you know, predictable cost and performance. And so when we look at a Yellowbrick customer that is moving, you know, they're they're kind of in that that that slow, yeah, we want to get there, uh, mode that many enterprises are. Um, but once they land there, their expectation is what Yellowbrick is already delivering to them. So, we're sort of doing this now as a in in in a different, you know, approach, so that we can sort of meet them when they when they get there in a lot of ways. Um, and and we've we also had, uh, our CTO and our chief product officer, you know, go to the the Iceberg Summit last week. And and what we're what we're hearing is customers are really kind of still struggling right now with the right path. Um, they haven't really even gotten to the point where they're trying to operate operationalize the the data once it lands. Um, so this is sort of an interesting place to be. Um, and, um, so we we're really excited to be doing this with Flow right now. Uh, sorry. Next slide. So, let me just break down how we're doing it. Um, first of all, we decided that the the kind of this integration hub, if you will, is an incredibly important thing. So, we're we we decided, well, the catalog of catalogs pattern, where we embrace many different open table formats, many different cloud providers providing that, um, is sort of front front and center. So, we're we've open-sourced Flowcat. Um, and Flowcat basically is, if you think of what uh, Postgres is today, it's it's kind of like PG catalog on steroids. Um, essentially, um, we'll be, uh, extensible to different personality engines. We talked about that last week. Um, that's been recorded. I don't want to go too much into this. Um, you can actually download and use it today. Um, we'd love to to to get more contributors on it. Um, anybody that wants to, um, surely can, uh, start to use it now. Um, an adjunct to that is something we call Flowscan. Um, this is, uh, part of Flow, uh, an extension to Flowcat to to deal with the indexing problem that I mentioned. Um, this is a not only an indexer but a read planning farm. It turns out that if you're at petabyte scale, the the the indices, the stats, everything about, you know, doing a large scale plan over huge fact tables is itself a data, uh, big data problem. Um, so we're we're we're actually sort of in this this research and and development mode right now building this. Um, but the the the basic gist of this is this component performs the the the table scan pruning for our compute clusters. Um, coming from Yellowbrick tech, um, so those two things are new. Um, Yellowbrick technology, um, we have our query planner. Um, it's it's started with a hard PG fork. Um, we had to add a number of uh, really important optimizations for MPP, uh, along the lines of, uh, data distribution. Um, it uses the statistics and the metadata from Flowcat, um, and it it, um, it also encompasses Flow SQL that that is, um, our query execution pipeline, um, that turns AST, um, parse parse trees into, uh, object code. And and that runs on dedicated tenant isolated compute, um, uses the Yellowbrick executor. Um, and then finally, the thing that, uh, we know will help tame agentic workloads, um, WLM, Flow WLM handles, um, isolation, fairness, prioritization. And, um, this is sort of the primary administrator tuning area where they can tune resource utilization, uh, lots of knobs, um, and ability to to, uh, control their workloads. And, uh, finally, I'm going to show how we're deploying this. Um, each one of these areas is sort of, if you read, uh, left to right, uh, with me, we have a couple different database clients. Um, we, as I mentioned, we're Postgres compatible, so we have a libpq database client entry, but it's it's sibling, um, is also Arrow Flight SQL, um, going to shared, uh, service infrastructure. Um, we're deploying on, uh, you know, a Kubernetes, uh, control plane. Um, uh, we have a a number of different open source components that are used to to not only steer traffic, um, using Istio, but all and and protect, um, components, you know, uh, or control authorization between, um, these microservices, but also, um, isolate, um, compute where the data the data touch point is for, um, uh, the you know, the the external object storage. And that's what is depicted in this this sort of green area. Um, query lifecycle would would land with our terminator Arrow Flight SQL, uh, authentication with a session service that speaks OAuth, um, mTLS, um, uses personal access tokens instead of passwords, all controlled through an account, um, user interface, um, that the customer interacts with. Um, we have tenant affinitized services and we have stateless services. So, this is uh, pretty important. Um, there's areas of the of the query life cycle that are that are sort of pinned and then there's areas that are that are stateless. A query goes through a number of transforms before it lands on our compute cluster and executes there. Results are returned in internally in in in either flat arrow format or row optimized for a row weight oriented client. And And that's that's what I prepared for today. I know I had about 10 minutes and I'm right up against that and would be happy to answer questions, Andy. All right. Round of applause. All right. Any questions? Okay. Yes. I have a non-database related question if that's allowed. I guess. Maybe. Um was there a reason you started out as music theory and then switched to CS? >> [laughter] >> Yeah, that's a great question. I It turns out I'm not a very good musician. >> [laughter] >> I did have a love for it. I you know, in in high school went to to music school part-time. Still a huge huge jazz fan and love live music. But you know, just didn't It also turned out that I I learned I was a really good coder. And and so when I when I realized I would be an average musician but a pretty darn good software engineer, it was a pretty obvious choice.