Floe: A SQL Compute Service for the Data Lakehouse (Kurt Westerfeld + Mark Cusack)

Okay, thanks very much Andy. Hi everyone. So my name is Mark Cusack. With me is my co-presenter Kurt Westerfeld. So I'm going to start off with a little bit of an introduction and maybe a little bit of that so-called breakup talk that Andy mentioned and I'll I'll set the record straight there perhaps as well. But to start with what we're doing new here is is a new um managed service called Flow. It's a SQL compute service for the data lakehouse. Um and you know, rumors of the demise of Yellowbrick are are not true. Yellowbrick is still going strong. We're still selling it. It's still doing very very well at our existing customers. But Flow is a departure, a completely separate business unit from Yellowbrick, but it's funded and wholly owned by Yellowbrick as well. Okay, so the reason we're kind of transitioning some of our IP from Yellowbrick to Flow is because of this new kind of agentic future that we see databases being a a significantly important tool around. So I'm going to start by a little bit of the motivation for really the sort of what our customers have been telling us at Yellowbrick and and the wider market as a whole. And what our customers have been telling us again and again is what actually matters going forwards open table formats. So formats like Iceberg and Delta table formats for example, where we move away from the idea that a an MPP scale out data warehouse like Yellowbrick owns all of the own its own data in its own proprietary columnar file format to a compute service, a SQL compute service that can operate on all of these different new open tables because customers typically going forward want one copy of their data in a data lake. And they want to apply the best tool for the best use case at hand to that open data. They're they're no longer satisfied with copying that data out into something like Yellowbrick doing high performance data warehousing there for example and then doing AI initiatives or something else in some other stack. So that's the kind of new world where we're all kind of adapting to. So it means a a data lake could be surrounded by many different technologies that are writing these open table formats and others that are are querying it and reading it for example as well. So a much more broader ecosystem and with it comes a lot of uh interesting use cases and new ideas but also a lot of challenges and and we'll get into those challenges as we go on. So what I'm going to do now is hand over to Kurt to talk about the Flow service in general and then he's going to hand back to me a bit later on and I'll talk about some of the the metadata services that that Flow needs to succeed. So over to you, Kurt. Yeah, thanks Mark. Um Andy, thank you for letting us have this opportunity to talk to you. I did want to say a little bit about why I'm excited about this. You know, we're we all like to talk about origin stories and I was told not to talk about the Yellowbrick one, but when I heard that we were speaking with you, I I guess going back to COVID days, I was super excited. I have a shared heritage. I I'm from you know, roots in Western Pennsylvania. A lot of the hard code that I wrote for Yellowbrick was written down the street from you. So it just it felt like a really great thing that I could get involved with. So really excited to be here today. And and I wanted to talk about you know, why Yellowbrick felt as if you know, just bolting on an open table format reader at the bottom of of Yellowbrick would work. And I was one of the people that resisted initially. I said, well, you know, we can just do parquet reading all day long. We already do this. We do data loads at high speed. But really the the shift that Mark alluded to really drove us to the point where we said, we need to sort of rethink how we do this. Another driver besides the the need for you know, living in an ecosystem rather than being the center of of the of the universe for data is customers want to consume as a service rather than I install something in my VPC or on premise. We still know that customers want to do that, but they're moving towards this sort of way of doing things. And so a warehouse has to sort of live in this highly integrated place. And integration is always messy. It's it's actually quite messy with open table formats. There's different formats for metadata. There's different ways deletion is represented. There's different vendors. They have the different dialects. So it's a it's an integration sort of major challenge. So right off the bat, you know, how do we solve analytics at scale the way Yellowbrick did? We know that we are going to have to really work hard to match the performance and reliability of of our current offering. So we have to supplant that with some new technologies and we wanted to start from you know, sort of a clean slate in some ways, but obviously bring over key technology from Yellowbrick that we know works and we know can be sort of recrafted into this new way of delivering a service for our customers. And and and part of that huge challenge is you know, we hear in the industry about unstructured data. I like to call this undisciplined data. You know, parquet can represent statistics, but all the writers out there are are really undisciplined and don't give us a chance to do really good planning. And if you have a large petabyte scale MPP data warehouse, you really need to have great statistics to drive the beast which is the planner. If you don't have a good plan, you're going to end up reading everything all the time for all your queries, which means you just can't scale. And another driver here is of course agentic workloads are really good at driving crazy SQL and and are much worse than ad hoc analytics that we've been challenged with with some key big customers. So we know that it's even going to get worse. So we need to be able to scale horizontally very rapidly for to delivering the service and we need to embrace the idea that we no longer own the right path. And that is a key part of why Flow has to exist rather than Yellowbrick which did own the right path for how data was was stored in its back end. So we're bringing over four key things. One of them we call Flow SQL. It's really three components of Yellowbrick. I mentioned we have a planner. It's a Postgres hard fork that you know, Andy I know you had you know, represented in your prior talk. It's really a planner that understands how to do MPP distribution and a lot of other tricks over the years to make this go fast. A transformation pipeline that moves queries from AST representation to different transforms all the way to object code where they are run on our compute clusters. So this is really what we're calling Flow SQL now. So three key contributions and and really kind of you know, expanding on that as we landed it with with Flow, but starting from a really grounded you know, footprint of of technology that we already invested in. So is Flow SQL a hard fork of Yellowbrick? Uh Or or is it or is it modularized? I think I'll probably answer yes. Okay. We're not you know, if if there is key things that we needed to backport and forward port, we would do that selectively, but we're not going to maintain the heritage and lineage back and forth. Um There's there's there's huge reasons about this. Our storage engine for example is completely new as as it relates to talking to open table formats. So our compute cluster workers are now divided into two sort of microservices under themselves. So we've got the the execution engine which is not predictably scheduled with a typical operating system you know, scheduler. And then we do have you know, a storage engine that does do it you know, sort of conventionally with memory model and such. Got it. Does that make sense? Yeah. So so we knew we knew how to for example transform queries into high highly efficient machine code. We knew we wanted to capture that. We knew they were going to run in a certain way in a scale out MPP. It's no longer shared nothing because of course you know, the data is stored in a in an object [clears throat] store and we don't even control that anymore. It's controlled by the customer. But we did know that that would really work well for open table formats especially you know, experimental work that we did. Flow WLM is super critical as well. This is the thing that we we do believe will let us tame misbehaving workloads. It lets customers It's really kind of the knobs that you use to tune Yellowbrick and we and we believe it will be also the same sort of mechanisms but that that administrators will use to control the the chaos that that they plan on using this system with. Okay. Um next slide, Mark. So I wanted to talk through you know, more specifics about the design center. So obviously we're open table format native. Multi-tenant. A huge part of what we needed to change. You know, I'm going to get into a little bit about what you you guys are discussing right now with with Postgres and how different Postgres data warehouses work and Postgres as a service and whatnot. But we knew that we we we were going to change things pretty radically. We knew that we were going to have the need for shared services. We were going towards a microservice architecture mainly to allow different parts of our system to evolve, but also sort of land in in in a lane if you will and and and exist and and and sort of um improve independently of one another. Um Zero downtime. A lot of customers a A of vendors talk about this. We knew that we needed to deliver it. Um it's the expectation the customer has. It's very different um from what we've seen in the past with, you know, offline upgrades with Postgres. Um and we needed to really know how we were going to nail this at the get-go. Catalog upgrade is a huge problem for Yellowbrick, for other vendors that are based on Postgres. It's it's a huge challenge. Um and we were planning on doing it in a completely differently. And and Mark's going to talk about that a lot with with um some technology we've open-sourced um to make this, you know, widely um available to anybody to use. Is Is there a control plane for you guys still based on Kubernetes? Yes. Got it. Make that Yeah. Okay. I I've got a good diagram of it. Um it's it's going to have to be multi-region, obviously, because the data gravity for where the stuff lives is where the customer wants it. Um but yes, it's Kubernetes. Um and from a connectivity standpoint, we still want to be Postgres compatible to the degree that we can. You know, I I I watched a one of the presentations the uh a vendor Yugabyte did uh a few weeks back. It's impressive what that company does in terms of Postgres compatibility. We're not approaching Postgres compatibility in the same exact way. Um we're an OLAP database. That's an OLTP-centered database. They really strive them on on and being, you know, really good at that compatibility layer. What we see Postgres compatibility is is a bridge to where the industry's going with Arrow Flight SQL. Um we know that a lot of the agenda workloads, a lot of the Python sort of centered workloads, want to consume data in columnar format as it's streaming back from the the vendor. Um so, uh you know, Postgres, we we do have a libpq compatible layer. We have a personality that looks like Postgres when you know, interact with the system catalogs. So, there's sort of the implied um parts of Postgres compatibility that we have as well. But it's not a Postgres fork anymore. Um there's parts of it that are originating from Postgres, but it's very different. Um authentication, authorization, all very modernized. Um we know that, you know, customers want to use uh mutual um mutual trust for zero um zero trust um mutual offers zero trust um OpenID Connect OAuth 2, personal access tokens that are instead of passwords and the whole password reset and all that kind of stuff that that customers really uh struggle with. And then, you know, one of the the main parts that's different about, you know, the integration sort of challenge that we have with catalogs is identity federation. We have to embrace this at the get-go. Um you know, a journey that a customer might take with us might start with static credentials, but they eventually want to host all the access control, the governance, um etc. within these uh you know, catalogs. And the catalogs then own authen- authorization. So, we have to play as a full-fledged participant in that in that world. Um obviously, non-goals, we're we're not initially going to have a traditional um asset transactional write path. Um we will have temporary tables um at that the at the start of beta. Um but not traditional Postgres uh at all. Uh next slide. So, high-level architecture, um I we see some vendors have uh like a reverse proxy in front of uh of a traditional Postgres with virtual machines. That's not what we're doing. We've rebuilt the libpq compatibility layer. We call it terminator. Um it's a sibling to Arrow Flight SQL. Um so, it's a full um these things both delegate their their a lot of their session state to a session service, which is also a horizontally scalable service. So, these three things are shared tenant infrastructure. Um you'll see that through some diagrams I'm going to show in a slide or two. Um The query life cycle, similarly, is shared. Um where we get through, you know, planning and optimization, these are scale-out stateless services that can do a little bit of unit of work very rapidly and return answers. Um so again, where where we were looking at, you know, Yellowbrick um as sort of the beginnings of scale-out with our compilation service, um we're really kind of embracing this across the board. And a lot of that also is because of, you know, experience in running Postgres with really bad uh query [laughter] plans. Um they can go wrong. It's it's the worst kind of noisy neighbor problem you could ever see where a memory um you know, crash in in one planner brings them all down. So, we want to build a service that's resilient to that kind of pattern. Um especially because we know that machine-generated SQL is the way of future. And um it's even worse, you know, coming with with agents doing the work for um for us. The The The thing I would say also is that um an- another huge component that doesn't exist in a sort of formal way when you look at a Postgres is the metadata services. Um and we can kind of lump the catalog uh of catalog management into this. Um but as well as when you don't own the right path, we're going to have to supplant that with um additional analysis of data over time um to do statistical sampling to be able to drive the beast, which is the planner. Um and um Mark's going to do a really great job of a number of slides on this, so I don't want to do too much talk on this, but it but it's also the place where we get our database personality. Um so, when you look at um Flowcat, it's it's it's super cool. It um it builds on the ability to to to have different personalities. Um initially, it will be Postgres, but um we we imagine there will be others that that show up later. Hey, quick question. When you say personality, what do you mean? Is like mimicking Postgres wire protocol and catalog and you can mimic, I don't know, MySQL wire protocol and catalog? Is that what you mean by personality? Exactly. So, it's not necessarily the wire protocol. Actually, the wire protocol for for Postgres is super simple. Um it's not a lot more than just send SQL, get an answer. There's some variants on that with the the prepared statements and such, but wire protocol's not hard. It's the all the implied things that you discover when you connect to a database that given it personality. Um so, this is, you know, something that we embrace as, you know, a possibility. Um we want um the the core engine to have a a Postgres flavor, um but it doesn't necessarily need to to stay that way. Um >> [clears throat] >> if that makes sense. Um it'll make a little more sense, I think, when when Mark talks about the the data representation. Um again, when we get back to core um principles, we need to be able to upgrade the catalog. And I'm going to get into this a little bit more of it, but we're going to have to have multiple versions of the catalog running for the same customer, which is a huge challenge. So, we've we've taken that head-on as well. Um so, cluster management is not just, you know, you get compute cheap and easy no matter when you want it. It's actually a lot more challenging in the real world. The myth of infinite scale really is is, you know, something that that customers sort of confront themselves with when they get to cloud. Um we have a a formal way like a a formal service, if you will, that that touches the cloud provider, um acquires and keeps warm compute resources to provide instant-on um cluster availability. Um and then, when we run queries, when they actually run machine code, they're actually running in a dedicated compute tier per tenant. we still have tenant isolation, whereas if if the control plane, you know, blur in terms of terminology, what is this thing that runs the shared services that get you to the show, which is running the query, um it is mostly shared infrastructure um with some stateful parts of it, which I'm going to talk about in the next slide. Um So, this is a depiction of a single flow cell. Um we have to deploy this wherever data lives for a customer. So, this would be in a particular AWS region, Azure region, GCP region. Um it is running on Kubernetes, uh and it is done a little bit differently so that we can have a a pretty elaborate service evolution that I'm going to talk about in the next slide. Um the green part of this this slide is is the um tenant isolated parts of the workload. So, this is where, you know, a query lands and actually runs. It actually touches the customer data at that point. Um it um it will cache, for example, reads that it's already done, so it doesn't have to do it repetitively. So, there's a a fair bit of sensitivity to how we deploy that. Um it's network isolated from one another. And um and the rest of the the the services here, um even though I I don't represent this in the picture as multi-pod, every single one of these sort of boxes, if you will, um the blue blobs are all horizontally scalable. Um when we look at a database client at the top connecting to our terminator, um it it immediately reaches a a scale-out service that deals with protocol termination. To um similarly, Arrow Flight SQL would would land in that same spot. Talking to a um a scale-out uh session service. A number of these services use KV store um as a temporary scratch pad for state, and and then others don't. And the ones that I've um labeled in blue in the center, these are what we call tenant affinitized services. Um some of them have a strong affinity for a tenant so that they can perform better. So, for example, um on-disk caching for the compile service. We um when we encounter a SQL plan, we we give it um a hash and decide, oh, well, I might have seen this before and I can actually just not compile it because I've already seen this exact AST before. It just changes by parameter. So, we gain some some speed that way with repetitive queries, kind of like a prepared statement, if you will. Um Cluster services, similarly, we we connect to these green um pods in a in a persistent a persistent bidirectional stream um for security reasons. Um the Flow Cat catalog also is preferred to be uh affinitized uh per tenant. So, this means that a particular part of this deployment um it might let's say we had eight uh query life cycle pods. This is where a WLM runs. Um if you're Coca-Cola or you're Pepsi, you're going to land in one lane or another and you're going to stay in that lane. And so, we have a a bit of technology called service routing, which isn't unique to Flow, but is is employed to make sure that happens. And then we have a fleet of planners. Um planners are tiny little single core um restricted memory things that we can just scale out into a dozens, if we need to. Um and uh compiling is a little bit more concurrent um we found so we can actually run many many different compiles all at once. Um and so, we but we do want to scale that out independently. Are are the services still communicating that with each other through your like custom like reliable protocol on top of UDP or you just going you're just using TCP now? Great question. So, we've embraced gRPC for all microservices except for distribution among worker pods. So, if we look at that uh green blob with three workers, they're using our custom protocol still for efficient distribution of data amongst themselves. And everything [clears throat] else everything else is using gRPC contract first. It was a huge win for us to to bring this up quickly um and has been really really awesome, actually. Interesting. Okay. Yep. All right. Mark, next slide. We'll try to keep on track. Um [gasps] This is where I think it'll be interesting for those that learn about Postgres. Uh I always felt as though Postgres was was well, initially it gives you so much, right? You've got a huge lift in terms of things that it can give give you. Um it's the the the protocol on the wire, compatibility with a huge ecosystem. I don't want to knock it. It's just massive in terms of win. Um but what what I do want to to to point out, I call it flying inverted, is a fair number of OLAP vendors, and there's a huge number of them. There's ParAccel that became Redshift, Netezza, Greenplum, uh Yellowbrick, um sort of start with Postgres and then chop off the the legs, uh we call it. Um And the storage engine the execution engine is the bottom half. I'm trying to think of the right analogy here for what we're doing. And it it if you think about, you know, where do we put the planner? It's subservient to the thing that talks to the end client tools. Um so, while it is parts of Postgres, it isn't the Postgres that we saw it being utilized before where you have sort of a monolithic deployment of Postgres. It is a tiny little part of a larger microservice ecosystem. Um and I call it sort of flying inverted. Um probably the best analogy is if if the planner is the brain of Postgres, then we've done a brain transplant. If that makes sense. Um the um the main way that OLAP has sort of been served is either to you know, do this with Postgres extensions or to virtualize it or you know, sort of do this sort of chop in half uh trickery. Um what we're doing in in instead, you know, um we have a new protocol handler front end. The microservices are orchestrated using gRPC. This is going to be huge for us to evolve Flow as a service um and deliver it to our customers. Each of the microservices um we've embraced polyglot. We have a distributed team. Um a lot of of our engineers are in the EU. We still have a number of you know, engineers in the US. Um and we're allowing the the the sort of independence, if you will, for the different teams. So, a fair amount of our code came over and is just staying the way it was. Some Java code some um some C++ code. But for example, we're using Go Lang extensively in several parts of the stack. Um especially where it touches the cloud provider. It's the best language for dealing with cloud cloud provider specifics, for dealing Kubernetes, cloud native concerns, this kind of thing. Um you know, we didn't utilize any of the the the benefits of of Postgres, but we also didn't really want, you know, a um a need to for example, customize pg_hba.conf for every uh client that connects. We needed a new way to deal with authentication, authorization, and then just embrace that we're just going to start over. Um so, it's OIDC, um mutual auth, passwordless access. Um and then, you know, full participants in this um one of the kind of interesting things that I am excited about is I I mentioned earlier, you know, noisy neighbor, you know, a Postgres plan for a a a service that you were doing monolithically, you know, if you have an agentic workload that writes an incredibly bad plan you know, with the old approach that Yellowbrick had, that plan sort of never landed in WLM until you know, we have a signal that it arrived a query arrived, but we don't have any control over its memory utilization or, you know, runaway process kinds of things. You know, and I would also say that, you know, the the memory model for Postgres is challenging to the best of the the developers in the industry. It's just really hard uh to innovate on. Um so, we've gotten good at different parts within the planner um and and so, we we feel as though we can innovate really rapidly in in in a number of directions uh doing things this way. So, we think it's somewhat unique, probably um in approach. All right. Uh next slide. So, um compute and infrastructure um I mentioned earlier we have shared infrastructure for the main uh processing um pipeline. I think I'm running uh about slow a little lag on time, so I'm going to try to go through this quickly. Uh this is a little bit um repetitive to earlier slides. Um You know, we are using microservices, as mentioned. I wanted to talk a little bit about the deployment and routing. Um we know that what we want to do as a Flow service is deliver uh multiple services concurrently. So, this means for example, we might have three versions. We might have a leading edge version that's being beta tested by customers. As a service, you might have to be um you know, holding on to a couple different versions lagging um for certain customers. They may pay to to have that. We haven't decided if that's the way we're going to do things, but we need to design for that. And so, we know that not only will we have multiple versions, you know, coexisting on the same infrastructure, but we also know that we want to test this way. We want to develop this way. We want to be able to deliver um change this way. Um patches this way. Um so, it means that you may have three of the 12 or so microservices that need to be changed. We deliver a change to those services and we route uh work for an account or a um subset of an account to use that, um test it, deliver it, and then and then Flow upgrades from that standpoint. And this will be a constant sort of evolution of the service that we deliver. And last slide, let Mark get into um Flow Cat. Um I've talked a little bit about, you know, upgrade requirements, um the coexistence of different versions. I just mentioned about the service routing. Um the catalog upgrade pain that we've you know, we've sort of suffered through with with uh um with Yellowbrick, it's a it's a different kind of part. We think we've done a really good job with Flow in dealing with multiple um readers writers of the catalog coming from different versions, fixing, you know, the need for ongoing catalog maintenance, vacuum, export, import during upgrades. And for that, I think it's a good tee up for for Mark to to talk about Flow Cat. Thanks very much, Kurt. Yeah, so um I'm going to talk over the next two sections on these metadata services that Kurt referred to up front. And and Flow Cat is a catalog of catalogs um for query planning across Iceberg and Delta tables. Um First of all, Flow Cat's actually open source. So, so while Flow itself won't be open source, we think there's actually merit in in open sourcing our new metadata catalog and uh providing the ability for folks potentially down the line to add particular database personalities to it. And I'll get onto what I mean by that a little later on. But the role of Flow Cat, first of all, is to federate metadata across a bunch of upstream catalogs, metastores, and tables. And and if you look at a you know, a complex modern data lake where data lakehouse ecosystem, you'll see lots of different metadata catalogs. If your chosen deployment's in AWS, you might be taking advantage of the Glue catalog there. Or if you're a Databricks user, your Delta tables might be managed by Unity. Um if you're a big fan of the open source world, then perhaps you're deploying Apache Polaris or Nessie in in your own cloud accounts and managing those Iceberg tables um that you yourself. Um so, what Flow Cat does is actually sits um sort of downstream of those catalogs and polls those catalogs to extract metadata from them, to extract descriptions of the Iceberg tables and Delta tables that they manage, and provide a a a kind of consistent front end that Flow can take advantage of. So, Flow Cat is also this metadata control plane, which means it's really about mediating access to these the metadata associated with the Iceberg tables and mutating that metadata, putting in place policy overlays, and providing kind of consistent unified set of APIs for accessing that metadata. Metadata normalization is one of the key roles of Flow Cat as well. And so, essentially, catalogs are representing statistics and metadata in their own way that they want to. And so, what we do in Flow Cat as we're ingesting this metadata from upstream catalogs is persisting that into a canonical form for consistent sort of retrieval by by the Flow SQL planner engine itself. So, from a planning and interoperability perspective as well, Flow Cat is serving planner ready to statistics that are kind of Postgres based planner optimizer need to get very very efficient sort of rewrites and plans that can go on into those worker nodes, those compute nodes for for the for the main sort of execution phase as well. Do you support DuckDB? So, we support we support certainly support DuckDB and I'll talk about DuckDB and how we interact with that later. But, anything that presents, for example, an Iceberg REST API to it, then we can we can we can pull in. So, that's one of our principal ways in which we interface with other lake houses and and catalogs. Got it. Okay, thanks. Okay, so what problems are we addressing? And I kind of mentioned there are lots of different open source catalogs. So, a great question is why are we building our own when there are others out there we could take advantage of? And, you know, I think I think at the moment with quite a fragmented picture of lake houses that we have today and all of these different catalog technologies all doing something different, you know, what we wanted to solve it with Flow is the idea of allowing customers essentially to start breaking down these silos of control over different catalogs and different places that Iceberg and Delta tables are located and provide a common front that Flow SQL could query against. So, you can effectively do things like join Delta tables to Iceberg tables in different silos and actually kind of get a much more unified view of your data estate in in your in your enterprise. Um what we also found when we were looking at, you know, what Iceberg and Delta offered in terms of statistics, we found it wanting. You know, for the Yellowbrick {slash} Flow planning step that we have, we need much more advanced statistics for planning and estimation than than you get with Iceberg today. And we also find that a lot of these sort of catalogs are really great at kind of capturing and checking and mutating snapshots as they get added to an Iceberg table, but they're not really tuned for high performance delivery of statistics and metadata to a planner that needs to answer sub-second queries within an SLA. And and last but not least, as Kurt mentioned earlier, there's we're seeing this kind of rise of agent KI and and sort of swarms of AI agents generating absolutely terrible SQL and throwing at a database as part of their kind of reasoning steps and hoping to get good answers out of that. And and you need to put checks and balances in place. You need to have a consistent semantic view of the world for these things to reason about and also guardrails to stop them kind of going going off the going off the reservation. So, from a high level architecture perspective, Flow Cat looks a little like like this. And so, we've got at the very bottom the kind of upstream catalogs like Iceberg REST catalogs, Unity, Glue, so on and so forth. And also those kind of raw Iceberg tables that are just in a bucket in in S3, for example. Those feed into Flow Cat. We have connectors in Flow Cat that periodically poll these system these tables and catalogs and get updates as these new snapshots are added to the tables that underpin underpin these these formats. And what's interesting is we're polling at at now, but we're seeing more and more of a call for more of an event based notification. And you'll be seeing that kind of thing added to the Iceberg REST specification. I was at the Iceberg Summit in San Francisco just last week and we were talking about these kinds of developments to get Flow much more synchronized with what's happening out in the rest of the ecosystem. Flow Cat has pluggable query planning engines. I I want to be very careful. That's probably a badly worded phrase for what we do, but this is the the Postgres personality. It's essentially the system catalog that Flow Cat presents that Flow SQL can use. We don't do the planning itself within Flow Cat. That's that's Flow SQL's job, but we provide the metadata in a nice sort of bundle with a bow wrapped around it that that Flow SQL can consume. I talked a little bit about metadata federation. We want customers to be able to join tables from very very distinct separate data silos across their business. And we want to provide the advanced statistics and generate those when they're missing from the underlying Iceberg table formats. In terms of query engines, the number one customer for Flow Cat today is of course Flow, the Flow service itself. But, because we've implemented the Iceberg REST catalog API on Flow Cat as well as Arrow Flight APIs, that opens up Flow Cat to be used by things that aren't anything to do with Flow. So, even today, you can go download Flow Cat from GitHub. You can point it at your Iceberg tables or Delta tables and then point DuckDB or Trino or whatever you want. Anything that's compatible with Iceberg REST catalog API works with with Flow Cat. Kurt, can you share more like for the the statistics side? If I remember correctly, the Iceberg statistics are pretty simple. It's like you know, like min, max, maybe number of nulls, but like obviously Postgres has histograms and other systems and other things. How do you and you want to unify these statistics, but if like if one catalog source doesn't have good doesn't have a good statistical representation, but like, you know, for example, Postgres does or at least a better one, do you try to unify that or you or and are you is Flow Cat allowed to go collect its own statistics? How does that work for you guys? Yes, I mean, it's essentially yes to all of those things. We do as you as you pointed out, there's a gap, a huge gap between what Iceberg has and what the the fairly simplistic planning that things like DuckDB and I don't mean to do them down cuz that's Yeah, you're right. You are right. Yeah. But, but but you know, yeah, and you're right. So, Iceberg gives you, you know, basic things like row counts and number of nulls and a min and max. You have to go out of your way, and it's still supported by Iceberg, to do things like number of distinct values. But, it doesn't say anything about distributions or most common values or or anything like that or average column widths, things that you need that Postgres needs. So, we have to generate those. And so, what we're doing in Flow Cat is on the fly augmenting any mis- any statistics that are missing from Iceberg and generating those stats ourselves and serving them up. And much more on much more detail on that to come in the in in the subsequent slides, actually. Okay. Okay, keep going. Okay, the metadata model is very simple, as you can imagine. We're a multi-tenant system, so we have an account layer. Then under that, we have a catalog, a sequence of catalogs belonging to a particular account. Within there, we have namespaces, which is the kind of schema level sort of divider that you might think of from Postgres or whatever. It's namespaces in the in the open table format world. And then under that, we have tables and and view definitions. Under that, for every table, you have a set of snapshots, and every mutation to an Iceberg table lays down a new snapshot. Now, in in our canonical metadata model here, we have you know, the the underlying data schema, the primary key, foreign key constraints that apply. Incidentally, that's another thing that um Iceberg doesn't track that we need. So, we we we have to augment, infer, and add that data around constraints. You have a physical location of the underlying Parquet files that and metadata files associated with the Iceberg table. But, of course, more important well, just as importantly, those statistics. And we have to, as I said, Andy, add the ones that are missing from Iceberg today. So, add the generate those histograms, generate the NDVs, the MCVs, put in place theta sketches that can be used ultimately to do multivariate statistics across columns for cross column correlation that we want to do later down the line. And you can with theta sketches, you can do nice things like union and and intersect them and things like that. So, we we we're we're kind of storing this in a in a canonical model within within Flow Cat. And the architectural principles are fairly simple. We talked already about multi-tenancy. Stateless services was very important to us. We don't want to manage persistent block storage and and Kubernetes storage volumes and things like that. We we we want this thing to be horizontally state scalable and stateless. Um So, from a from a consistency model, we also implement optimistic concurrency control it for our storage access here. And so, Flow Cat stores its metadata from these upstream catalogs and tables in a key value store in an object store. Our first deployment of this service will be in AWS. So, not surprisingly, the key value store is DynamoDB. The object store is S3. And we use compare and swap pointer updates to update versions of pointers when different upstream clients are adding or mutating data within Flow Cat. So, we don't have any global locks within within Flow Cat. So, um Kurt also mentioned about authentication authorization. In Flow Cat itself, it's defined at the boundaries principally by uh your identity provider uh that you use to do kind of single sign-on that defines the identity of the user and also the policies and the claims with which they can put in place that are in are in place that are enforced in Flow Cat based around access control and things like that. Okay, uh the major architectural components of Flow Cat, if we start with um the top kind of right, the catalog metadata, that's the core part of the storage piece for the metadata and statistics we collect. So, we handle kind of directory lookups, metadata handling, and and persistence, transactions, connectors uh for upstream uh catalogs are stored and retrieved from there, and all our stats are placed in there as well. And they talk talk to the pointer store and the blob store. Um the query planning side of things, this is really invoked uh when a query hits the planner within SQL Flow. What it does is the planner reaches out to Flow Cat and begins this kind of planning transaction in Flow Cat. And what that really means is it's the instruction that given a set of tables and columns that Flow Cat SQL provides so Flow SQL provides to Flow Cat, Flow Cat provides all the metadata and statistics needed to to go on and plan that query, okay, in a kind of single transaction. We have the set of scalable asynchronous workers which are busy uh mining upstream Iceberg tables and catalogs and bringing in the latest versions of those snapshots, and also doing things like garbage collection from the S3 uh blob storage and doing reconciliation and backfilling statistics and things like that. In front of all of that, we have this protocol gateway which is the route that um open source query engines like DuckDB, Trino, and others would would do would would come through to access um uh the the catalog metadata that we store here. And everything gets converted to gRPC. Uh so, all of these are separate microservices uh running in the context of Kubernetes in the broader Flow service. Okay, so um I want to talk about how Flow Cat makes query planning deterministic, and this is quite important for a lots of different reasons. And you can imagine I painted this picture of lots of different upstream Iceberg and Delta tables all all marching forward and all evolving on their own timescales, creating new snapshots all the time. So, we want to make sure that when we run a query and join two tables that are in different kind of silos, we can do that consistently and deterministically. And so, how it works is at the start of a query, Flow Cat pins the snapshots across all of these different tables. Uh then it resolves all the stats and constraints associated with that uh that kind of bundle of of of snapshots, um and then feeds that back to Flow SQL for planning. But it also means that we can record that, and in the event of a failed query or or something that's restarted, we can replan deterministically using that collected stats bundle that we've already done. So, we don't have to go all the way back to source to to get that. Um so, why we think this is a little bit different compared to other uh uh catalogs out there, really people really aren't thinking about cross-catalog snapshot pinning um and and and then serving up the fine-level granularity of data that you need to plan um a SQL query because you need kind of two kinds of grains. Obviously, at the Postgres side of things, you you you care about overall table statistics and column-level statistics across an entire table. When you get down to data skipping in the workers, you care about the file-level statistics and min-maxes within there. Um so, we're we're serving up that kind of information at two layers. So, let me just say so so if if someone comes through like like say through Trino, and but it but like Trino's going to send you a SQL query, and then uh but let's say that query touches data at two different backend sources, right? The Flow Cat planner is going to generate a physical plan for how to to run that that query, and then send that back to Trino to execute or No. No, it's far simpler than that. What we're doing is assembling all of the um Iceberg metadata that Trino itself needs to go off and run that query. So, we're telling it So, what we're presenting it is a a unified sort of catalog namespace table view of the world across all of these different uh you know, uh inconsistent actual upstream sources. You know, so and a very very nice clean namespace um across across for all these different tables in these different locations that you can organize yourself within Flow Cat. And so so, no, all all that Trino say sees as it queries through the Iceberg REST API, it says, you know, um you give me the the S3 location for the metadata.json file for this particular Iceberg table, and we we to go with it. Then it's Trino's job to go off and directly query those things. We don't plan on on on Trino's behalf. Got it. All right. All right. I understand. I guess. Okay. Okay, I wanted to dive into some of the statistics. Um you know, I I NDVs is is an incredibly important, and I'm sure you guys know. You know, if you look at the kind of the big three statistics for planning in Postgres, you've got basic row counts. You must have that. You Secondly, you need uh most common values. And and probably the number three, you need the number of distinct values. And number of distinct values is very important because it allows you to kind of get memory estimates for uh group by operations. You know how many groups you're going to be have having to deal with. It's very important for doing join uh reordering analysis to make sure the intermediate result sets between joins remain small. You know, there's a huge number of of tasks that NDV is important to. And Iceberg only pays lip service to NDV, and you you can generate NDVs, but people don't typically do it. If they are generated, they're stored in a side file in something called Puffin format. You can generate them in Trino, but it's not a default option um because typically they're quite expensive to generate. But then we really need these stats in in Flow. So so, what we've done, and actually I'll I'll talk about another interesting use case, which is what the title about of this slide is about sub-second planning and doing sampling to get NDVs because we have a use case where it may be the case that a new table has been discovered, but we don't have full statistics for a particular table. We might not have all the NDVs for a particular column that we need at plan time. So, we still want to give the um Flow planner a hint, and we want to sample instead of having to to scan petabytes of data across all of the Parquet files within an Iceberg table. And so, we we do this in in in a way called row group sampling. So, every Parquet file contains row groups. We have three different estimators within our NDV estimation. We We um and these run at the same time as we're as we're kind of scanning and sampling different Parquet files. We look at what we would do if we literally linearly extrapolated from this sample set of NDV values across all all Parquet files, and we we we look to see whether that converges or not. We for for and that's really really good for kind of high cardinality tables. For kind of mid cardinality tables, this frequency of frequencies approach is really good where you got singlets and doublets that you're looking for. Um and that gives you quite a nice way of and there's a very very simple formula to get from frequency of frequencies to to a number of distinct values. And then finally, we have this approach which we call kind of the discovery curve, which is um track the rate with which I'm seeing net unique values for a column arriving. And so, you can see that curve if that curve starts to converge, you know you've got um you know, you you know you're going to get a pretty accurate NDV count very early. And so, we play these three estimators off of each other, and that allows us to get some degree of um confidence. If two of the three agree, it's fairly good confidence that confidence that we we've approached the right sample estimate for an NDV. So, on the right-hand side, I've got an example of how our um sampling and stats generator works here for NDV. So, at the very bottom, you've got the percentage of the of the um Iceberg table we've sampled. Uh at the very top, you've got the wall clock time, and you've got the NDV in millions uh in on the Y axis here. Um and this is for a table that many of you will be familiar with. It's the major store sales table in TPCDS. It's scanning across the terabytes a total of data of that data set and and in that particular table, 2.8 billion records. Um and for the SS ticket number uh column, which is what we're sampling and showing in the graph here, you see, you know, for for 2 3%, we're not doing very well in terms of approaching the the true NDV value. But as soon as we hit kind of 5%, and that takes about 4 seconds on a a 16 VCPU node in AWS, we're kind of getting up to the 213 million NDVs out of the 2.8 billion records that um that's in place there. So, so we have this use case that we want to try and return this in in less than a second. But as you can see, we're doing it in 4 seconds here, but it's an embarrassingly parallel problem. So, we can we can scale this out and and get pretty accurate stats um into the query engine when there are no stats in place at runtime. So, Flow Cat's going to go fetch the file there and then the Parquet file for example, and extract some like it's going to do the analyze. It's not going to load it off to some other engine. Like No, Flow Cat [clears throat] Flow Cat does it that itself. So, it has a horizontal executor uh framework that will go off and and greedily sort of read these parquet files that constitute a particular table. And do you um with some statistics like can you carry them over from one file to the next assuming they're like in the same, you know, partition? Like I like number of distinct values, you don't need to know the exact values, you just need to know the number of distinct values. Can you extrapolate like from this file within some cluster of files or bucket or whatever? I know that these stats like these files are very similar, so therefore I'm going to assume or the, you know, the next file I read in the same bucket, I I, you know, I can converge more quickly and not do the full 20 seconds. >> Yeah, you you can and and that's very that's very related and and there are lots of corner cases where things get very hard when you have clustered and sorted data. And that's where things getting convergence and agreement across these estimators becomes very hard, but for for the low cardinality fields where you would see say dates, there are, you know, there's a finite number of dates that people typically care about, those are going to come up again and again in every parquet file. Unless they're unless they're sorted by date naturally, so yeah. Okay, okay, thanks. Pluggable system catalogs then, so talked about this a lot. What's what's the idea here? Well, you need much more than just table names and columns to plan a query. You know, SQL contains functions, operators, types, primary key, foreign key relations. You you need to present to a query planner this facade and present the system catalog in a way that it expects. So, as Kurt mentioned, the system catalog in Postgres isn't fit for purpose for what we needed in Yellowbrick to a degree from a zero downtime upgrade, it's not fit for purpose either when you take it over to the open table format world where where things have are very very different, but we still need to present a Postgres flavored system catalog to to Flow SQL. So, what we did was it's threw away the PG catalog and basically implemented a pluggable framework that would allow us to implement PG catalog in Flow SQL. So, that's what we have here. In principle, if someone wanted to rip out a catalog from another query engine and and create a catalog plugin for Flow SQL, they could do that. Uh Flow is the main customer and the only customer for this at the moment, but we wanted to make it flexible so that, you know, you didn't have to have a hardcoded one global built-in SQL as it well. You could you could provide a system catalog that other engines could consume in the flavor they expected. So, you can try Flow SQL. You can go to our GitHub page, download it. I won't read through all the details here just on the interest of time, but you you you can you can it's totally available, completely available today. Today we open sourced it back in January and so you feel free to to take a look. We'd love your feedback. Flow scan. Okay, so statistics is one side of the equation that we need to get Yellowbrick performance in in Flow against open table formats. The other side is indexing. Indexing is another area that's actually underserved in the Iceberg and and Delta world here. But something we we absolutely need within within Flow. To to illustrate the problem, one of those that store sales table that I talked about earlier, this is one of the parquet files out of that data set. It consists of around 800 megabytes of compressed data. There are six row groups, each about 130 megabytes. That table has 23 columns, so there are 23 column chunks per row group, about 5 meg each. And then under there, the structure is they have around 180 data pages per chunk. Okay. Brilliant. So, you actually got fine-grained skip level control that you could potentially use within parquet files. But there are limits with what you can do with current technologies and engines today. Part of the problem is the writers like Flink and Spark SQL that generate these Iceberg tables seldom populate the built-in page index that parquet supports. So, you you're kind of flying blind immediately. Why don't they? Well, they're expensive, they bloat your parquet files. But what that means is most tools like Trino or DuckDB, when they're skipping data at the parquet file level, are only doing it at the row group level. So, you can see that row groups those row groups are 130 meg and there's only six of them, right? So, using min max across row groups, you're really not skipping that much data. And that's far far away from the fine-grained skipping that we did in Yellowbrick. So, our goal is get the Yellowbrick skipping efficiency and and and, you know, claw that back for for Flow. And what's also really interesting, again motivated by a real world use case from from our customers, is, you know, we we have this we have customers that will do insane levels of of lengths of in lists. So, we have one customer that has an in list within their where clause that's 100,000 integers in length, right? And and they what they're doing is looking at multi needles in a haystack across tables with a trillion records in it. And the way that Yellowbrick works is we have something very pretty pretty neat called SIFT, which actually takes these very very huge index clauses in lists, rather, sorry, and then breaks those down into meaningful min max um intervals. And then in Yellowbrick what we were doing is able to apply up to 10,000 min max pairs per column and really fine-grained extract data out of Yellowbrick's data. Now, we're in a different world now. We need to do this in Iceberg. Um So, we're going to we're building these indexes down into the parquet data page level. We're going to first of all pull data out in a kind of S3 friendly size of 256, you know, 512 K blocks, cache it locally on NVMe and then use our existing IP around, you know, user space kernel bypassing and our own NVMe drivers to get that down to a 32K kind of seek within and look up within the NVMe caches themselves. That's how we're going to claw back Yellowbrick performance at against open tables effectively. Um Again, in the interest of time, I know we're kind of approaching the hour here. I'll I'll kind of gloss over here, but as you can imagine, actually Flow SQL is the ringmaster for the Flow scan architecture. It's responsible for managing the job queues for indexing new tables as they're discovered. We have a horizontal set of Flow scan indexing pods that are doing that the greedy work of scanning through these files and indexing them. They create parquet sidecar index files that sit alongside the Iceberg tables and parquet files that have offsets into data pages and and information about the dictionaries with which those data pages are encoded. Then we have a a Flow scan lookup service, which is ultimately providing um data page indexes and offsets to the Flow SQL workers. And so, on the right, the SIFT engine gets a really complicated, gnarly in list and very very detailed very deep sort of tree of ors and ands predicates within the query, decomposes that into a set of min max bounds, looks those up, hits the lookup service, gets the data pages that are relevant, feeds those into Flow SQL to to look up in the parallel workers. Hopefully that makes sense. All right, Mark, we're we're getting kicked out, sorry, of the room. You want to you want to finish up? Yeah, okay, so I won't talk about the index performance then. I know I won't kind of talk about the why it's still hard, but I'll I'll put leave the the slide up here because it does talk about convergence around statistics when you're sampling and things like that. So, apologies for kind of running out of time here. Yeah, there's you have so much so much to talk about. It's awesome. You know, we should have done this we could have done this in two talks. Any from Mark or Kurt before we before they get kicked out of here? I guess I mean maybe the one question would be like the has I mean you mentioned on the engineering side you're allowing the teams to be distributed in different services, different in different environments, different programming languages. Is there anything about sort of the modern era of agentic, you know, coding tools that's making this easier for you to support, you know, things outside the Yellowbrick ecosystem than there were? I think there's I think there's absolutely an astonishing amount of velocity that we're gaining coding with agents. It it it's it's it's sort of a a hallmark of a of a senior engineer to be able to estimate, you know, how long something's going to take and I'm blowing my estimates all the time in the wrong direction or the right direction as we you might look. It's like everybody thinks you're sandbagging because it takes four weeks less than what you estimated. That's definitely happening to really smart people within our Yellowbrick Flow team right now and it's just it's not slowing down.