When an AI Agent builds the data pipeline and tests it too

Good afternoon, everyone. Uh I am Mayur, co-founder of Orca Sheets and D Talk. Uh it's always feels great to be a back at uh Fifth Elephant. And I hope you all are having good time so far. Um so, let's make this little interesting. Uh If you ask a question about this talk and that I can't answer, then we will give you 1 year of Orca Sheets subscription for free. Uh sounds exciting? Maybe not yet because you probably don't know what Orca Sheet is, right? But give me some time. Uh you'll know what you'll be winning, okay? So, I want to start this session by uh with a confession. Uh like for years, uh if somebody handed me a data set, uh my first instinct was to just like spin up a uh glue or spark or some other cluster. Uh and as a data engineer, that's what we all are trained to do. Um but today, I want you to like I will walk you through the journey which convinced me that we were solving the wrong problem. And throughout this journey, we ended up building an AI system which writes the agents to generate other agents which do the work for you. Uh But before any of that, I want you to introduce I want to introduce you to people that we kept meeting. The spreadsheet analyst. Uh these people live in Google Sheet or Excels. And the moment the file crosses millions of rows, the tool dies. And they will never write a code. Obviously, why would they? Uh And you won't believe this until you see it yourself. Like we've seen people splitting the data across hundreds or thousands of Excel, analyzing them separately, and then stitching the results back together by hand. This is insane, right? And why are they doing this instead of like putting the data into database? The answer is very simple because they are Excel ninjas and obviously all those ninjas cannot solve the problem of reconciliation, right? Now comes SQL analyst. They have data warehouse and data lakes of the world like Snowflake, BigQuery, Redshift, powerful, right? But any like any exploratory query costs lot of money and most of the time they are wait behind other people's queries. Then comes data engineers like us who can run Spark, Glue, Airflow. But if you look at those DAGs process, most of the time they are just baby-sitting the whole cluster for 2 GB job. The last one, the regulated teams like banks, health care, consulting engagements. Their problem is very simple. The data is not allowed to leave the building. And yeah, so all those four different people, four different toolboxes and when we talk to them, the complaint was same. I wait too long. I wait I I pay too much money to get the answer from simple questions about my own data. >> [bell] >> And if you look at their day how their day looks like, they generally have the data warehouses and data lakes, great. But whenever you are analyzing something, you are always almost looking at the fraction of the data. Either last quarter, one segment, one customer or the one region, right? And if the data is not partitioned correctly, then you are almost paying the bills for whole warehouses to get the simple answers. And the flux like you are if you look at the service provided by Flexera, Uh, the industry consistently put wasted cloud storage at around 30%. Like there are idle clusters, over-provisioned warehouses, and forgotten jobs that people forgot to like switch off. And the last line is like it takes days to set up those pipelines and seconds to run. So, uh, while building all these ETLs over the years, uh, we realized uh something uncomfortable. Uh, like most of the production ETLs comfortably fit into the memory of a single machine. The laptops that we have today has more RAM than uh the Hadoop nodes uh uh when we uh started our careers on, right? So, yet we provide like we provision all these clusters, all these big uh warehouses, we send data to cloud, and we pay egress uh money to get it back. And there is a like yammer of infrastructure which is just sitting idle. So, we asked, "What if we move the compute to the person where it's sitting?" And these are the three things, like security, obviously rows are not going to leave your machine, no egress, no third-party preprocessors, or no data residency. For the regulated team, the laptop is the security parameter, right? And it's not like a workaround for them. It's actually only the acceptable architecture for the regulated teams. Cost, obviously you have already paid for the compute, you have those laptops, and all these modern laptops are capable enough to crunch millions of rows per second. And any any analysis uh for is at the cost of zero, like the speed, uh obviously you're not waiting on a queue, you're not spinning any clusters, no cold start. Uh results are getting in seconds still the cloud is uh, getting provisioned. And to the obvious objection, um, like you will say we have the data in warehouse not on the laptop. You are right. Like you can keep your archive where it is, but just bring the slice that you are actually analyzing to the local machine. Right? So, even if your warehouse has like billions of rows, the uh, the slice or the data that you want to analyze always almost fits into your machine's RAM. Here is the example from uh, last month. One of our customers loaded like 2GB CSV containing 12 million rows and asked very simple question, "Cluster my customers by their purchase behavior." And 4 seconds later, they got the results. In-memory processing came in clustering and dashboard in the end. And nothing happened on Spark, not a clue. It happened on their own laptop with plain English. Right? So, how it is working? So, I'm going to share the architecture of the Orca Sheets. As you can see at the top level, there are a lot of data sources. Um, it can be local files like CSV, Parquet, or Excels. Um, we can also connect to any of your databases like Postgres, MySQL, or any data data warehouse like Snowflake or ClickHouse. Uh, if you don't have data warehouse, then we also provide a universal endpoint uh, which you where you can send the data and we'll create a uh, data lake for you. Uh, we also provide integration with APIs like Shopify, uh, Stripe, Razorpay. Um, and using those data sources, you only pull uh, the slice that you need for analysis. And the second, the middle part is interesting one, the Orca Sheets compute engine. Uh, there is no single engine. We always route your uh, workload based on what kind of a engine can process that workload easily and better. So, for example, DeckDevi is for OLAP sequels, pivots, or joins. Polar is for transforming your data frames. And DataFusion is for like streaming your query plans. So, our learning is don't marry to a single engine. Composable engine can give you a like way of operating around different data sets. So, never marry a single engine. Um So, you have Arrow in memory, Parquet on disk, and columnar all the way down. That's how you can easily move between engines completely free, right? And at the bottom, you get like aggregated sheets or the visualizations and dashboards. And something called recipes that we'll talk later. But, the important note here is your AI never sees the actual data. It only sees the schema and the statistics. So, your your rows never leave your machine. Uh I think enough talking. I'll just quickly walk you through the Orca sheets. Uh So, I'm going to load car transactions data here. Uh which has around 13 million rows. As you can see, I've loaded this data. It has 13 million rows, like around 30 columns. I can scroll through the data. As you can see, the experience is similar to Excel. We can do everything what Excel can do, like filtering, macros, and everything. But, I'll quickly ask one question. Spending pattern across buckets of four age groups. And what I'm doing here is I'm just sending the schema and the statistics about this data, and done. I got the results in the table of format. I also got this visualization because obviously visualization gives you better understanding of the insight, right? So, within seconds, I was able to do these kinds of ad hoc analysis. Uh but when you ask one question, it's analysis. If you're asking same question again and again, then it's a pipeline. And like pipeline is basically a repeatable workflow that you can automate. For us, uh it's a recipe. So, we have created a recipe for this kind of a data set. So, I'll quickly show you how recipes run. It's Yeah, it's composition of multiple steps like you are cleaning your data, pre-processing, then transforming, and maybe also joining some other data sets. And in the end, you get different aggregations done. So, the run is complete. I'll show you the output. It generated all these kinds of output uh like cheap and swipe transactions analysis, geographical analysis. Let's look at the results. >> [snorts] >> And in the end, you also get this whole dashboard. So, within minutes, I was able to get the analysis done starting from the ad hoc analysis till the dashboards, right? So, let's get back to the presentation. So, yeah, we built recipes. Engineers were happy. But I Sorry. Yeah. But the analysts weren't happy because they How would they write that those big JSONs that we have created as a recipes, right? So, we asked this question, like how would you uh not let uh get them blocked because of us because every time when they wanted to write a recipe, they had to come to us. So, it's like a services company wearing clothes of product company, right? So, it won't scale. >> [bell] >> I think I'm missing a slide, but anyway. So, this is as you can see recipe is just a uh some steps in a JSON form. If you have worked in on a any ETL frameworks, you know the shape. Uh steps are generally DAG task. It contains SQL ML commands and dashboards in order. Ours are generally provided to the pipeline as a parameters. Uh picker sheet or year or threshold, anything. And the state, obviously, the steps are communicating with each other. So, you need to maintain the step state. For example, if I'm providing output of one step to other, so I'll have to communicate that. You can think of it as a XCom uh without the Airflow installation. And this is the real example of orders analysis. As you can see, this is the JSON where in the first step, uh we are actually filtering the raw orders data. Uh and that orders data is getting fed into the second step, as you can see as in the args. The JSON path is getting the output of first step. And then you can run aggregations or generate visualizations or even run the commands like K-means. So, right now there are like production running recipes like Shopify analytics or ONDC order analysis. Uh there is also 11-step building rating system uh which contains like 23 SQL actions and four ML commands. Yeah, as I said, like every recipe all like came through us. Uh we uh they were blocked by us. So, we asked this question, how can we unblock them? So, that question created this idea. Uh where they'll provide like a problem statement in plain English what they exactly want and outcomes the validated uh repeatable recipe. Basically the agent that writes agent that does the work for you. And the important part is you only keep the artifact not the chat history. And how we are able to generate those kinds of recipes? So there is no single model obviously like we've been hearing lot about the multimodal architecture in previous talks as well. Here also we are not dependent on a single model. So there are three layers. First layer is a planner which gets the problem statement where you define what you want exactly from the recipe and there we provide the schema, we provide the column stats like min, max, cardinality or null counts and it just spit out the plan. No sequel yet and that plan is given to the generator. Now this generator provides or generates the full recipe which contains the steps, the odds and the sequels and chart configs. This generator has knowledge of the best recipes that we have created so far. And in the end obviously you need to validate the recipe generated by one model. So we have another model which status like syntactically and structurally checks the JSON. It validates the JSON, it then validates the references of the columns and then also checks whether the sequel is at least passing, right? So important take here is three small jobs beat one giant prompt. So if you have smaller models which are doing small jobs then it's easier to steer, right? And if something fails then it's easier to blame. Obviously these three checks were happening in the cloud against your metadata. Your cloud or your models did not even see uh a single row. So, recipe uh came to Orca sheet locally. Now, the real thing. Now, we need to do a dry run on the real data set. So, now Orca sheet uh converts those arguments, those sample values, and try to run every SQL action headlessly. And if there is a any real engine errors, then we send back those errors with the again with the details of the schema and everything back to the LLMs, so it can regenerate. Right? So, for example, if like uh the model created this uh column called takes and did, which does not exist, it should have been a transaction date, then it will regenerate. And this loop it uh it works only for like three times. After that, we uh stop so that there is no infinite loop and no silent failures. So, simple thing is cloud check the grammar, but only your machine can check the facts. Now, this is actually a tricky part like which uh almost took weeks to resolve. Uh As you know, like recipes are DAGs. So, step five uh like the is dependent on the output of step four. Step four is dependent on the step two. So, how I'm going to validate output for uh step five without even running all the previous or intermediate steps, right? So, am I going to run the whole recipe? So, it's not a dry run, right? So, to answer that, like we ask model to emit one more artifact, like a validation SQL. And validation SQL is just a chain CTs. If you look at it, the first step would be a with block, and then the second step as a select, and then the filters. And in the end, we are just doing a select star limit zero. So, this is a single sequel come converting your whole bag into a single sequel which can run without even touching a single row. Yeah, so obviously everything is generated by models. So, models can confidently invent new columns and we don't want to hallucinate. So, any unknown columns are immediately rejected. Any unknown identifier is always checked against the real schema. So, or even the functions getting used in the recipe are only allowed listed by the DSLs. So, whatever output is provided by the models, we don't generally execute them directly. We come We just compile those outputs. So, when you get those recipes, obviously like when you those models are going to generate something more than what you're expecting. So, you start getting new ideas like get me some outliers now or get me forecasting for next month. So, there is again iteration. There are a lot of iterations. So, whenever there is a iteration, we only keep the diff and those diff diffs are repeatable and those are stored version. So, we are not actually storing the chat history. We are only storing those artifacts. >> [bell] >> Okay, this is my favorite like don't trust the model, trust the loop. The The agents that can only succeed are not production-ready. The agent that can fail loudly and regenerate with the errors as evidence and store for version artifact is the real agents. Obviously, I like just before the demo this is not limited to sequels. The recipes can contain like clustering, forecasting, anomaly detection. And all those ML libraries are not like written by the models. Those are part of the auto sheets. There are big set of tools that we have written in Rust. And those are like binaries are max binary size was like around 5 MBs, not like 500 or 600 MBs which are written in Python. Okay, so it's time for the demo and this time I'm going to show you recorded demo because I don't want my demo to fail. As you can see we have loaded this hospital admissions data. It has around 37 million rows. So let's see how we can generate recipe on this data. So we go we go to recipe. And Okay. So you select a source. It can be a data set. It can be a connector, anything. You provide a problem statement as brief as possible. Like what exactly you are expecting out of this recipe. You can also provide all the job and that you your data has. And obviously if there is any cleaning or pre-processing needed you all you know everything about your data, right? So you provide all those information. If you're done any ad those as a part of the recipe, you can select then that into the existing analysis. And just say if you want to also include a dashboard. Done. Now what this is going to do, obviously it will send all the schema, all the statistics to LLM models with your problem statement and it will go through all these three layers and comes back the recipe. And once we get those recipe, we start validating those on the local machine. If we see any errors, then we loop back those errors back to the model and that loop happens as you can see it has generated different aggregations and dashboard. Done. Now you can name your recipe. Now it's ready for you to test locally. You can save it as draft. I can quickly try it, test it out. Select the same as we did already, right? So Done. So it is going to run all those steps. It is going to show you all the results for the steps. And exactly like I demoed earlier, right? So you'll see all the visualizations, all the data and the aggregations. And in the end he created that dashboard. Now that dashboard can be easily shared with your stakeholders using a simple browser link. If you want to generate report out of these dashboards, that is also done. You It's a one click away. Okay, let's get back to the presentation. Now let's look at the numbers from the workload at the beginning of the story. If you go to the serverless cloud ETL path, it will take minutes to cold start before any work happens. Build per compute hour with per run minimum. And your data leaves the network twice. But if you take the local path, it only takes 4 seconds to process 2 GB of data, 12 million rows, and every new analysis at the marginal cost of zero, zero bytes of egress. And these are the things I want you to take away from this talk. If you are building agents which are going to provide a structural output, then you should look at these five things, like split the generation into plan, generate, and review. Small jobs are steerable. One giant prompt is not. Ground in the real schema, send the metadata, never the rows. And validate where data leaves. Dry run locally. And what whenever you see the errors, just send it back. Any real error is better than the clever prompt. Make the model emit its own test. The CT chain validation sequels, it is actually creating a sequel for the whole DAG with without even touching a single row. And in the end, version the artifact, not the conversation. Sorry, and the eighth end state is agent is not the product. It is only an assembly line. The recipe is the product. And if data fits into your memory, then question the cluster. Thank you. I'm open for questions now. >> Let me uh Um If you take your customers, mhm, >> uh uh let's say Let's say you are talking about hospital customer, right? So, there can be four, five people who want to do analysis. >> Yeah. >> And do you expect the copy of the data in all the those laptops? Or how it generally happens in that scenario? >> Yes. So, actually this is uh very good question because uh all those four people actually might be looking at the different data sets. Even if you have one uh source which has all the data, but I am sitting in one region. I I should only get that kind of a data only for that regional data. And we have the IM control over that. So, if you have seen IMs uh in other uh like cloud storages, they are either on the table level or either on the columnar level. But here we are providing the row level uh also. So, we can filter the those data sets and you'll only get the uh filter data which is uh like uh permission to you. So, yes, uh obviously like there will be copies, uh but those are optimized to your disk. >> So, my source of truth is cloud again, right? I mean, S3 or something. >> Like you if your data residing is in cloud, yes. >> So, during the the analysis, it will come to my laptop based on my credentials. >> Yes. >> Authorization. >> Yeah. >> Okay. >> Um okay. >> [clears throat] >> Hey, um >> Yes. >> See, you um first of all, very interesting premise, right? And like at the start of our talk, when I was introducing this, I think I got confused because I was like, "Agent set right anything? Am I Am I uh introducing the right talk? Right? Because I was like, oh, this is about like um moving processing locally. >> Yeah. >> Um I think I have one question and um you you know that part about um the the sequel which which represents a DAG, right? That was very interesting. Uh but would it apply to uh let's say like other problem classes like clustering or anomaly detection? Would you would you still be able to represent that as sequel because that >> No. No. >> Yeah. >> Those are the tools. So, you cannot represent them as a sequel, but even before getting to the those ML commands, you are doing some kind of a cleaning or pre-processing. So, obviously, now you it's really uh a right question to ask because now as I'm not going to spit any output because I'm just validating using the dry run, but for the ML to validate those commands, I need some output, right? So, that's why you saw the select star from and with the limit zero. But in case of ML commands, we give some sample output. It's not the whole run, but we give like 10 20 sample outputs and then validate those ML commands, right? >> Right, because I I think uh when when you were presenting, I think what I took away was that I took away I I I interpret that as a general thing, which is you have a DAG. How do you the problem being how do you verify stage N without running stage N minus one? Oh, you can represent it as a CT, which obviously makes sense for you know, like data analysis, but I was wondering if like it I I thought it didn't immediately apply itself to like, you know, ML ML pipelines or anomaly detection or you know, K-means clustering. So, that's where the question came from. Thank you. >> Yeah. >> No, so no, no, no, no. So, we have Yeah. >> So, do you do you recommend your customers you should have this spec with this feature hardware? >> No, we No, no, we have benchmark this on a 2GB RAM, two core CPUs. It will be a little slow, obviously, but it won't crash on you. It will never crash on you. So, that's our guarantee. And actually, we have also one step further. We are actually capturing the system configurations, and that is also going to the LLMs. So, the queries which is getting generated also taking account in all those things. So, that is actually help whether while writing the queries because, obviously, LLM models are going to provide you optimized queries. You don't have to worry about all those things, but those parameters generally help. >> Okay. I think And this one will support multi like Windows >> Yeah, we are on all platforms. >> So, you kind of create a sandbox or docker kind of thing inside. So, virtualization or it's different because your engine may you might have optimized for some environment, right? >> No, no, no. So, it is built on Rust. It can work on any platform. >> Okay. >> Yeah. Thank you. I think the I mean, I was just thinking out loud. I think the if you look at what's happening with models like when that is, you know, from the cloud to moving models locally. I think this is generally in the same space. Like so, is that is that also sort of motivated? I mean is is that what you Do you see that in the same direction or is this is this >> I mean we can see that. I mean for some task for some smaller task for example even the visualizations that you saw were generated by the models, right? So, those kinds of small task you can obviously have a smaller models locally running and you don't pay for everything. So, yeah we can maybe think about those things. But not for everything. >> Okay. >> Because those models are heavy. >> Okay, I have one final interesting question. Orca sheets, how did you pick the Orca in the Orca sheets? >> So, that is the answer I think yes should take. That's the who is the inventor of this product name. But it's obviously it's a Orca is a killer whale at the top of the chain and most intelligent one. So, that's the reason behind the Orca. >> close to humans. I mean you know. >> Human friendly, yeah.