Analytics Architecture: Data Pipeline Engineering Strategies

Hello and welcome. My name is Mark Horseman and I am a data evangelist with data. We would like to thank you for joining the latest installment of the monthly data webinar series analytics architecture with William Mcnite. Today our topic is data pipeline engineering strategies sponsored by Informatica from Salesforce. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them by the Q&A section. If you would like to chat with us or chat with each other, we certainly encourage you to do so. And just to note, the Zoom chat defaults to send to just the panelists, but you can absolutely switch that to network with everyone. to open up the Q&A or the chat sections. You'll see the icons for those features in the bottom middle of your screen. As always, we will send a follow-up email within a couple of business days containing links to the slides, a recording of this session, and any additional information requested throughout the webinar. Now, with that, all of our logistics out of the way, let me pass it over to Anchor from Informatica from Salesforce. Anker, hello and welcome. my friend. Please take it away whenever you're ready. >> Thank you, Mark. Let me start sharing Mark. You can see the screen. >> You bet. >> All right. All right. Thank you for joining today. U I want to start with a very simple question. Why does it feel like we are moving more data than ever? Right. And yet it feels uh like not enough. The answer is not about volume. It's about whether that data actually arrives trustworthy on time and ready to use. Great to be here today. I am Ankur. I lead product management at Informatica by Salesforce. I'm passionate about uh helping organization create a trusted data foundation that fuels AI analytics and smarter decision making across their businesses. Right. All right. Let's get started. All right. uh enterprise today are shifting hard towards AI. U roughly about 88% of our organization that we talk to now say that they plan to deploy AI agents and about 60% of them tells us that AI is ready already sits at the center of how they run their businesses today. So we are not talking about a future state that's happening right now. What's really changing underneath is the move from system uh that just execute task to system that reasons, decides and act towards an outcome. Uh that shifts puts enormous pressure on the pipeline feeding those systems because a reasoning agent is only as good as the data part that feeds it. Right? Here's the catch, right? your AI and honestly your analytics too is only as smart as data you feed in right uh companies with the strongest AI ready data foundation uh sees up to 60 65% great business outcome uh revenue cost savings etc focusing on that some foundation can lift AI accuracy up to 80%. uh in the AI era, competitive advantage will not come from having the best model to be very honest. Uh it comes from having the most trusted foundation and building that foundation is exactly the job of a data pipeline engineer. Right? So why does that foundation is hard to build? U if you really ask me uh very honestly data today is fragmented. It's scattered. It's messy. It sits across hundreds of systems, apps, database files, streaming, you name it. Stored in every format imaginable and arriving at every possible speed, left, right, center, batch, microbatch, real time. Uh, and that data somehow has to land somewhere usable for analytics and for AI agents. Right now, that's fragmentation is the number one reason analytics and AI initiative fail to scale. Uh, it's almost never the model that's the problem. the models are generally fine. Uh it's a data pipeline underneath um or the lack of one that's holding everything back. So let's let me give you an example right before I go a little deep. Um two AI agents exact same request exact same model uh the the the the chat that I gave them is close this customer support ticket. One simply closes it. 3 weeks later that customer turns. Now the other agent the models are same by the way but the other agent has the context it knows this issue keeps recurring and this is a top account at rest so it escalates instead of closing right and the renewal is saved same model same request same underlying AI the only difference was context and context isn't is something you buy off the shelf to be very honest it's something your pipeline built ticket by ticket table by table and event by event right So what we call or how do you build that context at scale right? Uh it come downs to we popularly call it data value chain. Uh it maps directly onto what pipeline engineers already do every day. So it's not something that uh is changing. It's just adding another layer. What you do is first you unlock your data u you connect it wherever it lives onremise cloud multicloud uh you understand what you're looking at right and then you build a trust on top of it you validate you protect it and you unified it into a version um that is clean matching and governed right and once all of this is done you finally activate the data you deliver that clean trusted data into analytics and into agents right And under all of this sits a metadata layer that glues it together. Uh data metadata is mostly about data uh uh what it means, where it comes from, how it is used and who can access it. Right now everything that I talked about runs on one single platform and that's IDMC Informatica's intelligent data management cloud. Now, IDMC brings together data engineering, data quality, governance, cataloging, master data management and many more as a unified service on a single platform with metadata as a foundation layer right underneath everything. For a data engineer uh engineering specifically, IDMC is where the pipeline work actually happens. Uh more importantly, it is headless now. So you can build and deploy a pipeline directly in code without waiting on a separate tool or a team handoff right and it runs the same way whether you are doing a batch ETL change data capture streaming or data replication right so IDMC is also increasingly agentic and we have uh AI um uh long before chat GP came into picture we call it CLA uh it's a AI engine that powers IDMC uh it can recommend mapping things catch data qualities issues and help you build pipeline alongside you. So in nutshell, IDMC becomes a single control point for everything, every pipeline that you run, no matter where your data lives, right? >> Anchor, you're at you're at time, buddy. Uh just closing. >> Yeah, thanks bud. >> Okay, so here's bottom line. Every AI project, every analytics dashboard um comes down to one question. Can you trust your data? Right? And that trust is built on one pipeline at a time. get that foundation right and everything else analytics, AI, agents becomes possible. Thank you, Mark. >> Thank you, Anchor. And and um we'll we'll have this slide deck uh sent along uh with everything else and uh and there's lots of good stuff in there to check out. So, I encourage everybody uh watching this after the fact and and attending live to check that out uh um after uh after uh the webinar is concluded. Uh with that now let me introduce to you our main speaker for this series William Mcnite. William has advised many of the world's best known organizations. His strategies formed the information management plan for leading companies in numerous industries. He is a prolific author and popular keynote speaker and trainer. He has performed dozens of benchmarks on leading database data lake streaming and data integration products. William is a global influencer in data warehousing and master data management. and he leads Mcnite Consulting Group which has thrice placed on the Incorporated 5000 list. And with that, let me hand the floor over to William. Hello and welcome, my friend. >> Hello. Thank you, Mark. And thank you, Ankar. Welcome everybody to this month's installment. Um, Ankar, I will see you next week at Dreamforce and maybe I'll see some of uh others of you at Dreamforce. Looking forward to that. So today is going to feel maybe a little more educational than usual because there's a lot to discuss uh in in in kind of retrenching ourselves in good practice around data pipelines because they are so important as Ankar mentioned to AI and to the direction that our enterprises are going. So, let's talk about what it is and how to do it right. And let's remember some of the things that we've done with data pipelines over the years that are still relevant. I like to say that the data platform is kind of separate from the usage of the data, be that BI or AI as the case may be. um it's not quite 100% true but we can think about it and there are that way and there are a lot of us that are going to uh find our way in one side or the other of AI but there must be bridges made across the two of course but we're going to be a little more on this side today on the data platform side and talking about how to make this perhaps chaos that we have created created over the course of years, many of us have into something that is a palatable platform for AI and all the strategic objectives that our companies have for us. Okay, so this is a partial client list. This is where I get my great ideas from. Thank you. And share them back as I go forward. So, lots of uh different industries in here. kind of a focus recently in financial and um uh I would say insurance and healthcare. Yeah. Recently, but we're across the board uh when it comes to data, lots and lots of tools and so forth. Now, let's talk about how we got here. How we got here. We we we started with great intentions, didn't we? When the company was formed, don't they always start with great intentions? We're going to keep everything clean. And doesn't take long when it comes to data for things to kind of go off uh the rails here. And we end up with multiple of this and multiple of that and users here and there and everywhere. Uh and let's just see well what happens here. And this is a microcosm really of of any kind of major enterprise. We add users, we add tools, we add some old cobalt. Yeah, we did that. We do that. Uh we add some functions. Oh, now we got to focus on compliance and audit. Didn't have to do that before. Uh we've occasionally we'll remove some things. That's very hard to do, but occasionally things will go away. And of course, of course, we have the cloud. So we had this movement maybe five, six years ago, everything to the cloud. And now, and we're big cloud advocates here, but a lot of the work we're doing is determining what makes sense to repatriate back to onrem. So, we're doing some of that. So, we have this split between uh the cloud and and onrem and other things too, right? And now we have AI, we have data owners and stewards and user communities all over the place. I don't have to belabor this. I think you get what I'm what I'm saying here that that you have multiple data stores with overlapping data and data at different standards, maybe different quality standards, different granularity standards and so on so forth. If you have this today, you're not alone. You're not alone. So just know that, okay? And feel feel okay about that at least. All right? But we still have to move forward, right? And this has to be our foundation for AI. As much as it doesn't look perfect, there are challenges in this environment where we have complexity, we have context, which is huge right now. We're trying we we've learned that giving AI context and making sure it goes to the right data at the right time and understands what it's looking at is actually quite important. And so the three C's, cost, the other challenge of today's environment, the cost is growing exponentially. I can't tell you how many times we've been called about my co cloud costs are way more than we anticipated. For some reason, we undericipated those cloud costs and now here we are. So we're facing these challenges today, most of us. All right. uh restructuring old expensive systems. Many of the warehouses out there have been in production for two decades or longer. I even just this morning was talking about a client that has old cobalt and has Windows was it 98 or something like that? Uh I think it was 98 in production still. Uh so you know you have different generations of technology involved here as well. They've mutated and grown and they're frequently on products that face an uncertain future. Maybe even some of the ones on the slide. These companies ecosystems need to be reimagined. We need to attend to data architecture. Time to time we have to come back to our data architecture and say we need to actually apply some discipline to what we're doing here. Maybe remove some things, maybe add some things, merge some things. Okay, nobody's ever given me for example a blank sheet of paper and said, "Yeah, we got all this stuff going on, but tell us what tell us what we what we should do." as if we're starting from scratch because nobody can but we have to slowly move what we've got into a proper direction and AI uh Ankar said it AI is just totally dependent upon this totally dependent upon great data but if we give it if we try to give it fragile pipelines data swamps and cost overruns we will not succeed as a company with AI So many people do data pipeline engineering but there is an emerging title I say emerging maybe over the past 5 10 years of data pipeline engineers maybe some of you have that title it's a great title if I were running the shop there would be specialists in data pipeline engineering and they would be called data pipeline engineers but in some shops it's the data architects it's even data modelers that do this it's DBAs which still exist uh and there are you know any number of data related titles that have to do this but in my view this is the dominant function of data in a shop so that being the case I would want to have some real special focus on it and make sure we're exploiting data pipelining to the nth degree and for that I want specialists but anyway Okay, what is a data pipeline? So I'm talking all about this. What is it? It's these seven things from source all the way to consumption. So the source is where we're going to get the data from obviously. Uh this is common input including relational and NoSQL databases, APIs, web services, file systems, and streaming data feeds. Ingestion is the process of incorporating that data. So this is the methods, the schedules and the protocols used to pull or stream that raw data from sources into the pipeline architecture. Then you have transformation. This is where we add data quality. This is where we add data quality. This is where we change data from source to make it more suitable for the uses of the data in the target which may be different than the uses of the data in the source. Therefore, there could be these different quality standards, you might say, for data in different data stores. That's okay. We have to understand the use of the data and go forward uh appropriately. I'm not very frankly I'm not very frequently given the red the green light to go ahead and change data from source to target. Uh so what I will tend to want to do is to bring over the old data and and have alongside it the good new data that is more appropriate for what it's where it's now at like a data lake. Okay. The target is where you're putting the data and uh in a simple world there'd be a very finite set of what these targets are destination repositories like uh cloud data warehouses databases and data links where transform data is stored. However, in a messy enterprise where speed is paramount uh and data architecture sometimes gets the short straw uh that those targets can be anywhere. Those targets can be anywhere. Uh I want them to be good places. Please, if you're asked to do something that's clearly abusive to data long term in your environment, you speak up. you you put put a word in for data architecture or if you are the data architect make sure it's architected well and you have a controlled number of these data targets. Okay, data orchestration is the whole process of flowing the it's the operational control components like workflow automation, dependency management, scheduling triggers and performance monitoring. And then there's the data analysis. What are you going to do with the data? Are you analyzing the data as it moves? So this summarizes of the analytical methods applied to pipeline outputs spanning descriptive, diagnostic, predictive and prescriptive analytics. And finally, the ultimate consumption might be by man, might be by machine these days. So I'm giving you a little definition of data pipeline engineering here. It's the process of designing, building and managing data pipelines that facilitate the movement, transformation, storage of data from various sources, blah blah blah. Um, it's sort of an evolved term from what we used to call data integration, but now we have uh so much of this going on in the enterprise, it's just sort of taken on a life of its own. We call it data pipeline. Here are some uh here are some dimensions of data pipeline engineering ETL. Okay, I I don't need to belabor this. It's extract, transform, load, right? We know the tools. We've we've used them all. Uh probably over the course of time. It's the paradigm by which data is moved. Uh a lot of data is moved. And this is not a mutually exclusive concept, but batch processing is handinhand with ETL, something that is done on a batched up set of data. Large volumes of data processed in batches at scheduled intervals. This is best for use cases where real-time data is not required. However, if you've ever worked with me in terms of launching a data pipeline, I'm going to push us to real time because it's so much more possible today than it ever was in the future. But I acknowledge there are times that batch is still appropriate going forward. Um, as a matter of fact, I would say that the majority of of pipelines are batch processing today still because they're they're still in play from 5 10 years ago and so forth when that was more more common when data wasn't quite as important and we could deal with batch. Okay, there are replication pipelines which is just taking data from one source as soon as it hits that source you pop it up over here somewhere else of course is creating redundant data something that we want to try to minimize but there are times there are times when this is all you need you don't need transformation and this is used for maintaining near realtime replicas of data or to perform the extract load work without the T without the transformation in an ELT pipeline. Perhaps you're replicating because you're going to apply the transformations in the destination in the target. There are aggregation pipelines. This is focused on summarizing or aggregating aggregating large data sets to create summary tables or views. Now I would caution you that so much of the processing going forward wants that detailed data but obviously you have it you have it in your in your source uh data for the pipeline. So a lot of times we'll create a summary pip a summary data store or aggregate some data in a data store because it's going to make that operation perform better because that's all it needs whatever whatever it's doing reporting or whatnot. Okay. So aggregation pipelines and the last concept uh in regards to data pipelines I'm sharing with you today is event driven. when something happens to data in the source that triggers the movement of data through the pipeline to the target. So this is triggered by specific events or changes in the data such as a new record in a database almost sounds like replication there or an API call. Ideal for building applications that respond to specific actions like updating customer data in real time. Did a customer just make a purchase that put them into a new uh category like you know your gold, silver, bronze category, put them in a new category, we should treat them differently immediately or did they just do something that put them in a category that they need to be marketed to in a specific way because maybe they're looking at something online and they need this they need this other popup screen or what have you. Okay, so there are some example tools there for event driven functions. Now, as I mentioned before, I'm going to push you to real time. I'm going to push you to think about real time. Now, if you want that though, you will need to decouple your ingestion engine from the downstream target. So, you decouple the two things. uh you this is a way of transitioning the architecture from fragile tightly coupled batch ETL to resilient eventdriven streaming by placing a central message Q I like to say it's postman uh between the incoming data streams and the downstream consumers. This isolates the ingestion layer from compute with storage enabling independent scaling where resources expand only where bottlenecks occur rather than across an entire monolith. So you have you actually don't have direct pipelines source to target. It's more like you have subscriptions by various targets to various topics. So as that data lands in the queue, you're really decoupling the source and the target in that way. And so in this way you your your targets can be picking up data from wherever it is created in the enterprise that is appropriate to the target. So I'll get into tools here in a bit, but there are a lot of tools that are geared around this idea. This is the idea. So I'm going to talk about this a little bit more. Data stream processing. So when when do you need data stream processing? Other than William says do it all the time. When do you really need it? Well, traditional integration is insufficient for this combination right here. Data platforms operating at an enterprisewide scale. Okay, so it's big. a high variety of data sources, real-time streaming, pipeline forces, either real time loading without being scalable or scalability with batch load. Hope that makes sense. Data produced from numerous sources is a torn or flowing information needing to be timestamped, dispatched or even duplicated to prevent against data loss. And a postman is needed to distribute data from message senders to receivers. Now if you know more about kind of the paradigm here, you can apply it to your situation and understand if it makes sense for the pipeline that you are building. Now do you have real- time data? Real time data is is flowing continuously. It's being created continuously. It's like sensor reads or or the click stream or or the phone calls if you're a telecom etc. Messaging. This is also called messaging, live feeds, real time uh and event driven. So these these terms all kind of flow together. But this this data comes in continuously and often quickly and often 24 by7. This data needs special attention and can be of immense value but only if we are alerted in time. And I would say only in the past five years or so have we grown our capabilities in enterprises to really deal with this data. Not only from a storage and access perspective, but what do we do with it? You know, how do we improve the business based upon the click stream or the sensor reads so forth? Well, we figured that out the large and now we just need to get the data. And so this type of data is appropriate for data stream processing. When you are engineered for data stream processing, you have these concepts that you have to keep in mind. The latency of data we do, by the way, we do we've done a ton of benchmarks in this area. So we have numbers uh behind these uh these concepts. Evaluate the acceptable latency for data processing as this will determine the choice of stream processing engines and data storage solutions. That's the other part of this. What is important here to you? Is it performance through latency? Is it scalability? Is it fault tolerance or data consistency? The things on here and that will help you determine what the right tool is for you to uh get for this. Um that's what I will say about Okay. Oh, I do want to point out message delivery semantics deciding between at most once, at least once or exactly once message delivery. So these are uh different methods of delivery that balance throughput, latency and data loss. Okay, if you cannot tolerate any data loss, you want exactly once. You don't want at most once. You don't want at least once. You want exactly once. This is the balance point between throughput, latency, and data loss. So you have to uh some of you I'll say some of your pipelines need to understand this distinction and make an appropriate choice based upon this element of the tool. Not everyone. It's usually good enough. Uh only one time have I experienced this being a problem in a production pipeline. But if you cannot tolerate uh loss, okay, there are solutions for that. Now enter message orienting middleware aka streaming and message queuing technology. These messages can be anything. This is these are intelligent platforms for fast data. Connect, process, and store data in real time in a unified flexible solution able to meet demanding SLAs's at scale without operational burdens and complexity. So these are messages that are sent by what we call producers towards a broker and the broker brokers the information to the consumer. Simply put, here's another slide that graphically illustrates this. You got all these apps that that are producing data streaming uh the streaming platform in the middle kind of in the middle conceptually and it distributes the data to mostly warehouses and lakes um other things as well like I said before back into back into other various apps and so on. Everyone's doing crazy things and everyone's got a spaghetti architecture doing things differently. So the streaming platform in the middle acts as a central nervous system for organizational data flows. This is a very different concept than ETL than a hard-coded pipeline. And I'm just reinforcing this in different ways. So there is a concept that I also want you to be aware of having looked at these tools in depth. One of them is uh one of them very important is the egress leak. The egress leak in cloud computing a company's private network the VPN and the cloud providers they charge fees these are egress tolls some of you are some of you are well aware of this it's become a problem into your shop whenever data travels outside of that boundary so under a legacy software as a service model continuous data streams must leave the enterprise VPC to reach a remote vendor's cloud for processing creating compounding egress costs network latency and increased security exposure as data crosses external perimeters. In contrast, a bring your own cloud deployment installs a streaming engine directly within the customer's VPC. So that ensures that data never crosses the external cloud boundaries which eliminates your egress fees and I think that's pretty important. So that's something to look for. um the legacy in other words the legacy will force the data out of the customer's VPC whereas the BYOC keeps both the broker compute and the data plane entirely within the VPC it's a beautiful thing now Apache Kafka because this is this is my this is the thing that we all know about so I'm going to bring it up here I'm not saying I'm not saying Kafka is appropriate for everything I'm not use open- source Kafka uh everywhere uh but uh it is well known it is used quite a bit it is an open source streaming platform developed at LinkedIn long time only I think it was around 2002 something like that uh so it's been around a while it's a distributed public subscribed messaging system that maintains feeds of messages which we call topics the publishers write data to topics and subscribers read from the topics. This enables the source to sync data pipelines and these messages are usually simple some number of bite long arrays that can store objects in virtually any format and here you see some of the related tools that you need if you're going to use Apache Kafka. Now there are plenty I shouldn't say plenty there are several other tools that do this function maybe differently. There is one that is a C++ drop in replacement for Kafka. There is uh open source, there's Pulsar, there's a Jetream, Knaps Jetstream. Um they don't all use Kafka. Some do. Some have just something that's very similar and that's okay. So you can look at all of that. Now when it comes to operations uh for your data pipeline, these are the things to keep in mind. orchestration which is coordinating and automating pipeline tasks and workflows. There's the monitoring and then there's observability getting deep insights into pipeline behavior, data lineage and root cause analysis. So there are tools that do some of these things that are probably different tools for you than the data pipeline engineering tools like the Kofkas and so on that we just talked about. Another concept that's very important here is automated remediation and self-healing pipelines. This is something that has grown in the past few years. transitioning data operations from reactive alerts to proactive ML driven interventions through self-healing pipelines. So the core mechanism is the dynamic autoreovery loop which is driven by continuously observing telemetry to automate policy adjustments supported by the critical safety controls that you see here. circuit breaker pattern that halts calls to failing data sources to prevent compute exhaustion and cascading crashes. And then there's anomaly quarantine powered by ML models that deferred distribution drifts into isolation for human review without stalling the main data flow so it can continue. And then there's fallback hashing to serve the last verified good data state and preserve operational uptime while the repairs occur to the pipeline. So automated remediation uh something that is taking cycles out of the whole engineering effort for data pipelines. And indeed in indeed these pipelines have seen a lot of of effort uh taken out of them but for whatever reason even that being the case they're still pretty errorprone. They're still prone to a lot of misuse and um overdoing pipelines, meaning drawing in way too much data, drawing in the wrong data. A lot of problems are still occurring in this area. That's why it's important to retrench in these practices and remember all the different styles of data pipelines and all the things that we can do with data pipelines, which is what I'm bringing you today. Now about two years ago we uh at the firm we did a fullscale study of data pipeline tools across the market. I think we looked at about 12 10 to 12 tools and um so we put a lot of thought into what are the capabilities of pipeline tools that are important and these are them and they're still true. connectivity and multilatency. Uh this provides universal source integration across streaming, CDC, microbatching and reverse ETL data transformation. What is the richness of the set of transformations that are built in that you can just call? How many data quality violations can you fix with the transformation set that is in the tool? orchestration and workflow which automates the execution via event triggers, complex DAG scheduling, unified visibility and backlog aware SLA management, enterprise scale and performance. Yes, how much throughut can we get in this tool? Um through dynamic linear autoscaling. So scaling up as appropriate to for for high value workloads or longunning workloads. Adaptive query optimization and strict multi-tenant resource isolation to eliminate noisy neighbor bottlenecks. Now which of these is most important? I'm going to get to that. I'm not going to leave you hanging on that. Security, governance, and PHOPS is the other area that we have four different broad categories that you'll want to look at. Feel free to screenshot this when when you get the slides. Uh pull that into your evaluations. So security and access control uh which is protecting data assets using granular arbback uh permissions, automated dynamic masking, KMS managed field level encryption and active DLP quality and governance uh cataloging and lineage. Lineage has become really important lately for all of the regulations that we're trying to follow. PHOPS and compliance which balances operational rigor and efficiency via query cost attribution auto suspending compute multicloud portability zero downtime high availability and built-in regulatory regulatory compliance. Now, if you want this report, by the way, just hit me up. I'm happy to send you the report. It's a couple years old, but I think it's still pretty relevant. Finally, we get to analytics, automation, and AI capabilities. I want to see AI capabilities in there. Uh, I want to see leveraging selfhealing pipelines through AI for handling those timeouts and schema shifts, promptbased NLP pipeline synthesis, automated ML discovery of join paths and entity relationships. And finally, I want seamless integration with BI and analytics tools. Yes, they're still out there, but still interesting. Tableau, PowerBI, Looker, Hotspot, what have you. and a high-speed caching layer as part of that uh which ensures subsecond interactive ad hoc queries and centralized semantic layer synchronization across these tools. Now what's important here? This is what we gave a lot of thought to and came up with and used when we ranked all of the tools in the market for this. And you can agree or disagree of course for your situation. These are our recommendations though. Highway categories are the 15enters. The differentiators native connectivity and multilatency in in ingestion. Without a lot of that you're it's going to bring your bring your whole operation down and and take your workload up quite a bit. So we we call that out. analytics, automation and AI, Gen AI integration, automated pipeline generation and self-healing features that drive higher ROI and future proof the investment. And finally, data cataloging and metadata management is the third really important category in here. In modern architectures, the pipeline tool cannot be an isolated black box. Metadata sharing, data discovery, and lineage across tools are critical for data mesh and unified architectures. So that's data pipeline engineering, the basics of it. Anyway, what are some recommendations here in terms of good practices going forward as you build out your pipelines or if the case may be going back and remediating some of the pipelines that you have designing for modularity and scalability. Okay. Number one, break down the pipeline into smaller reusable components that can be independently developed, tested, and deployed. Don't build this huge pipeline that um it's very that anything can bring it down to its knees. Okay? You know what I mean? Create more smaller pipelines, thinner pipelines that if something goes wrong, the rest of the operation continues to function. So this is in your ingestion, transformation, and storage functions. Isolate your failures. Make it easier to update specific parts. Build a build it scalable horizontally so that you can add more nodes or instances vertically so that you can enhance the resource on a single instance. Both ways are ways that you may want to scale up your pipeline. And so make sure that you're able to do that, you're ready to do that in the event that new pipeline requirements come up today. Data partitioning. Okay, this is a great thing you can do on the data side to facilitate great pipeline engineering. Partitioning the data, partition those large data sets to optimize performance and make the processing more efficient. Next, and I've alluded to this, design for real time needs. You may not have real time needs today. That doesn't mean you won't have them in the future. hybrid data processing which is both real time and batch. Consider both when it comes to your pipelines. Real time streaming for immediate insights. Batch processing for periodic reports which is becoming a smaller part of the overall pie. I didn't say it was shrinking. I said it's becoming a smaller part of the overall pie which is growing. To be sure, data pipeline engineering is growing and the need for pipelines is growing. Despite the fact that as time goes on, theoretically, we would get better at data architecture and have fewer source, excuse me, destination data source data targets. Uh that is just simply not the case. Uh we still have uh quite a crazy mix of things in the environment. As I mentioned before, event driven processing, that's a that's a good practice. Use event- driven architectures for real- time data ingestion and processing to react to data changes immediately and then do the right thing in the architecture. Make sure the data is where it needs to be in order to do the right next best action. Uh whether it's an AI agent or otherwise focus on data quality and validation. You know, I've heard so much over the years, less lately, but I still hear it. Garbage in, garbage out. garbage in garbage. They give me garbage data, they're going to get it on this end. That's not good enough. We have to be sure as data pipeline engineers that we are delivering data into in the pipeline that is appropriate for use that is appropriate for use and adheres to the quality standards of the use of that data. Sometimes operational data is uh very difficult to to change. It's difficult to change those operations that create bad operational data. We just get that data. But we have to look at it. We can't just pass it along as if we're building widgets. We have to look at that data and ensure that is met. It meets the standard. Now hopefully you have data governance in place that helps you as the engineer determine what those standards are. That is really a huge purpose of data governance in a shop and it's one of the first things that I'm going to want to develop if it's not there already if I'm asked to develop pipelines in a shop for sure. Uh focus on data quality also means schema validation. I had a long talk this morning about schemas and the importance of modeling still even with with AI. Now the good news is that a lot of the tools you know we're talking about ER studio Irwin these are the main tools right um what is the what is the SAP one that's getting getting uh getting u sunsetted next next year it's still out there quite quite a plenty drop it in the comment if you know what I'm talking about but um there's a there's a finite set of these tools but they are getting um enhanced with AI such that you can build your data models now through conversation with a headless conversation about what it is that you want to see in the model which is a beautiful and wonderful thing uh over the next year this is going to become more and more automated uh those schema cells they still need to be validated okay even though they're created out of great conversation I heard you can drop a an email structured requirement into these tools and that will reflect in the model immediately. Wow, I'm I'm going to see that soon. And uh and as as someone who's done a lot of data modeling over the years, uh I'm looking forward to seeing that and thinking about hm what does this really mean for data modeling going forward? Anyway, modeling is a large part of data quality. Without a great schema, you're you're not going to have great data. uh this it's it's it's it can be very inongruent. So we want to build great schemas implement error handling and monitoring logging those errors automated retry mechanisms like I mentioned before part of the whole automated handsoff process within your pipeline of fixing itself and then monitoring and observability using monitoring tools tracking meth metrics such as data processing times latency throughput error rates okay tracking these things so you can Are they going in a good direction or are they going in a bad direction and should we intervene now versus later when it's a real problem? These are the stats that you need to have about it. Design with data lineage and metadata management because they are becoming so important so important to all sorts of regulations knowing where the data comes from. So implementing tools or frameworks that track the flow of data throughout the pipeline including including transformations and data movements, debugging, auditing and understanding data dependencies and then metadata management. Use a metadata catalog to keep track of the data schemas sources and transformations. And uh this is this is starting to take hold out there finally. Uh we're starting to see cataloges, data cataloges as part of the basic stack. Power designer, by the way, is the name of that tool. Some of you probably beat me to it in the in the comments there, I see. But anyway, optimize for cost and performance. Cost is very important these days. for the past few years it sort of drifted from consciousness and uh I wasn't being I know I wasn't being asked to do as many as many um ROI evaluations as I used to which it used to be a staple of our practice and I was kind of a go-to resource for determining you know what the ROI is of certain practices around data in an organization. Well, it's come back. has come back and the reason it's come back is because of the escalating costs that I mentioned before. So, it's important to make sure that our pipelines, even though we're creating a pipeline down here, eventually we have to take our focus down to the pipeline that we're creating, somebody needs to keep the focus on the bigger picture in the organization and the number of pipelines that are being created and the fact that we're moving data all over the place and where can we optimize. I really think that I really think that every shop needs that focus on what's going on in the shop. Call it data architecture uh data architecture committee what have you but that focus needs to be there and that will help probably more than optimizing the pipeline in terms of overall cost. But in terms of the pipeline you're looking at things like resource optimization, data compression and even data retention policies. data retention policies. It's a two-edged sword. Sometimes you want to keep data for a long time. You all you have to do there is look at the cost, make sure you're you're okay with it. Sometimes you you actually want to get rid of it after a certain number of years. So follow the policy in terms of that. Adopt continuous integration and development for your data pipelines. These are not one and done. Never one and done. Continuous integration means you're using version control systems like git is very popular to manage code and configuration changes. You're integrating automated testing to validate changes before they are merged into the main branch. So this whole um es uh publication process I would say for these pipelines that's a whole focus. If you're in a large shop you need a focus on that. You need to be sure you're doing regression testing on every change and then you're promoting then you're promoting it to QA and onto production. But regression testing is an important part of continuous integration, continuous deployment. Uh gone are the days when when we had uh you know u every uh I'm I'm I'm recalling a specific situation. Every three months we did a we did a promotion to production and and here's here's the cut off date. You know, get your get your enhancements in by this date and it goes on this day. That's no more. I don't know anybody that does it like that anymore. It's it's a continuous flow. It's happening tonight. You know, that's more like it. Automating the development the deployment of pipeline components to different environments to ensure consistency and reduce human errors. And then automated testing, all that regression testing I talked about, creating unit tests specific for whatever it is that you're building the pipeline around, putting some putting some data quality checks in there. And a lot of times people will say, "Well, what do you mean data quality checks? We're doing data pipeline engineering here. We're moving data." Well, as I said before, you're responsible for the data quality that goes through that pipeline and ends up in the data target and make you got to make sure it's fit for purpose. All right, moving right along here. Fostering collaboration and a data ops culture. Don't be insular in the development of this. This is not data pipeline engineering is not a strict technical function. You need to get the business involved. You need to get the uh the stakeholders involved at both ends of the pipeline, especially the target. How do you know that you're moving the right data if they're not involved in this? To me, this all gets back to a data governance program. Very helpful in this. And then for whatever you do, you educate the organization around it through documentation and training. Prioritize data security and compliance. Never let raw PII or PHI data land in plain text in the raw data lake. Use short-lived rotated credentials or workload identity federation rather than storing static API keys or god forbid database passwords in pipeline configurations. And then you can also use tags like confidentiality equals high or geography equals EU whatever that however you want to distinguish up the uh the security. You can use that uh to enforce dynamic access policies across the analytical engines and then apply masking policies directly within the query engine or the table format later layer like using Unity catalog snowflake tags apache ranger. So raw files stay on touch while unprivileged queries automatically see redacted hashes or partial string. Okay, you got to consider that as well. So your AI ready blueprint and I always start at the edge. We don't all start at the edge but starting even out there at the edge if you have it all the way through the human and AI intelligence node. execute the schema validation, the dduplication and the initial event enrichment on the edge nodes or wherever the data starts. So only clean compact records hit the centralized lakehouse compute cache high frequency operational reads in local memory. Um again back to the edge might be Reddus or something like that out there. while you're streaming change events asynchronously to open lakehouse formats like iceberg or delta for heavy analytical queries. So you want to route the transformations within the same cloud region or availability zone as your object store be before you move the data to avoid inter region network egress penalties. A lot of things to consider there. It sounds so simple in concept, right? You just move data from here to there. Well, I've given you today that uh there are many dimensions to that. There can be complications to it as well and there are a lot of things to consider as we deal with this very important asset of the organization and to where the organization is going with AI. So in summary, these are some of the main points. Um modern architecture evolves from rigid batch ETL decoupled the ingestion. Okay, automated self-healing. That may be a concept that you weren't thinking too much about coming in, but you need to think about that as well. Governance and PHOPS. I've talked about the importance of those AI and BI accelerated acceleration using semantic layers for sub-second dashboards and contextaware AI agents. We got to get the data there a lot of times now more than ever in real time. And then platform evaluation. I've given you six dimensions. how no more than that probably like eight dimensions to it. And I prioritize these three for you when you go look at your platforms for this native connectivity and multilatency data ingestion, analytics, automation and AI and data cataloging and metadata management. And finally, don't do it in an insular fashion. Make sure the enterprise is ready. Design robust architectures that minimize downtime. So don't just stay everybody in the shop. Stay focused on pipelines, one-off pipelines. The bigger picture must continually be looked at to make sure that the whole operation is efficient. And with that, uh, brings me to the end of the formal part. And I hope I've saved a few minutes here for questions. Mark, >> well, we had two nice and and quick ones. And then somebody asked a great question that I think we should just do a whole other webinar on in the future. Uh the first question I got by a direct message uh is for Anchor. Um uh where can people go to learn more about the IDMC? Uh so do you want to give a brief uh answer to to that? >> Sure Mark. So typically we don't provide uh sandboxes. I I think that was one of the question I received a message as well. But we have lot of learning material on YouTube and even on our website. The best place is start with your account rep. He'll be able to guide you if you want to do a PC, quick check, everything is through account reps at Informatica. >> Awesome. Thank you so much. Um, we had somebody in the Q&A panel uh William that asked a question that I would have asked as well. Uh, I'm curious to know the need for designing in real time. Uh, traditional businesses that don't need data in real time uh for like monthly billing statements. Uh, for me it was like quarterly KPIs and things. Uh, why overengineer if you don't have to? And I I suspect I know your answer to this, William. Uh, but I'm I'm interested to to to hear your thoughts. >> Well, if you if you're if you're creating a data store for that specific purpose of monthin reporting or what have you, um, okay, I could definitely see that. But great architecture means you're building something once and you're using it for many things. You're you're you're you're you're piling on the uses here. So you don't have to duplicate the data all over the shop in in perhaps different profiles in different ways. And so with that in mind, the month-end reporting is is not all that this data is going to be used for. As a matter of fact, of course, I know the first one, two or three, you know, uses of data for any target that I build, but I don't know the fourth, fifth, sixth, and and the eventual, you know, need for that. And you mentioned overgineering, but that may have been true a few years ago. Now, >> I'm not sure is true anymore. I think it's it's almost just as easy to do real time than it is to do traditional batch. >> Yeah, it used to be so expensive, too. And I I don't think that's true anymore. >> Um Yeah. >> Yep. Uh well, we're we're at time. Uh and we're getting lots of questions in now. It's so unfair. But thank you everybody for uh the wonderful engagement in chat and in Q&A. And uh Anchor, I'll give you the final word. Any last thoughts before I hit the end webinar button my friend? >> No, thank you Mark. I mean great a punch today and good to good talk uh William. I mean I echo the sentiments and data pipeline is uh good about uh as good as your data model right. So thank you. >> Have a wonderful day everybody.