Data Quality & Observability by Chris Mucheka

to join the session. Um I guess other people will also join or they were here and they left because we started late but they will catch us as we continue. Please also uh make sure that you engage with the presentation and ask questions if there are any questions that you want to ask um uh Chris. And before we even start, I want to also read Chris's u and introduce you to Chris and read his bio. Um, okay. Just give me a second to pull it up. Okay. So Christopher Macheka is a data engineer with experience from fintech, retail and technology startup. He has worked as a data scientist at Sirita Insight, a data analyst at Shopright, a data engineer um at P and Pay and and Oni. He is skilled in Python, SQL, Pispark, AWS, Snowflake, modern ELT tools. He also specializes in building scalable data pipelines, ensuring data quality and delivering analytics ready data set. Chris is also passionate about he's passionate about mentoring and he also helps teams adopt best practices in data engineering and obser observability to drive trusted datadriven decision making. And ladies and gentlemen, let's welcome Chris as he talks to us about data quality and observability. Over to you Chris. Uh thank you very much uh Elizabeth. Um so uh today we are going to talk about uh data quality and obser observability uh in action. So um so I'm uh as as as uh Elizabeth has done uh the uh uh introduction about me. I'm a data engineer. Uh currently working at at an oni which is a fintech uh startup company. Uh I have various uh experience uh in different roles data scientist data analytics and currently did engineer. So my path into data what uh it's actually started uh in a way that I had not planned uh when I finished my um metric. Um apparently I wanted to do uh pathology but then yeah God had his own plans. So uh I had applied for only pathology and I didn't get any place uh at D and UK Zen. Then uh when January came I didn't have any place then I went to water culu with my results as a walk-in then when I went there I was like looking for any course then they saw my results they were like ah you did well in mathematics so there is space in BC mathematics then I went there then I did register so when I registered I had majors with maths and statistics uh then uh some electives which were like computer science um that's where I grew uh the love of technology uh programming and statistics so with those skills uh I wanted to actually solve real world problem using uh uh my programming skills and also statistics so that's how I ended up being in uh data but then when I was in data there are different roles that are there in data so uh I started working as a data scientist ated insights where I was actually building some models uh machine learning models. Um then I uh when I finished working on the project that I was working on I then went to show PR as a data analyst where I did a lot of analytic works. Um while I was doing analytics work um there was some data engineering thing that I started working on. uh that's where I actually fell in love with D engineering. So now I'm into data engineering. So uh enough about me. So now uh about uh the data quality um in each and every organization uh they have data they have information that is stored uh and that they want to use or utilize to make decision. So you for you to for them to actually use that data they must actually have uh trust in the data. So why data quality matters is because it matters because uh each and every dashboard uh which actually give uh business stakeholders in insights uh each and every AI model that you would want to use uh and every decision relies on this data. So this data must be of high quality uh because um a lot is on stake. Uh each end each decision that is made based on the insight based on this data. Uh so the it must be of high uh high high quality and um if the the data is poor then uh the decisions uh which are made are not uh going to direct people into the uh right direction. So the uh the business is at risk of making poor decision or bad decision which will actually impact in terms of its growth, revenue uh and profits. Uh I'll give an example because I've been working in a retail. Um so yeah in the retail uh space we use a lot of data in terms of like uh promotions, campaigns, um who to target um uh at what price would they sell uh their their their products. So if we have wrong data uh for example uh uh there is data about people who purchases uh pumpers right then if our data is not of quality uh if it's missing uh key features and key fields or it's if it's actually incorrect then it means that we are likely going to promote our or products to the wrong people, you end up promoting to people who don't have kids in terms of like if it's pumpers or people who have actually uh who don't have kids or who are not even planning to have kids. So that promotion that they have already that that they will do or that they will run based on the poor quality of of data will actually cost them a lot. Um then in terms of uh the data uh I mean data uh at this point or in this age uh a age it's uh it's it's very important uh we can say data it's actually gold and as you know people are in in the mining um uh gold it's uh it goes through stages to be purified to be cleansed and refined so that It's of higher quality just the same as our our data. It must be purified. It must be cleansed. It must be refined so that we actually have uh the quality of of data that is of higher standards. So, so we have some core principle of actually data quality and observability. So these are the core principle of actually checking our data quality. So we have accuracy where we actually checking how accurate is our data. Uh does it reflect reality? Uh I always give example of uh customer data like in the retail spaces. Uh do you see something whereby you find a a customer with an age of thousand? Uh is it accurate to have someone who is 1,00 years old uh or someone who have um a um who is with uh zero years of uh with zero years of age. Uh so those kind of things that uh people actually do when you are doing uh your data quality checkups. Is it complete? Uh so whenever you're building data pipelines in databases you have to the are required fields we have uh things like primary keys uh foreign keys. So these fields that are that really matters to to the business uh they will have to be checked if they actually when when when a person for example when you're registering a member did this person actually have all these fields present. So does is our our data complete? Then in terms of consistent uh does the same values across systems uh uh for for example you you find uh in some case uh for gender uh you you have male you have females then next thing you find uh integers or double or double is it is it consistent in terms of how we actually capturing our data. Then uh you also have to check uh timeliness. Uh is our data up to date? Uh you don't want to rely on data that is not up to date. Um for example, a data that was last uh updated in 2015. Uh can you can you can you trust those data pipeline that is not being updated? uh if it's for uh member datas or customers that haven't updated their uh phone numbers or their addresses, are you going to uh are you going to to trust the data that hasn't been uh up to date? Then in terms of trustworthiness and uh governancy and ownership, this is where you'll be like checking uh who owns this data, where is it coming from, is it being uh kept uh in a manner that no one can actually make any changes to that. So if the data is not secured, it's accessible to anyone and anyone can make any changes. How can you trust such data that anyone can actually make and uh make any changes? Then we we also come to observability which is like where you actually checking your anomalies is your lineage and flows how the data is actually being moved from one place to another. Is there any anomalies? Uh something that I've mentioned in terms of accuracy where you find someone who has uh a member or a person that has age of thousand or age of zero in in a data of people who are purchasing uh things uh or item or products uh and the retail spaces. So these core principle uh principles of data quality and observability we have to make sure that we we we we observe these and cater for these when you're actually building your your your data pipelines. Then uh then we move we move uh to the best practices and it's either Azour pipelines or AWS uh pipelines. So in in the data engineering uh uh space where you actually writing uh it uh pipelines that actually ingest data. So when you are ingesting uh data you have to kind of like look and validate the data before you actually load it somewhere else. You don't want to load something uh data then you start uh investigating. So these are the the the practices that you have to make sure that you have in your pipelines when you're actually building your pipeline that before you ingest your data did you validate did you check the schema uh are the fields that are required there um is it stored in the right format um you check at uh at like uh null values duplicate. So this will help you to actually prevent your uh prevent errors when you're now loading your data into like your databases, your data warehouse whereby you find your your your your data pipelines they are failing because probably there's a no value on on on primary cues or on values that are not supposed to be null which you should have catched if before you actually ingest the data. Then um the other uh practice is to make sure that your column mapping is consistent in terms of types. uh the names uh when you're talking about types, you're talking about data types which is like integer, double, string and make sure that each and every um field that you are going to ingest uh in your pipelines they actually mapping correctly to uh the correct fields and the data type and this actually helps in terms of like uh solving the issues of schemas whereby you you uh a a a field that is mapped to a wrong uh uh field, you will be able to to to to to capture this and to notice this before you even run your your your data uh pipelines. Then we have to to look at also the transformation and match statements. Uh this is like where you're now doing your updates or your inserts into uh different t uh table to make sure that each and every statement that is uh that that is going to be updated before that whatever transformation uh that has been done it's actually doing what's supposed to do in terms of what what's supposed to be updated and insert. You don't have to insert uh data in the wrong tables or where it's not supposed to be uh inserted. Then we move into to error handling. How do you handle all these issues uh now that you refined in terms of uh quarantine the bait data uh or is are you going to just ignore it? Are you going to handle it? So the best way is to handle whatever errors that you find so that your pipelines they don't just fail um and you don't have any way of actually moving data even if there is some issues. I'm going to present uh um I'm going to present uh a a hands on on this error handling and quarantining the B data. Then uh the other best practice is to or automate uh data profiling. So what we mean by uh automate data profiling is where you don't have to do this manually because when whenever there's manual work or human interventions that's where a lot of things are being missed or a lot of um mistakes are actually uh done. Then at the end what you would want to do whenever you are you're building your pipelines is to have a monitoring dashboard or a list that you actually receive. It's either through the Slack or Teams or email. So, so that you know when you are running your your your your data pipelines that there is some issues that are that are happening uh on on your database other than to wait up to wait and hear from stakeholders that is something that is not making uh sense. Um at this point I just want to u take a break and ask if there's any questions that you would want to ask. Okay. Um then moving to the emergence emerging threats and advanc uh advanced considerations. So with with data if you're now having uh wrong in uh wrong inputs or um data that is incorrect that's that's kind of like poisoning your data or making your data to be dated. So they say there is a say that say garbage in garbage out. Um imagine uh you uh uh in a scenario where chbt is being given incorrect inputs. Uh when you're talking about malicious, we're talking about data that is uh uh that is dangerous for for for a AI to be exposed to in terms of like um suicidal thoughts. So it end up uh let me say uh advising probably underageed kids or advising people wrong advices because it have been exposed to incorrect data uh or have actually been uh given data that is uh malicious. So this as you can see the way uh these these colors are it actually shows how dangerous it is or how how those threats um how how how those threats are actually uh emerging in in this age. So moving down here we have uh extended observability to data supply chains. So here is whereby the data supply chains is not observed uh and we we not cleaning our data we just allowing each and every data to come as true as it is. This can be things like um false news where you you you train your data or your your your models based on data supply that is not being uh that isn't being double checked. Then um we also have to consider things like uh providing tools and controls. So these are the the controls that uh I had mentioned that you have to check um your how accurate your data is how how complete it is. So these are the the the the the measures that you have to to talk uh to to to to actually put in place because if you don't then they're going to be like a lot of threads in terms of how your data is. Um then we have business uh defining uh uh the quality standards. So business should each and every business have its uh its rules. Um so defining those standards and rules and put them in place will actually help us to to overcome the the threats. If the those uh standards or rules are not put in place or they actually incorrect then uh we're going to have uh threats and they going to poison our data. Then the other is everyone should protect and use uh data responsibility. So whatever data you will come ac across uh you should you should protect it uh the is poppy act um if it's not protected and you you you're not using it uh responsibility responsible then you the chances of you being sued uh or the chances of the business getting sued are very high. Uh so these are the threats uh and the emerging threats that are actually coming in with the uh data and AI uh area that we are in. So uh at the moment I'm just going to share some of uh uh the I'm just going to share some of the uh data pipelines and how you actually uh enable uh data quality in terms of how you you build your your data pipelines, how you observe your your data qualities. So uh just going to switch my I'm going to share my screen so that we can go into some practical example. If there's any questions you can ask while I'm preparing this, >> there was one hand uh while you were talking. I don't know. Dang, do you still want to ask your question? Can you hear me? >> Yes. >> Yes, we can hear you. >> So, you you keep on mentioning data pipelines. I just want to understand what what is a pipeline? What is a data pipeline? I'm not sure if you explained this in the beginning, but I just want to understand that. >> Okay, cool. Um um I'm going to um so I'm going to explain it uh now. Uh I'm going to show it. So can you see the screen of my web that I'm actually sharing? Apparently I can't see what I'm sharing. >> Yes. >> Okay. So um uh this is uh yeah dbt uh that is a tool that is used to to actually build uh data pipelines. Uh that's the question that you asked. So yes we currently see the presentation that all right display. Okay, let me see. Okay, can you see now? >> Yes. >> Okay, awesome. Um, so this is uh DPT uh a tool that is used to to actually build data pipeline. So a data pipeline it's kind of like a code that you write to move data from one place to another. So um there are different ways or different tools that you can actually use to to to build your data pipelines. Um so um so in in DPTA the way you build your pipelines it's actually through uh models they call uh the models. So for example u uh so this is what we can actually call a data pipeline where you you have your source data that's being ingested or being uh um transformed through these models that you can see here. So this whole thing it's we can call it a data pipeline. uh in in DBT they call it lineage uh which is also like yeah a data pipeline so you have your source data that get transformed you uh when you're talking about transformation like either like we're talking about ETR where you extract your data you transform it then you load it somewhere else or in this case this is kind of like ELT where you extract your data uh trans uh you extract your data you load it and then you're transforming it. So DBT it enable um uh ELT instead of ETL where you extract load then transform your data. So uh so uh this is uh customer and uh order uh order data. So, uh, I'm just going to go to stuff like where my data is. So, from uh the snowflake, this is a data warehouse where I'm actually storing my data. Um then this is the so if you see uh where I was actually creating my data uh so we have like some um customer data here. So we have three rows of data. So if you look at the emos that you have, this is not a valid email. So when you're talking about data quality, we we want to ensure that the data that we have is actually uh as I said, it's actually accurate. It's uh consistent. Um so this person if the business really uh value uh members or validate members based on the emails then we shouldn't be able to push this data in our data pip in our data warehouse because of what? because we don't have uh a correct or a a valid uh a valid email address. So when you're building now your your data pipelines, you must make sure that you you cater or you check if there is certain issues like this for instance this person uh or this uh customer didn't have a an email address. So I'm just going to check show you two ways of actually how uh people actually build uh their data pipelines. So I'm going to start with this one which is uh our staging uh customer. So this you in this way we actually build a data pipeline where we actually pulling all our raw data here. then uh create a view then we'll push it into the st staging. So here we actually cleaning kind of like our data in terms of like uh we we want our data to be virtual we we want to trim and remove uh excess spaces. Um and this is kind of like just cleaning your your your your data. So, so in this uh in this uh way of uh building a data pipeline, it's it's actually trying to remove or make sure that everyone have what have an email that is that is valid. So, I'm just going to uh run this so that you see what will happen when I run this. So uh on dbt when you're building your or running your uh data pipelines you click on build then you need to uh run the whole uh uh uh pipeline. So you see here it actually fails. Why is it failing? It's failing because there is data that is that is not correct. Um so the way I had built this is like okay so if there's any issues with our data it must show that there is something wrong with our data then this error now will actually tell you oh we have an error then you will be able to watch to to solve it but then there are issues where by here it's actually giving you an error what if the data pipeline was actually running uh at 2:00 a.m. in at night. Um and probably you are not going to be there at work after 9. So how is it going to be um for for for business uh stakeholders that likely going to use uh for for instance uh in this me just maybe uh since this is the staging customers there is fact orders which is supposed to actually have data that have failed here. So which means our data here is going to be what it's not going to be up to date uh in the morning and probably there are some decision that needs to be made uh based on that. So to to solve issues like this, we actually have this is uh the the best way to to actually build your data pipelines where you have your your your your clean data uh your your your model. Then in your model you have another one that actually does what that actually runs and actually store all those information or or the data that have like issues. So here you're checking okay uh everything we don't want anyone that um uh if if if an email is it's actually no and we we have an email that is that doesn't have the uh correct uh format then we're going to store this data then at later stages now you'll be able to investigate and make sure and investigate in terms of like where is this data coming from? Uh is it probably someone who was trying to hack or is actually the process of actually emboding new members is actually wrong. So these are the issues or the way you can actually um uh use to to to to actually um make sure that the data that you you have is actually uh incorrect. Um I'm not sure if you understand what I'm trying to show here. Is there >> I I have a question please. So so when it comes to the error handling data what happens for instances where say column A depends on well rather row B depends on row A. However, row A failed and row B succeeded because we are only um loading data into a different environment. Do we then also just create one uh pipeline that's going to I'm not sure if I'm using the right word, but do we write some similar logic that's going to do an extraction and find that error and skip the dependencies or are we going to load uh with or without a dependency and figure out later figure out the issue later? >> Okay, that that's a good uh that's a good question. So in terms of when when when you are dealing with uh critical data if it's actually critical when when you're talking about critical it's like they are a lot or there is uh a lot of dependence on this data then on this on in that case you can't skip it. The reason why you can't skip it because uh if you skip it which means your insights are going to be inconsistent. There's a lot of inconsistence now that you have uh ignored or skipped uh let me say for this example now that you have skipped uh let me say um the uh let me say there is a critical field that depends if it's actually a field now then it's going to be a big problem because now uh if the the field probably you're using it for a join which means how are you going to link those different models, it's going to be a problem. Uh, it's going to give you an error. I don't know if I'm I'm I'm I'm making sense there. >> Yeah, it makes sense. But I guess then it it goes down to the decision of what's more important, killing the entire workload or only skipping and breaking your insights. I guess it depends. >> Yeah. So, so this is what uh I did mention whereby uh business they have to let me see if I can say. So, business should actually define the this the the the standards and the the um the rules. So, if the business uh cares about certain things, then it's something that you can't skip. You have to probably have to fix or handle. Yeah, makes sense. Makes sense. >> Yeah. So in in in that in that manner then it actually um uh in in in that scenario now it's it's whereby how data will actually enable to to solve uh problems within uh business processes. Uh for example um now uh in in this scenario where the email address was was was was not valid which means either you have to know to go to like software engineers to say okay guys on your form when someone is registering as a customer what these are the fields that we really care you they should be able to put filters in the first place that a person can't go through with such an email. >> Okay. >> Yes, >> I know that makes sense. >> Yeah. >> Uh, any other question? >> Okay. >> Yes, Chris, I have a a followup question. Um, >> yes, >> it's a follow question to to question. uh generally I think I'm answered in terms of uh to say uh how you handle uh malicious data or missing data is dependent on business decisions right >> yes >> but in this example that you have let's say a customer didn't didn't have an email like this uh how would you handle it because I saw in other columns sorry in other tables there you have uh the very same customer that you have skipped He has orders, right? >> Yes. >> How how how are you going to handle this one? >> This specific case. >> So, so it it goes to now investigating how did this person ended up being in our database. So, the first thing that I would do is actually engage with the engineers, the software engineers to be like, okay, what's the process? Uh is there uh um a filter that or a way that uh you use to to actually uh enforce people to when they when they actually registering uh to make sure that that emo is an emo that is that is valid. Um if they say that they have then we have to try to figure out where this data is coming from. Now we're now checking our sources. we're talking how our databases were actually uh created uh in the first place. So it's it's now become an investigation. Uh if we we we we see that uh our processes of onboarding a member, it's actually it doesn't allow a member in that then we have to delete that uh member. probably there is um probably it will be a hacker that will have had uh access in our databases and using that to commit probably fraud. >> Okay. So it means to say the the the orders that are linked to that very same customers will be discarded as well. >> Uh >> yeah based on the investigation. >> Yeah. It's not based on that. uh in in in in in it will also depend with the business uh um uh rules. What do you do with with certain data when there is a a possibility of of of fraud? Do you discard data? Um in this case, I don't think you there is a possibility of discarding uh the orders. Uh the only thing that we can actually delete is actually that member on that member level. You can actually delete so that probably they are using that uh to actually log in uh as a member and actually end up make making those orders. Probably they found a loophole. Um so you wouldn't delete such entries because probably you want to keep records to be like okay these are the fraud uh transaction that we had uh in 2015. Uh these are the insights that we'll be using now to be like okay these are the transaction that we actually ended up having. uh and is in that case you you're looking at um if uh your your fraud uh incidents actually going down or the number of people who are committing fraud or actually going down or not. Then those are uh we actually end up using this information to make like decision uh decision to say that okay let's change our process based on the number of fraud or these kind of members or orders that we actually receiving from people who don't have uh uh emails that really make sense. >> Okay. Thank you. >> Yeah. >> Yeah. So, so um is uh I think I did explain that the way the uh way of actually building uh pipelines. So now this when you're actually building the the cleaner one uh version this is whereby you actually have your you actually have your rules. Um now and this this is what how I would actually uh make sure that I actually test. So um when I'm actually building that customer clean uh uh model or data pipeline uh first thing before I even load it is to make sure that I test uh the the the emails to say okay it's not now because you don't want the customer that doesn't have a a now email or uh you also want to check the validity of of that email. Is it that emo is that valid or not? So these are the things that I would actually put into consideration before you actually have your your your your data or your model to run. Okay. All right. Let me share my Okay. Are there any other questions from the team? Um, can I ask um Chris? >> Yes. Before you close, in terms of what you you you just presented, um I know this was not part of your talk, but in terms of AI, um and in the space that you are in where you're building models and um cleaning up the data and uh looking at the quality of the data and observability of the data. My question then arises in terms of what are the opportunities um especially with AI automation. Um have you start venturing into looking at what are the kind of um AI models that you can build to enhance or to assist with data quality challenges? Um and if so um can you just give us um yeah an example of where you used it? Thanks. >> Um in terms of uh data quality uh checking those things um it's it's difficult to to integrate uh with with AI because of how uh our business uh the business uh rules that we have probably AI doesn't it's not exposed to to such so um And because of how AI is not yet matured to the extent of handling each and everything and also the expenses the cost that it actually that is involved in um integrating let me say chargbt uh into the data pipelines. It's it's it's expensive. uh uh there is a risk of uh your your your your data security how your data governance with with with with uh with AI uh there's still fear within organization to expose uh data to uh AI in terms of like using the mo models that are there right now uh people are still in this stage where they fear to expose their data um uh where's their pipelines. Um so at the moment uh we we in in most of the organization that I've worked it's it's it's a place where we be like we should be careful with our data. So the data governancy of most of organization they don't allow people to expose data uh because they fear copy act uh if there is a data breach and they realize that the data was being exposed to uh AI models like deepse and uh who actually cause uh chaos or it will it will come with um cost uh with within the organization. So yeah, those are the things that organization are very hesitant to do at this point. I don't know if I answered your question. Um so the what is the skills in tech uh in tech stick in data engineering right now it's um if someone wants to go into data engineering uh you must be comfortable working with SQL that's the first or basic uh skill that you should have. You must be comfortable working with Core. You must have some basic understanding uh in Python. Um then you also have to know cloud, how to use your cloud. Uh uh your AWS, your Microsoft Azure, your GCP. Uh now you are now looking at Snowflake, you're now looking at DPT, uh data bricks. So if you know your SQL Python and have basic understanding of cloud then as a beginner um you'll be able to navigate yourself. So I'm I'm I'm answering to the question that uh D asked uh what are the what skills in tech state uh data engineering now typically for beginners have a weirdish question. So with AI we know especially with large uh models you know that they may say that data is clean but it does come with some biases. >> Exactly. Yeah. uh most obvious one I think um there was an instance I'm not sure but maybe let me ignore this but let let's let's say how how do you control the biases especially introduced by your data and and there are obvious ones where you can look at outliers and where your data sits outside the median but uh more on the softer ones like uh like uh Let's make this example of misdiagnosis client misdiagnosis ending up with issues. However, maybe they have similar symptoms and then we end up with a bias that says if this diagnosis is done, you might end up with a patient dying. How do you take care of such biases? Especially when we're dealing with large uh models. Um so I'm I'm not really into uh building uh data models. Uh in data engineering it's usually uh data pipeline. Um I'm not sure if you're asking of like using uh models like yeah the AI models that are there like CHPT uh uh DeepS and others how biased they are. Well, no. I'm I'm just speaking about models in general, not the the ones that are already there in existence. Uh in terms of training, maybe even besides looking at actual models, but just looking at how do you preserve data um knowing that it stays as truthful as possible. However, it carries minimal bias. >> Okay. uh I think it's something that I did uh mentioned in my slide in terms of uh making sure that where your data is coming from uh understanding the flow uh and the lineage where your data is actually coming from uh is it a reliable uh source uh how trustworth it is is it being governanced and it have proper ownership yeah I I'll give you an example imagine uh you you want to build a model, right? You want to train your model. You find probably your model it's based on um people uh purchasing behavior. Then you find a a CSV file with data of people who have purchased certain products and you just take it and be like, "Oh, okay. This is just like uh uh data that I can actually use to train my model." First of all, you don't know who the owner of that. You don't know if it's being kept um secured in terms of no one has access to just like make any uh entries within the data and to also ensure that does that data reflect reality. So these are the the the the the qualities that or the the the principles that you have to to use whenever you're going to use your data like how accurate it how accurate it is, how complete it is, how trustworthy it is. So by following the core principle of of data quality and observability it will help you to have that confidence or remove the biasness of um making such issues. >> Okay. Okay. Understood. Any other questions? >> Hey, last one from my side, Chris. Um, >> is data engineering um is is it going to be disrupted by uh agentic AI? Like are we going to see agents writing uh pipelines anytime soon? Uh no it it can it data engineering it's it's it's someone or it's it's it's a role that is between uh the software engineers the data analysts the data scientists the machine learning uh developers whatever you call it as a data engineer I interact I communicate within different teams and make sure to ensure that our data is accurate, it's consistent, to make sure the governance is being taken care of. So um the only thing AI can do is to reduce the number of uh engineers or data engineers. Either either way it will only the only thing that you can do to most uh u um most career uh in in it is to reduce the number of people uh because now instead of me as to have five data engineer writing data pipeline my it can only speed up the way I actually build my data pipelines. Not that it can do that because um what usually happens in in organizations uh for data engineers is whereby the software engineers they make changes uh let me say from the front end. Let me say for for that example that I showed you of the emails, they make a a change in the in the front end or in their in their process of on boarding a a new member or new customers. Probably they someone uh removed a certain part of the the code that actually validates someone's email. um that change now I'll be able to pick it up and go back to be like data engineer uh to to software engineers guys we see that there are people who are being registered without valid emails can you check your your processes in terms of how you onboard someone and this is whereby now they have to go and check and make sure that okay their process is actually correct and if there is a bug then they will actually correct it. It's now the data that is helping to actually fixing some of the processes in the business. So if I don't think an AI have that capability of being able to communicate uh within teams and resolve issues that are complicated like that. >> Yes. >> Well understood. Oh, hello. I wanted to ask about like how exactly you you broke into like the the field of data because I saw you said you started with the with data science like was it a full-time role or was it a grad program and like like what were like the the minimum skills that you had back then that allowed you to like to get into that role >> and like your degrees and stuff and your projects. So, so uh I uh how I started working on that project, it was it was based on a personal project that I was actually working on after my postgrade. Uh so I did postring data analytics and business intelligence. So it's based on the project that I was actually building which uh which showed uh the the guys that I ended up working for that I had the skills to actually build something. Uh so in terms of the skills that I had I had Python skills uh I had understanding of the machine learning uh and also the statistics those are the skills that I had at that time. And in terms of cloud, I didn't have any experience working in cloud. So I had to learn on job. >> Oh, so was it a full-time role or was it a internship? >> Uh it was a full-time. >> Okay. Thank you. >> Uh so yeah to just uh so when I was doing my my my postgrade uh they when you doing your research you had to do uh internship. So you have to to do your internship while you're doing your practical like you have to be a practical hands-on uh research. So that's when I did my intern. Then after that I was doing a side uh project that I actually presented and uh the guys that I ended up working with they they liked it. So it's very important if you want to uh break into uh this data engineering or data space to have kind of like a portfolio or uh some work that you actually do on the side not only to have a CV. Yeah, Chris, not to discourage for you, right? Uh uh uh you can actually uh build that portfolio of projects that uh Chris is talking about by yourself uh through self uh uh learning. Uh there are a lot of platforms, there are a lot of blogs uh you can you can you can uh uh use to to to build uh your your data engineering portfolio of projects and break into the field. >> 100%. Yeah. >> Thank you. Yeah, just to uh uh wrap up uh in uh in our key uh take takeway. Uh so so when when whenever we're talking about uh data quality uh we we looking at how robust your pipelines are, how scale and how trustworth they are in terms of how they actually make the data to uh to to be of higher quality. uh uh for the data to be uh accurate. Um so um that's one of the the the most important thing that uh data quality observability actually help you to actually build uh robust uh pipelines and in terms of uh that data quality it should be a habit uh not just a project. Uh so whenever you building a data pipeline you must have in in the back of your mind uh the core principles of data qualities uh and you should make sure that you observe it when you're actually building uh that not only to have uh one project that has it. Um then in terms of the cost that you have to pay for bad data, sometimes you don't see it uh uh when the data is small but then as data grows the data uh the cost can be very expensive uh in terms of how you can make a bad decision based on uh the data. It can be really costly. Uh then in terms of how you can make sure that uh your organization have proper data quality um you should focus on automating uh monitoring and governance in your data. So what I mean by that is like if your data pipeline they are automated and there's con consistent monitoring and you have your governance with in place you have uh alerts if there's any data that is uh that doesn't pass your test uh your pipeline should either fail not just accept each and every data that comes through. So these are the tests that you must put so that your your data it's it's it's it's it's it's good. It's it's actually of higher quality. Um then yeah that's it in terms of uh data quality. I think we we did have uh some um questions. I don't know if there's any other questions. Yes. for you. >> And last question I wanted to ask like in the workplace, how exactly are like what do you call them the the data warehouses, how are they like populated? Is it like through web scraping or is it like through like getting it from like a form of a website? Like how exactly is it like populated? >> Okay. Um I'm going to share something. You see my screen maybe I didn't u really touch on this. So your your your question is based on uh how your warehouse uh it's actually populated. So, so on this data pipeline here that you see here, right, this is how you'll be building probably a data warehouse. So, in in in this case, I'm actually just showing you one uh data pipeline that is actually just showing uh or populating orders only. In an organization, you have customer data, you have employees data, you have events. Uh when when I'm talking about events, you're talking about like um uh how what people actually doing on the website. People are being tracked in terms of like okay this person, they logged in, this is what they did, they purchased this, they clicked here, they they cancelled this, this is where they drop off. So in most organization uh they have um different tools that they actually use to to to do data mining or to capture or to collect data. Uh there is some tool that I've worked on in a certain organization called postwork which is a tool that is used to track and record each and every session each and everything that you do on a website. Let me say I'm on their website right now. It will be able to track even the way my case is actually moving. Uh each and every place that I click if it's a button, it's actually recorded. So those events, it really means something to that organization. So it's actually captured and stored somewhere. So all those details uh they end up being in data tables. So it's usually used uh you have a source probably that source is coming from yeah let me say that uh product which is called postwork. So you have probably like another data pipeline coming with post uh data which is like all the events and those all events will be connected to probably like a member or a customer. Then you'll be like, "Oh, this member, he spent 50 minutes on their website and they only made five uh they only made five orders." So they can make decision to be like, "Okay, this is the time that people are spending on their website. This is the area that they actually cleaning." So yeah, it's there are different way they actually collect and process the data into uh into the data warehouse. So um I'm not sure if you you know the difference between uh data warehouse, data lake and a database. Uh >> no, can you explain it? So, so for for for instance this is this is a database right that have row data then now here um we actually process it into what into a data warehouse. So in this case our data warehouse we actually creating using snowflake. So uh so this is where I'm actually building my warehouse in. So a warehouse this is where you now have aggregated aggregated data. Uh a database it's probably just a a table uh or data that is being stored. Then a data data link it's usually where you just dump your data. It's it's only that I don't have uh this uh I was just playing I was just using uh snowflake. I would have shown you um uh probably AWS, Amazon uh web services where you have you actually build your data uh data lake where you just pull all the data and uh store it without being processed or sometimes you do a little bit of processing it but then the aggregated data now it's where you actually pushing it into the data warehouse. Um thank you very much Chris and thank you very much to all the participants who um are on the session. Um I think Chris your um your session has sparked a lot of discussion and and and debates and I think people are more interested in finding out more because I think the minute you went too technical and shared your screen and showed people all this and the pipelines and all that. So it sparked a lot of conversation and discussion and I I take one of those questions in the chat that say that talks about what are the skills that people need or the type of technical stack that people need in order for them to get into the data engineering um space as well. And I think that would require a another session where we go into this in more detail and especially also like also clarifying those type of terms in terms of what a database is, what a data lake is and all that. uh it it it it can take a um a a session on its own and I'm also mindful that we are 20 minutes um over time and um I don't also want to waste more of your time. So on behalf of Pambilia Analytics and the mentorship program, I would like to say thank you for sharing with us all your insights and for contributing your knowledge and imparting that knowledge um with everyone who joined this. We truly appreciate um what you've brought to the discussion as well. And uh we hope that you from now on you will be coming on board and sharing more knowledge with the participants as and when you have the availability as well. And to all the participants online, thank you for taking time out of your busy schedule and on a Friday noal um to be here and to listen to Chris to share. I hope this session was worthwhile. Please let's give Chris a warmly a online virtual round of applause. >> Thank you Chris. >> And Chris I I guess by show of those gratitude it really shows that you did deliver and you did um assist them to get some insights out of this session. Uh and on that note I will declare this session over. Thank you very much and see you on another session. Thank you. >> Thank you very much everyone. >> Bye >> bye. Goodbye.