Hello everyone. So my name is Kumar Mala. Thanks for joining my sessions on optimizing data pipelines with Azure Data Factory. In these sessions, I will walk you through how to improve your data flows for faster processing and loss cost, and less manual work. Let's get into the session. So today's data challenges, as you all know, that today data is huge. So we are getting the data from multiple systems like I streaming systems. And different files and file systems and different sources. So to handling this much huge data is nightmare and complex and challenges and integrating all of these systems is definitely complex because there is no relations between these systems and they don't talk each other. So to integrating these many systems with volume of this much volume of data is complex and challenges. And because of that, our process gets delays. And getting the right insights and decision making gets poor and it's it's like main maintenance cost is more effective, so more cost so to to overcome these challenges. I'm trying to introduce, I will introduce new technology. So that is Azure Data Factory. The Azure Data Factory is one of the Azure services. It's a cloud data integration tool. So there is a lot of advantages of the Azure Data Factory. I usually call it as a DF, so a DF because it's, it's supposed to integrate as many as data sources, like it's supposed to add RBMS and NoSQL databases and file systems and realtime systems. And API systems it'll integrate, it'll integrate all of the systems and it'll load into your desired system. So let's get more insights about the. So basically Azure data factory architecture depends on four more components four or more components, but the main components are the four. One is the linkage services. The linkage services is like connections between your data sources and your target systems. For example, if you are, if you like to connect your Azure data factory to Salesforce systems and you want to extract and transform and load into your data lake, so you need to create the linker services to connect your source system so that you know it'll like a bridge connection bridge between your source systems and Azure Data Factory. And then another component is data sets. Data sets is the real data structures. It's like your tables or files, data structure, what table, what columns you need to for your target systems and activities. The activities is the main configurable data transformations where you use. Actually your transformations and business tools and then pipeline. Pipeline needs nothing but like end-to-end solution from where your link services, data sets and activities altogether, orchestration, sequence of end-to-end solution is called pipeline. Let's get into another slide. So it's not only provides these many benefits and it's also reusable templates, pipeline templates. Suppose once you create one, one pipeline, you can use that for another requirements. Suppose if you have a Salesforce data, sales data and HR data, you can use the same pipeline for that, but with little tweaks, but you can reusable that pipeline. So because of that, you can reduce your manual work and cost. And another advantage is dynamic orchestrations. If, suppose as I said, an example, suppose if there is a new data comes into your source system or there is a data is edited, updated in your source system, the Azure data factory pipeline will automatically figure it out and it'll runs the pipeline and load into your target system. So there is no manually you need to log into the system and you no need to rerun the pipeline or run the pipeline. So it'll automatically detect and it'll load into your target systems. So we call it that event triggered. Execution because of that. So there is no manual intervention and it'll reduce your cast. And another thing is error handling. So when a system, when a pipeline handles huge data with the different data sources, definitely there is an address handling mechanism should be there. So the data factories also handles the errors very smartly. There is a retrain mechanism in case if your pipeline fails in between, so it'll retrace where it failed and from the checkpoint, and it'll process it further, even in case if it is fails, and it'll send to alerts through your emails or your messages or your teams channels so far for your prompt response to fix the issues. And because of these advantages, the, when you use the A DF and it'll improve your business and faster processing, 78% faster processing and cost reduction goes to 63%. And resource efficiency can use be 42% and you can make 90% of automation. So there is a case study, like a financial services. There is an ETL pipeline with the legacy tools and that was running for eight hours. And when we introduced the A DF pipeline and with the same transformations and business logics, it got down to 45 minutes because of that. And the processing time dramatically registered eight from eight hours to 45 minutes, like 91% improvement. So because of that, the maintenance cost register at 70% and delivering significance within quarters and within timelines. Implement these pipelines. So the best, we should follow the best practices organize your pipelines. I just said if you, if your organization has the sales data and HR data or other CRM data, so you organize your pipelines accordingly and always often. Check in your code into the gate and leverage runtime integrations and also enable your Azure Monitor Azure monitoring system so that so for a prompt response in case of any failures. What I would suggest to the organizations is so looking at your current pipelines and analyze what it slows and why it's cost, and then introduce Azure Data Factory for better, faster performance, for the lower cost, and more efficient way. Thank you so much for watching and joining my show. Azure Data Factory has helped me a lot and as well as my team as well to better to build a better and faster pipelines. I hope it helps you too, so feel free to connect me if you would like to chat more. Thank you so much. Have a nice day.