Hi folks. I'm Santos. I'm a senior database administrator. As a senior database administrator. I spent so long talked to service, started to understand my jokes, mostly the s ql o and stuff. So today I'm gonna talk about AI driven anomaly detection for cloud data pipelines. So picture this you got all this amazing data. Flowing into your company, right? It's like a Jane Digital River powering everything from showing you relevant cat videos. Okay? Actually running the business now, keeping that river clean and flowing smoothly, that's where the things get tricky. So think about it. We are talking about tons of data coming from everywhere, going through all sort of twists and turns into the cloud. Is that trying to manage the plumbing in a city like City of Texas? One little clog, one tiny leak, and suddenly your reports are garbage. So your decisions are based on who knows what, and you're potentially facing some C ds Oops moments with the folks in suits. Traditional ways of checking if things are going wrong. They're like having a guy with checklists standing by the river, looking for a specific types of trash. But what if the problem isn't obviously trash? What if it's just a water turning a weird shade off green, slowly poisoning everything downstream? So that's where our superhero comes in, ai. Over next 20, 25 to 30 minutes, I'm going to show you how we have built this smart system that's like a superpower water quality expert for your driver, data drivers. It use all the fancy AI stuff, machine learning that actually learns. What normal water looks like. Deep learning that can spot even the subtle changes in the flow, and even some brainy stuff that tries to understand what the data means. This isn't just about finding big, obvious errors, it's about getting a real healthy report on your data so we can fix things before they become a disaster. Think of it as a preventative medicine for your data. We are better than waiting for a data to get sick, right? Hey. Again, my name is Santos and I'm still chatting with you all day today. I'm gonna talk about the data changes of modern data pipelines. Where why AI superheroes so necessary in the first place of modern data pipelines. They're not your grandpa Grand hose. They're more like a massive, interconnected networks of pipes, pumps, and filters spread across and entire digital landscape. And that complexity, it brings a whole heap of headaches. So complex architecture, the web of interconnections, think of it as a game. Digital Spa Monster data comes in from a million different places. Maybe it's just customers like a website, sunsets in a factory, sales figures from a dozen different systems. Then it gets tossed around between all these different cloud services. One thing transforms is another store. It. And another analyze it, and then it gets sent to even more places. Each time it jumps from one system to other. There's a chance to think to go wrong. Maybe the data gets transferred incorrectly, maybe the little bit gets lost in the transit, or maybe the system just have a different idea about what a data. Date or a number even means this tangled web of connections is just begging for subtle errors to sneak it and trying to track them down the old school methods is like trying to find a specific knowledge in what whole mess I. Good luck with that. So difficult detection, finding the needle in a haystack. So you've got this crazy complex system with tons of data whizzing around how do you even know when something goes wrong? Traditionally monitoring is setting up a few basic alarms. If the water pressure drops below this point, some the other. But if a problem isn't a certain drop, what if the gradual decrease over time? Or a slight change in our color that only trained, I would notice this subtle shift. These tiny deviations from the usual flow might not trigger any of those basic alarms, but they could be early signs of much bigger problem brewing. And with the sheer volume of data we are dealing with manually, looking for these little clues is basically impossible. You need any army of, you need army of data detection. Starting at endless streams of numbers made a smart system that can learn what normal look like, so it can automatically flag even the quietest something, not right signals. So it can automatically flag those without, with those, a little bit of tiny minute, something not right signals. So financial impact, the hidden cost of bad data. Now we are, we might be thinking, okay, so some data gets a little wonky. Big deal. Wrong. So bad data ist just a technical problem. It hits the company where it really hurts. The valid and the numbers are actually pretty scary. Studies shows the companies can lose around 20% of their revenue due to poor data quality. Let that sink in for a second. That's like throwing one out of every $5 straight into a digital garbage care. And where does this money go? Wasted time, compliance, nightmares, dumb decisions, and a angry customer and a missed opportunity. It all adds up. So yeah, good data is in just a tech thing. It's a straight up business necessity. So few of the key features of our solution. So how does our AI power system actually step in and save the day? It's, it got a whole toolbox of clever features designed to be ultimate data quality guardians and advanced detection algorithm. The multilayered approach to intelligent. We don't just throw one type of AI at this problem and help hope for the best. We got a whole team of digital detectives each with their own specific skills, the old school smarties, using statistics to spot obvious awareness and the modern brain using deep learning to catch the real subtle stuff. Adaptive learning, continuous implement to experience this system is in just one trick pony. It actually learns from data, it sees the feedback it gets from your smart humans consistently getting better at knowing what normal and what's not. So real time processing, immediate insight and actionability in the cloud. Things move fast and then so does our system. It's spots problem as they happen, letting us jump in and then things fix things before they cause real chaos. Our AI driven. Anomaly detection system tackles these challenges head on with a suit of sophisticated features designed for proactive and intelligence, data quality management. So when it comes to advanced data detection, I. Advanced detection algorithm and multiple layered approach to intelligence. Our system doesn't rely on single magic bullet algorithm Instead. Instead, it employs a careful curated combination of techniques to provide comprehensive and accurate anomaly detection. Traditional statistics techniques. We leverage a. Establish statistics methods like control charts for monitoring data distribution over time. I identifying out outliners based on standard deviations and employing time series analysis to forecast expected values and detect deviations from these forecasts. These provides a strong foundation for identifying well-defined anomalies, modern deep learning models to capture more complex and new. Nuisance patterns. We integrated cutting edge deep learning models, variational auto encoders, learning the underlying structure of normal data, and can identify subtle anomalies by detect instance that are difficult to reconstruct. Long short term memory networks are crucial for understanding temporal dependencies. In sequence. Networks are crucial for understanding temporal dependencies in sequential data, allowing us to detect anomalies. That manifest as usual sequence of events or shift in trend over time. Assembly methods, the wisdom of crowd to further enhance the accuracy and robust offer detection, be utilize assembly methods. This involves combination of predictions from multiple different algorithms. By leveraging the strength of each individual model, we can achieve higher oral accuracy and reduce the currency of false posture. For example, a statistical model might quickly flag a larger outliner, while a deep learning model might detect subtle but persistent drift in a complex data relationship, so real time processing, immediate insight. In immediate insight in the dynamics of the cloud data pipelines speed in the Paramount, our system is engineered for realtime processing as data streams through the pipelines. It is con, continuously analyzed by our detection algorithm. When anomalies identified, an alert is generated or almost instantly. This is realtime capability is crucial for several reasons. Preventing error propagation, faster incident response, enabling realtime decision making for applications that rely on realtime data immediately. Anomaly detection ensures that decisions are based on most accurate and accurate information. Adaptive learning, continuous improvements through experience. Data. Pipelines are leaving. Breathing systems. That consistently evolve. Our system is designed to adopt these changes through continuous learning, learning from historical data, incorporating user feedback, implicit learning through remediations dynamic threshold adjustments. So based on continuous feedback, observation, data patterns, the system dynamically adjusts its detection thresholds. This ensures that system remains s due to genuine anomalies while minimizing false alarm as underlying data characteristics exchange over time. Now, system design and architecture. Alright, let's speak. Under the hood and see how we are actually build the brainy system. We designed it specifically for the cloud, thinking about making it flexible, able to handle huge amount of data and easy to maintain, flexible and scalable adapting to diverse cloud landscapes. We know you folks are in all using the same brand of cloud, so you may. We made sure our system plays well with all the major players and can handle your growth. Growing data needs, it's container, containerizing, microservices, independent unit of functionality instead of one big complicated thing. We got a bunch of smaller independent parts like Legos, each doing a specific job. This, make it easy to deploy, scale the right parts, and makes the whole system more reliable. Modular components, adaptability and evolution. This Lego like approach means we can easily update, replace, or even customize the parts of systems without bringing everything down. So in depth in the cloud. The challenges and opportunities of cloud environment in mind, emphasizing flexibility, scalability, and maintainability. So adapting to the diverse landscape, recognizing that our users operate in diverse cloud environments. We are architecting our system to be highly flexible and adaptable. It's designed to integrate seamlessly with major cloud provider like AWS, Azure and GCP, as well as hybrid cloud deployments. This flex. This flexibility extends to the type of data sources and pipeline components it can monitor. Furthermore, scalability is a core tenant of Artisan. As your data volume and the processing demand grow in the cloud, our system can scalable scale horizontally by adding more instance of its microservices, enduring consistent performance without bottleneck. Containerizing microservices. Independent unit of functionality. We embrace a microservices architecture where system is broken down into collection of small independent services, each responsible for a specific function. Data ingestions features engineering, statistical analysis, deep learning interference, alert management, and a feedback processing. Those microservices. Are a package into a lead lightweight container using technology like Docker. This container offers several key advantages, simplified deployment, independent scaling fault, isolation, technology agnostic. So microservices allows us to choose the best technology stack of each specific function promoting innovative innovation and efficiency. Modular components, so the modular design of our system goes hand in hand with the microservices approach. Each microservices represents a distinct replaceable module. This modularity provides significant benefits for a long time. Maintainability, and evaluation simplified updates. Individual module can be updated or passed without requiring a complete system redeployment, minimizing downtime and risk technology upgrades. As a new and more efficient algorithms are technologies become available, we can seamlessly integrate them by replacing existing audios. Customizing and extensibility The modeler design allows for potential customization and extensions of system to meet specific user requirement. In the future, new modules for specialized data types or detection techniques could be added without affecting the more functionalities, data ingestion, and traffic. Okay, so before our AI brains can do their magic, they need good clean data to work with. Think of it like a chef needing good ingredients. This slide is about how we grab the data and then get it ready for the analysis. So data capture, comprehensive monitoring coverage. We like, we are like thorough detective gathering all the source of clue, not just like a main data, but also the data about the data. How the pipeline is behaving and even little snapshots of data itself, and you can tell us what clue are most important to collect Data captures comprehensive monitoring coverage. Our system is designed to a holistic monitoring solution for your data pipelines. This means we don't just focus on raw transactional data, we also collect a rich set of contextual information. Metadata, operational metrics, sample data, configuration collection. So this includes information about the data such as source schema, data types, timestamps and lineages. Changes in metadata can often be early indicators of the problem, so we also gather performance metrics from various components of data pipeline itself, such as processing time, CPU, memory. Network latency and error rates. Anomalies in these operational metrics can often correlate with even precise data called tissues. So to understand the actual content and structure of the data, we strategically sample data at various points in the pipeline. This allows our algorithm to learn the normal patterns and identify deviations in the data values themselves. The specific data points and metrics we collect are configurable, allowing you to tailor the monitoring to the most cri critical aspect of your data pipeline. So feature engineering, e extracting meaning the raw data is often messy. Feature engineering is like our expert chef, taking those raw ingredients and prepping them just right, so our AI models can actually understand them. The raw data is just a native format, is often not directly suitable for machine learning analysis. Feature engineering is a crucial process of transforming this raw data into a set of meaningful features that are anomaly detects. Models can effectively learn from. This involves several techniques, data cleaning, handling missing values, correcting, inconsistency, and removing noise from the data transformation. Scaling numeric features ENC coding, categorial variables, and applying mathematical transformations to make the data more suitable for the algorithm creation of new features, deriving new features from the existing data from the high highlights. Potential anomalies. For example, calculating the rate of change of a metric over time, or creating interaction features between different data points, dimensional direction in the high dimensional dataset. The techniques like principle component analysis. Also, CALS, PCA, can be used to reduce the number of features while preserving the most important information, improving model efficiency and reducing noise. Next, coming to the format optimization. So different AM models are picky, so we make sure that the data is in the perfect format for each one of its best work. It's all about making sure it's good data, good insights out. Our pre-processing pipelines ensure that the engineering features are in the I format for each of the anomaly detection algorithm we employ. This might involve normalization and scaling. Enduring. All numerical features are on a similar scale to prevent certain features from dominating the CER learning process, data type conversions, converting data into appropriate data types except. Expected by the models structuring data for specific models. For example, formatting time series data into a sequence for LSGM networks, or creating input vector for VAEs. This meticulous data tion and tree processing data is fundamental to overall accuracy and efficiency of our anomaly detection system. Garbage in, garbage out. We ensure that our AI are working with clean, relevant, and well formatted data. Anomaly detect detection algorithms. Alright, let's get a little more technical and. Talk about the actual brains behind our anomaly detection system. The algorithms, we got a whole stable of different mod models working together. Statistical model, there are more reliable workhorses using classic methods to spot unusual stuff. Variational auto encoders. These are the deep learning art. Artist learning to recognize complex patterns. LSGM networks, these are the memory experts. Great for spotting weird sequences in the data. Over time. Assembly assemble methods. We combine the strength of all these different brains to get the most accurate results. It's having a team of specialists all working together to solve the case. It's a time series analysis. For time dependent data like pipeline metrics, these models forecast expected features, values based on historical trends and seasonality. Deviations from these forecasts are flagged as potential anomalies. Con control chats. These visually tracked data points over time. Agne, statistically calculated upper and lower control limits. Points failing outside these limits are considered statistically significant anomalies, distribution based methods. Example gas and mixture models. These models learn the underlying probability distribution of the data points with the lower probability of belonging of the learning distributions are flagged as anomalies. Proximity based methods. These methods identify anomalies based on the distance to their nearest neighbors in the data space, the data points that are significantly far from other data points are considered unusual assembly methods, combining strength for superior performance. Our assembly approach strategically combines the output of these diverse statistical models and our approach. Advanced deep learning models to achieve more accurate and reliable anomaly. Detections weighted averaging, assigning different weights to the predictions of individual models based on their historical performance and expertise in detecting specific type of anomalies, working, combining the predictions of multiple models through major. Majority voting and other aggregation techniques. Stacking using a metal learner model to learn how to best combine the predictions of base level models. So these strategies helps to mitigate the weakness of individual models and leverage their strengths leading to a more robust and accurate overall detection system. So variational auto encoders, unsupervised learning for complex anomalies are powerful class of deep learning models. Particularly well suited for unsupervised anomaly detection in complex, high dimensional data, learning latent representations. So VAEs learns a compressed, low dimensional representation and latent space for normal data distributions. So continuous feedback and adaptions. So now our AI system isn't just a set. Set it and forget it kind of thing to keep it sharp. We are built. We have built in a way for it to consistently learn and improve detection, the system flag. Potentially the system flags for potential weird feedback. You are smart humans. Tells it is a right or wrong adaption. The system uses the that feedback to get back over time, adjust its rules, and getting smarter at what that normal looks like. What the normal looks like. It's like a training, a puppy deployment, scaling strategies. When we rolled this out, we didn't just throw it at everything at once. We looked at careful approach, isolated testing. We started small, testing it out on less critical pipelines. First, education, educate, identification. We use these initial tests to figure out any weird edge cases, like sudden changes in data structures or very new pipeline startups. Gradual expansion. Once we are confident, we started rolling out to the bigger and complex systems. Full implementation. The goal is to have this smart monitoring across the entire companies, keeping all our data reverse clean. Alright, so all that fancy AI stuff and cloud engineering, it's actually delivered some pretty sweet results. So we are not just talking theory here. You have seen some real world wins, the 40% reduction in data quality issues. Think about it. That's like a cutting down the number of oops movements with your, with our data by almost half fewer corrupted files, fewer inconsistencies, fewer time. Someone says, wait, is this data even right? It means our reports are cleaner, our analysis are more reliable. And we spend less time chasing down data governance, basically less headaches for everyone. So 97, 99 0.9, 99.7%, reduction in reduction times. So this one mind, this one is mind blowing. Mind blowing. Before, when someone went wrong, something went wrong. It could take hours, sometimes even days or manual digging through logs and trying to figure out what happened. Now our AI spots is where things almost instantly, it's like going from snail mail to instantly may messaging for finding problems. The speed means we can jump on issues before they can cause bigger problems downstream, and 84, 80 4% fewer false postures. Nobody likes getting a bunch of alerts that turns out to be nothing. It's like a car alarm that keeps going off for no reason. You start to ignore it. Our system is much better at telling the difference between a real time problem and a just a normal, slightly unusual data point. That means our team can trust the alerts they get an then focus on real fires instead of chasing shadows. Overall, this AI system has really made our data environments way more stable. It's like we have got dedicated data health monitor that always on the lookout catching the problems early and letting our talented teams focus on building cool stuff and getting real insights from data instead of just constantly firefighting. I. So to give you a taste of how this works in the real world, let's talk about big financial institutions we've worked with. You can imagine in the world of real time trading, even the tiniest data glitch can have massive financial consequences. Think millions of dollars in the blink of ice. They needed a way to monitor this increasingly fasting moving. Data streams and catching more anomalies immediately. So we plugged our AI driven system into their. Trading data pipelines, the results are pretty dramatic. 43.7 reduction in data issues. They say a huge drop in number of data quality incidents that meant fewer errors in the data, trading data leading to more reliable operations and fewer moments. So this freed up their highly skilled expert from spending time on tedious data cleanups and let them focus on what strategic high value task 3.27 mil million annual savings. And here is a kicker. All that early detection and fewer data problems translated into some serious cost saving. This came from a few different areas, less time spent on manual monitoring and fixing errors, avoiding potentially huge losses from bad trading decisions based on faulty data, and even reducing the risk of complicated analysis due to inaccurate reporting. That's real money back in their pockets. All thanks to our smart data watchdogs. Now we are pretty proud of what we have built, but we are not planning on just kicking back and watching the data flow. We are always looking for a way to make our system even better and smarter. Here's a sneak peek of what we are working on, concept drift management. Imagine the normal of our data slowly changing our time, like the season changing. We are working on making our AI even better, recognizing these long-term shifts so it doesn't start flagging the new normal as an anomaly. It's like teaching it to understand the changing weather patterns of our data. Enhancing computational efficiency. We are always looking for a way to make our algorithms run faster and use fewer resources. Think of it as masking our AI brain even more efficient so it can process more data without getting a headache. This might involve smart algorithms or even using specialized computer hardware real time remediation. This is the holy grail, the feature where you, our system doesn't just tell you that there's a problem. But actually fixes it automatically without needing a human to step in. Imagine a self feeling data pipeline. That's the direction we are heading. Creating a closed loop system that keeps your data flowing smoothly and minimal human intervention. So to wrap things up, are AI driven. Anomaly detection system is a significant leap forward in keeping your cloud data healthy and reliable. Just smart. It's, it learns, it scales. It delivers critical results. It gives business a powerful tool. Not only catch problems early, but also build more trustworthy data foundations improve how they operate and ultimately make better decisions. We are excited about the future of this technology and the potential to make managing complex cloud data environments a whole. A lot less chaotic. Thanks for your time. I'm happy to answer any questions you might have. Thank you.