Hey everyone, this is Krishna Gal. I'm a senior software engineer working at Meta in the ads privacy infra team. I work closely in the intersection of privacy, infrastructure and scalable distributed systems at Meta. Today I'll walk you through why observability is becoming absolutely foundational to privacy aware infrastructures, especially at hyper massive scale. Like the ones being dealt at. Big tech companies. We'll talk about the challenges, architectural patterns, real world examples from companies like Meta and what the future looks like in this domain. And that's what the agenda here covers. We'll also go over the key use cases that we have in privacy systems from the perspective of observability and also the architectural patterns, benefits, and other. Major advancements in this area. So now what are privacy aware systems? Privacy aware systems were initially designed to ensure that user data is handled correctly. It's very important in today's age to be aware and also respect the user's consent in handling user data across. Service or a whole company stack as well. Since privacy and regulations are tightening every day across multiple countries, these, the systems are a part of a broader initiative, which is also called Privacy Aware, infra across the industry, which aims to embed privacy deeply into the technical fabric of the company's infra. Now, what does observability mean? Observability. In systems engineering, it meant inferring in internal system states from external outputs. That includes three primarily key pillars, logs, metrics, and traces In privacy, infra specifically, it's about reconstructing who touched a user's data y and whether they were allowed to touch that user data. Think of it as the ability to reconstruct a privacy black box recording for every interaction. That's critical when regulators or users ask What happened to my data? That's exactly what observability helps tackle in case of privacy aware infra coming to why privacy aware info needs observability. Data privacy today is under. Regulatory scrutiny some of which are G-D-P-R-C-C-P-A, India's DPDP, and now. These fines are really high. If you're caught not complying to these regulations, it could be up to 4% of your global company revenue. But what's more important than fines is trust. Your customers or users would just stop trusting your services if you do not handle and respect their privacy well, and it's not embedded well in your company's infra. But that's more important than fines and user trust. It's also very important for a company to be able to answer to their customers who access their data, what is going on with their data and where it's being used for. That's where observability comes in, not just for uptime or latency, but to trace audit and enforce data policies in real time. Real time is the key here. Now we'll touch upon a few. Really important key use cases in the privacy world. In terms of your observability there are five real world use cases that we encounter on a daily basis at scale. First is data lineage, which is basically tracking the entire flow of user data. It could start from where was it generated, who processed it, and was it shared externally? The other use case is consent propagation, which is basically if a user re revokes their consent in one ui. We need to ensure all systems respond to that in real time. One lagging cache could cause a compliance failure. To give you an example, for instance, if a user is okay with sharing their location for showing, let's say, personalized ads to them, and later they go and update their, settings to not share their location anymore. We have to make sure that for whatever cases their location was being used for, let's say for some kind of data modeling or some kind of machine learning model or training use cases, right? We have to go and make sure that the user's location data is wiped out from those models completely. That's what consent propagation is. The other use case is purpose limitation. Data collected for personalization should not be used for targeting or training ML models, as I said, without explicit consent. The other use case that we can think of is anomaly detection, for instance, login from a new location or accessing full history. Are these use cases expected? Is there a policy mismatch somewhere? All these. Questions could be answered really well if there's a very well oiled observability machine embedded in the whole privacy scenario. And then the last use case that I could come up with is policy drift monitoring, which is basically over time does enforcement. We can, our system silently bypassing updated constraints. All these questions should be answerable via privacy of our infra and without observability, it's not possible to answer these questions. Each of these need traceability or adaptability, and in many cases, automated enforcement. Now coming to what other system design challenges at implementing observability at such a big scale of privacy infra. First is volume. At hyperscale, you are dealing with billions of log events daily and petabytes of data. To be able to process all of this and generate meaningful observability out of this whole privacy data is a massive challenge. The second part is speed. In this case, you could, you cannot take hours or days to process this data since most of these services that a lot of these billion. Customer companies deal with are realtime use cases. So you need to handle and observability at the speed of subsecond latency. The other design challenge that one needs to deal with is granularity. Observability must be at a per user level, not just per system level. Since every user's privacy settings, privacy comfort is very different. It also depends on the ecosystem they are in. So it has to happen at a per use user in some cases per impression level. The last part is security. Observability data itself is very sensitive since it contains a lot of information about how the user interacted with the system, details about the user et cetera. So it must not leak PII or personally identifiable information. As it'll amplify the risk further and be a loss at compliance. So we need to design the speed, scale, and security all at once. Coming to what the architectural patterns you've observed in this industry. So to be able to build observability for privacy, the first thing is event driven lineage. We use services or systems such as Kafka or any kind of pub sub, system to track data flows in real time. The second part of the co whole ecosystem is consent aware tagging where we try to map every data payload to a specific purpose and scope. By pur, what I mean by purpose and scope here is we mark every data with what kind of systems can absorb that data. For instance, if it's. An ads ecosystem and a user has revoked their cons, has not given their consent to the service to use it, to use, let's say, their location for it. We have to tag their location data to not be specifically used for ad systems and maybe be able to use it for other systems. So consentable tagging is very important here. The third part is embedded policy engines. There has to be a lot of runtime guard checks if data access aligns with consent. The fourth part is cryptographic audit trails. Maybe use Merkel trees or blockchain style structures for immutable verifiable logs which help in audit trails and are also cryptographic at the same time too. As we spoke. Observability itself is a very sensitive privacy use case. These patterns are not just theoretical, they're being deployed in production by privacy focused teams across Vic Tech. Now, coming to some of the case studies especially lineage data lineage at scale, take to take an example of let's say meta. Linear systems track data across thousands of internal sources. Consent status propagates within the data and policy engines enforce compliance throughout these data sources. These systems power realtime visualizations of how data moves and are used by auditors, legal and engineering teams alike to be able to. Figure out if there's any kind of privacy leaks or if there's any kind of policy dilution or if there's any kind of misses. All these things are enabled through observability at such massive hyperscale. What used to take days of queuing logs is now automated in real time. This enables almost a 40 to 50% reduction in audit prep time across several inter internal teams at meta if applied at in the right ways using observability for such privacy aware infras structures. Now, what are the benefits of privacy observability? Faster audits as we spoke about it, reduce which reduce compliance friction, and build confidence with not just the regulators, but with legal, with customers itself. They enable reduced fine risk which basically helps in early detection of misuse of what, and he hence prohibit any kind of escalation. It also helps, provide enough team, enough time for teams to fix those at scale. Third part is internal trust. Developers have visibility into how their systems behave. So it's not just the end con consumer, but internally developers are confident of where and what is happening with a certain kind of data, with observability. And then the last part is safer. ML only compliant data reaches downstream models, reducing the risk of training or unauthorized input. In short, observability reduces both legal and operational risk while improving the system hygiene incredibly. Coming to what the future looks like in this domain there are still a lot of exciting open areas of research in privacy aware infrastructure. Looking specifically from the lens of observability, we can, let's say, use machine learning models to detect privacy anomalies in real time. And to be, to enable real time observability while these models are being trained. The inferred data is a big challenge in itself to be able to observe systems. Become policy enforcement systems both at the same time. That's again, a big area of research. The third part could be should we standardize policy observability like we did with open telemetry for system traces. There's no industry standard at the moment, especially to handle this at scale and in a very compliant and standard manner. This is a huge area, in my opinion, where researchers and developers could invest heavily with. The goal is to move from the fact of inspection to real time Automatic enforcement in a privacy control plane and observability helps move there faster. In conclusion observability in privacy infra is not optional anymore. If you can't see it, you can't enforce it. That's the that's my takeaway from. My experience working in this domain as systems scale only observable systems can remain compliant, ethical, and trustworthy at the same time. And the investments we make today are what will power responsible innovation tomorrow. Thank you.