Conf42 Golang 2025 - Online

- premiere 5PM GMT

Go-Powered Data Architecture: Solving Enterprise Growth Challenges at Scale

Video size:

Abstract

Discover how Go is revolutionizing enterprise data architecture to tame the 42.2% annual data growth beast! Learn battle-tested patterns and tools to transform data chaos into competitive advantage—where others see overwhelming complexity, you’ll see opportunity

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hi everyone. Do you guys know enterprise data is growing at 42.2% annually, and in that yet 68% of that remains unutilized. I am sala. And today let's explore how go based solutions transform enterprise data architecture to meet tomorrow's challenges. Let's talk about some facts around the enterprise data crisis. As I mentioned earlier, 42.2% of annual growth year over year, yet 68, 60 8% is underutilized and 32% are an inadequate systems. They, most of the organizations and IT leaders believe that the existing data architectures cannot handle future, future demand. and 44% of the time of other, of data professionals on a single workday is just lost in trying to find validating data across the board to achieve one single goal. Why Go is a valuable asset to, to a data architecture solution? Three things. One, it provides superior concurrency. Number two, it provides immense performance. Number three, it en enables integrations with cloud native ecosystems. Let's talk about the first one, the superior concurrency. Go lightweight. Go routines basically enable to effortlessly, run parallel data processing. It can integrate across multiple data sources very seamlessly and enable that parallel data processing generating. Super, performance when it comes down to running real time jobs. Let's come to per let's talk about performance. Go is basically a compiled language. It's basically with intelligent garbage collection, it delivers nearly c performance. with developer friendly syn taxes, it processes data significantly faster than interpreted languages like Python, JavaScript, while maintain quota liability. it is also widely u widely popular language lately, and people are started to realize the performance that is bringing to the table in order to enable the jobs in order to make sure the pipeline's done as fast as possible. Next is integrations with cloud native ecosystems. Over the years, over the last couple of years, we have seen many organizations. Start to modernize their, on-prem systems over to the cloud, whether it's Azure, whether it's, GCP or whether it is AWS go. Basically enables, integrations with tools like Kubernetes, Docker, Prometheus, which enables scalability, which brings you, e easier integrations. And it also basically provides extensive libraries and SDKs which integrate natively with modern cloud platforms. and with these integrations door, it basically simplifies deployment and scalability of the entire data architecture as well. So in a typical, and in a typical organization, there is a, there is various levels of data maturity. If you see, as you're seeing on the slide, every organization basically should, starts with slide, with point number four. Wherein its basic collection, there is multiple systems, multiple applications within an environment. There's tons of data available. Everything's very fragmented, very reactive. It's that you, your first activity of maturity, basic needs to collect everything. Step number three basically is about structuring the data that you've gathered. So basically trying to have it plugged into an active database, ensuring that you have a structure enabled to it. It doesn't have to have a formalized model though, but there has to be some level of structure, integrity available for the data sets. Step number two is basically wherein the advanced analytics comes in picture. This is where you basically define a purpose-built data model, depending on the industry and your business value. And then you run high analytics using that data, specific data model itself. And the peak of this chain is basically that it's air readiness. It's basically self-optimizing systems wherein you don't have to, explicitly define something. The ai, basically, you develop a data mature strategy so well that the AI basically starts to self realize what the, what each data point is and tries to generate solutions for you for various value added cases, eight, surprisingly, 87% of the organizations, they're remain, they're trapped with the, with three and four, with Go. It enables this maturity, it accelerates this maturity level by 3.5 times. And enables companies to basically transform raw data into actionable intelligence and predictive capabilities. Let's talk about the various, phases in basically defining a data pipelines. We have ingestions, we have transformations, we have an analyzing and then distribution, right? So the first step is ingestion, right? Primarily, in any data architecture, when you have to basically start to get information from various sources, you ingest it into one data platform with Go, it enables you to con connect to 10 different sources simultaneously and processes up to 2 million events per second with minimum C minimal CPO overhead. This kind of real time capability is a huge accelerator for any kind of an. a large data architecture in any environment. The second step is basically the transformation piece. This is where you write to data pipelines. You include the logic. You ensure that the logic basically adheres to the model and ensures that from the source system to the model. You, you develop the entire pipeline. Go routines basically provide you very lightweight, ETL capabilities, but it provides you the capability to execute complex ETL operations as well. In parallel, using these go routines, you will be able to achieve 85% less latency while scaling linearly across CPU course. So basically, with this, with, a smaller scale, the smaller scale, compute, you are able to achieve very high performance. The next phase is basically the analyze. This is where go basically interfaces with applications like TensorFlow, frameworks like Apache Spark and customized custom analytical engines. It enables subsequent decision making on streaming data. So as we've talked about earlier in the ingestion, when we are able to bring in the capability of generating data in near real time, it also enables you to analyze or basically provide you insights out of the data in the same fashion. It enables you to do smarter decision making and enables you to basically reflect results in, in, in a very fast manner. And the last is distribution. So how do we basically have this data available to other data sources within the organization? So the Z it Go basically offers a zero copy delivery to data lakes, warehouses, and applications. It maintains cryptographically verified data needs throughout the pipeline. Then next, let's talk about the data governance framework. as part of the overall design, with the various phases occurring. Governance is a very key aspect in the future of any organization with so many data sources, so many data attributes available out there, I. A need for data governance is mandated across, especially our organizations when there is financial transactions involved, when there's health healthcare transactions involved, whether there's insurance transactions involved, anything wherein there is so many data sources available across, across the organization. This governance framework is mandated and much money, much needed. It. So what does governance framework basically offer? It basically allow, offers you metadata management, basically allows you to catalog, create a data catalog. It also enables you to show you how data lineage, how data moves from one system to another system, what one attribute means in one system, how it transform to another attribute system. The second is about security and compliance to basically. Flag the flag attributes, which are P-H-I-P-I-I with sensitive information. You basically ensure you read the reg regulatory requirements and make sure you have all audit checks and, definition audit auditable definitions created for those attributes. Third is basically quality control. Quality control is wherein you ensure that a specific attribute is defined the way it's supposed to. You ensure that if that is basically not satisfying the original, originally defined manner, it basically fails quality, so ensure that integrity maintains all the way through. The last is basically access management. This is where, it enables users, to access the data through role-based access control, depending on the level and the type of access that they will need. How Go helps in all of this go basically go offers concurrent microservices architecture, enables governance frameworks that scale elastically and automatically adapting to new regulations while reducing compliance overhead by up to 40% compared to tradition systems. That is a huge value, that's a huge shoutout that we need to give to the existing Go framework. Let's talk about an active use case, of where Go has added or go has become, a game changer in this. So the problem, the problem statement in this specific use case is basically, in this specific healthcare organization, 36%, they was, they were facing 36% annual data growth across multiple silos or silo legacy systems within the organization. This patient records were scattered across multiple departments. There is no way basically to bring the data together. And what was the end result? It was basically causing critical delays in, in care delivery. So for example, if some doctor basically goes and requests for a certain patient records for the organization to provide information back, it takes a couple of hours. That kind of, delay basically, causes, a huge gap in how patients, how doctors engage with patients and try to provide the best care possible. How go solve this problem? It go basically, the Go Solution basically implemented robust microservices architecture, connecting 17 disparate systems. It developed real time HL seven FHAR translation engine with h HIPAA compliant verification protocol. Doing that. it, it enabled lightning speed availability of data. It also basically provided you 83 per percent reduction in the overall clinical data retrieval time. Comprehensive patient, as I mentioned earlier, now, the comprehensive patient histories are now available in seconds rather than hours, which basically improved overall care decisions. Another case study now talking about the How Go has impact, how a current problem in financial services and how go was able to help. So in this financial institution, basically there are fragmented regulatory compliances across 28 disparate systems creating, which was creating significant operational burden. Monthly compliance reporting require it took almost 160 hours staff hours to basically create compliance reporting. One compliance, one, one monthly compliance report, which is insane with the existing market that we currently have. What go did is basically deployed an image source data fabric with cryptographically secured audited trails. What is enabled is faster near real time availability of, of all of the in data and enable compliance monitoring on top of it with automatic automated anomaly detection. it, it is basically a combination of how you have machine learning and a good quality data with high availability and speed. So what it did is basically it reduced the overall compliance reporting time and 94% while eliminating manual error. It enabled continuous risk assessments across all trading platforms with millisecond level visibility. That is the power of go. Next, let's talk about basically, the how we arrived at cloud data, native data architectures. So as I mentioned earlier, one of the key features of, of Go is basically to successfully integrate with these cloud native data architectures. as you can see on the screen, the, how our cloud native has evolved as part of the modernization of many organizations over the last couple of years. It started with the traditional on-prem, then go with the hybrid cloud system, then the multi-cloud system, then very cloud native capabilities. So in here, basically if you see, goes lightweight concurrency model and efficient resource utilization, make it exceptionally well suited for cloud native data architectures. In a typical cloud data, native data architectures, the growth rate is around 14.2%. CAGR. By leveraging go's ability to support for containers, Kubernetes, microservice, serverless architectures, and SDKs to integrate with cloud native, it significantly reduce operational overhead and deployment complexities. Then let's talk about go libraries for enterprise data. So these are some of the key libraries that, that we need to focus on from a, from GO'S capability perspective. First in, in high power, high performance, ETL. Goum provides advanced numerical com computations with near native speed. GIA enables GPU Accelerator machine learning pipelines. Nats delivers Ultrafast message streaming with 10, 10 million messages per second. Second is governance spiff inspire, implement implements, zero trust identity frameworks across distributed systems. Ori Hydra provides scalable O-O-I-D-C authorizations. Jagger Pro enables comprehensive data in tracing with microsecond precision. All of these key components are what comprises of the entire data F frame data governance framework. The next is the data storage and retrieval. Badger DP offers embedded key value storage with SSD performance optimized performance enterprise Ready Go Client. Seamlessly integrated with Kafka, Cassandra, MongoDB, and Elasticsearch ecosystems. The last piece of the cloud integration as we mentioned earlier, native go SDKS for A-W-S-G-C-P and Azure deliver optimized cloud for cloud performance. Terraform provides enable in as a code with declarative data architecture deployments. Next. How do you basically come up with your with Go data strategy? In a typical, ideal environment, very first thing is we do an assessment. We basically go conduct a complete comprehensive data architecture audit, identify critical performance bottlenecks, security vulnerabilities, and governance gaps, and then we start to create a prioritized roadmap based off the business impact and implementation complexity. Then we basically come up with an MVP or a pilot. We create a set, a high visibility constrained scope initiative for, the initial go implementation, establish clear KPIs and benchmarks against existing systems, and then develop internal technical expertise. We showcasing early wins. Then once we are able to successfully achieve that MVP, then we expand and integrate. So we basically use Go Powered and microservices across, with prior data flows. And start to scale it out across, across the other use cases, within the organization. And then you basically implement compress and data governance with automated compliance controls. And the last maturity level, transform then scale solutions to basically enterprise by data, fabric architecture, establishing go powered foundation for advanced AI MAL initiatives achieve 50% reduction in unutilized data. While improving excess P 2%, 75%, this is where you get your full near real time capabilities. your transforming, ETL optimizations. Then it also eventually feeds into your AI ML initiatives as well. So overall, as I mentioned, go is a very powerful platform or a powerful utility, which covers you many tools for you to satisfy in, in, in building an entirely, robust, cost effective, scalable data architectures, not just from 1 1 1 perspective. There are multiple use cases within the industry that can be solved using those architectures. So looking forward to more questions. Thank you for everybody's time.
...

Sanath Chilakala

Director, AI & Digital Solution Architecture @ NTT DATA

Sanath Chilakala's LinkedIn account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)