Transcript
This transcript was autogenerated. To make changes, submit a PR.
Hi everyone.
Do you guys know enterprise data is growing at 42.2% annually, and in that
yet 68% of that remains unutilized.
I am sala.
And today let's explore how go based solutions transform
enterprise data architecture to meet tomorrow's challenges.
Let's talk about some facts around the enterprise data crisis.
As I mentioned earlier, 42.2% of annual growth year over year, yet
68, 60 8% is underutilized and 32% are an inadequate systems.
They, most of the organizations and IT leaders believe that the
existing data architectures cannot handle future, future demand.
and 44% of the time of other, of data professionals on a single
workday is just lost in trying to find validating data across the
board to achieve one single goal.
Why Go is a valuable asset to, to a data architecture solution?
Three things.
One, it provides superior concurrency.
Number two, it provides immense performance.
Number three, it en enables integrations with cloud native ecosystems.
Let's talk about the first one, the superior concurrency.
Go lightweight.
Go routines basically enable to effortlessly, run
parallel data processing.
It can integrate across multiple data sources very seamlessly and enable that
parallel data processing generating.
Super, performance when it comes down to running real time jobs.
Let's come to per let's talk about performance.
Go is basically a compiled language.
It's basically with intelligent garbage collection, it
delivers nearly c performance.
with developer friendly syn taxes, it processes data significantly
faster than interpreted languages like Python, JavaScript, while
maintain quota liability.
it is also widely u widely popular language lately, and people are started
to realize the performance that is bringing to the table in order to enable
the jobs in order to make sure the pipeline's done as fast as possible.
Next is integrations with cloud native ecosystems.
Over the years, over the last couple of years, we have seen many organizations.
Start to modernize their, on-prem systems over to the cloud, whether it's Azure,
whether it's, GCP or whether it is AWS go.
Basically enables, integrations with tools like Kubernetes, Docker,
Prometheus, which enables scalability, which brings you, e easier integrations.
And it also basically provides extensive libraries and SDKs which integrate
natively with modern cloud platforms.
and with these integrations door, it basically simplifies
deployment and scalability of the entire data architecture as well.
So in a typical, and in a typical organization, there is a, there is
various levels of data maturity.
If you see, as you're seeing on the slide, every organization basically should,
starts with slide, with point number four.
Wherein its basic collection, there is multiple systems, multiple
applications within an environment.
There's tons of data available.
Everything's very fragmented, very reactive.
It's that you, your first activity of maturity, basic
needs to collect everything.
Step number three basically is about structuring the data that you've gathered.
So basically trying to have it plugged into an active database, ensuring that
you have a structure enabled to it.
It doesn't have to have a formalized model though, but there has to be
some level of structure, integrity available for the data sets.
Step number two is basically wherein the advanced analytics comes in picture.
This is where you basically define a purpose-built data model, depending on
the industry and your business value.
And then you run high analytics using that data, specific data model itself.
And the peak of this chain is basically that it's air readiness.
It's basically self-optimizing systems wherein you don't have
to, explicitly define something.
The ai, basically, you develop a data mature strategy so well that the
AI basically starts to self realize what the, what each data point is and
tries to generate solutions for you for various value added cases, eight,
surprisingly, 87% of the organizations, they're remain, they're trapped with
the, with three and four, with Go.
It enables this maturity, it accelerates this maturity level by 3.5 times.
And enables companies to basically transform raw data into actionable
intelligence and predictive capabilities.
Let's talk about the various, phases in basically defining a data pipelines.
We have ingestions, we have transformations, we have an analyzing
and then distribution, right?
So the first step is ingestion, right?
Primarily, in any data architecture, when you have to basically start
to get information from various sources, you ingest it into one
data platform with Go, it enables you to con connect to 10 different
sources simultaneously and processes up to 2 million events per second
with minimum C minimal CPO overhead.
This kind of real time capability is a huge accelerator for any kind of an.
a large data architecture in any environment.
The second step is basically the transformation piece.
This is where you write to data pipelines.
You include the logic.
You ensure that the logic basically adheres to the model and ensures that
from the source system to the model.
You, you develop the entire pipeline.
Go routines basically provide you very lightweight, ETL capabilities,
but it provides you the capability to execute complex ETL operations as well.
In parallel, using these go routines, you will be able to achieve 85% less latency
while scaling linearly across CPU course.
So basically, with this, with, a smaller scale, the smaller scale, compute, you
are able to achieve very high performance.
The next phase is basically the analyze.
This is where go basically interfaces with applications like TensorFlow,
frameworks like Apache Spark and customized custom analytical engines.
It enables subsequent decision making on streaming data.
So as we've talked about earlier in the ingestion, when we are able to bring
in the capability of generating data in near real time, it also enables you to
analyze or basically provide you insights out of the data in the same fashion.
It enables you to do smarter decision making and enables you
to basically reflect results in, in, in a very fast manner.
And the last is distribution.
So how do we basically have this data available to other data
sources within the organization?
So the Z it Go basically offers a zero copy delivery to data lakes,
warehouses, and applications.
It maintains cryptographically verified data needs throughout the pipeline.
Then next, let's talk about the data governance framework.
as part of the overall design, with the various phases occurring.
Governance is a very key aspect in the future of any organization with so many
data sources, so many data attributes available out there, I. A need for
data governance is mandated across, especially our organizations when there
is financial transactions involved, when there's health healthcare transactions
involved, whether there's insurance transactions involved, anything wherein
there is so many data sources available across, across the organization.
This governance framework is mandated and much money, much needed.
It.
So what does governance framework basically offer?
It basically allow, offers you metadata management, basically allows you
to catalog, create a data catalog.
It also enables you to show you how data lineage, how data moves from one
system to another system, what one attribute means in one system, how it
transform to another attribute system.
The second is about security and compliance to basically.
Flag the flag attributes, which are P-H-I-P-I-I with sensitive information.
You basically ensure you read the reg regulatory requirements and make
sure you have all audit checks and, definition audit auditable definitions
created for those attributes.
Third is basically quality control.
Quality control is wherein you ensure that a specific attribute is
defined the way it's supposed to.
You ensure that if that is basically not satisfying the original, originally
defined manner, it basically fails quality, so ensure that integrity
maintains all the way through.
The last is basically access management.
This is where, it enables users, to access the data through role-based access
control, depending on the level and the type of access that they will need.
How Go helps in all of this go basically go offers concurrent microservices
architecture, enables governance frameworks that scale elastically and
automatically adapting to new regulations while reducing compliance overhead by
up to 40% compared to tradition systems.
That is a huge value, that's a huge shoutout that we need to
give to the existing Go framework.
Let's talk about an active use case, of where Go has added or go
has become, a game changer in this.
So the problem, the problem statement in this specific use case is basically, in
this specific healthcare organization, 36%, they was, they were facing 36% annual
data growth across multiple silos or silo legacy systems within the organization.
This patient records were scattered across multiple departments.
There is no way basically to bring the data together.
And what was the end result?
It was basically causing critical delays in, in care delivery.
So for example, if some doctor basically goes and requests for
a certain patient records for the organization to provide information
back, it takes a couple of hours.
That kind of, delay basically, causes, a huge gap in how patients, how
doctors engage with patients and try to provide the best care possible.
How go solve this problem?
It go basically, the Go Solution basically implemented robust
microservices architecture, connecting 17 disparate systems.
It developed real time HL seven FHAR translation engine with h HIPAA
compliant verification protocol.
Doing that.
it, it enabled lightning speed availability of data.
It also basically provided you 83 per percent reduction in the overall
clinical data retrieval time.
Comprehensive patient, as I mentioned earlier, now, the comprehensive patient
histories are now available in seconds rather than hours, which basically
improved overall care decisions.
Another case study now talking about the How Go has impact, how
a current problem in financial services and how go was able to help.
So in this financial institution, basically there are fragmented
regulatory compliances across 28 disparate systems creating, which was
creating significant operational burden.
Monthly compliance reporting require it took almost 160 hours staff hours to
basically create compliance reporting.
One compliance, one, one monthly compliance report, which
is insane with the existing market that we currently have.
What go did is basically deployed an image source data fabric with
cryptographically secured audited trails.
What is enabled is faster near real time availability of, of all of
the in data and enable compliance monitoring on top of it with
automatic automated anomaly detection.
it, it is basically a combination of how you have machine learning
and a good quality data with high availability and speed.
So what it did is basically it reduced the overall compliance reporting time
and 94% while eliminating manual error.
It enabled continuous risk assessments across all trading platforms with
millisecond level visibility.
That is the power of go.
Next, let's talk about basically, the how we arrived at cloud
data, native data architectures.
So as I mentioned earlier, one of the key features of, of Go is basically
to successfully integrate with these cloud native data architectures.
as you can see on the screen, the, how our cloud native has evolved as part of
the modernization of many organizations over the last couple of years.
It started with the traditional on-prem, then go with the hybrid cloud
system, then the multi-cloud system, then very cloud native capabilities.
So in here, basically if you see, goes lightweight concurrency model
and efficient resource utilization, make it exceptionally well suited
for cloud native data architectures.
In a typical cloud data, native data architectures, the
growth rate is around 14.2%.
CAGR.
By leveraging go's ability to support for containers, Kubernetes, microservice,
serverless architectures, and SDKs to integrate with cloud native, it
significantly reduce operational overhead and deployment complexities.
Then let's talk about go libraries for enterprise data.
So these are some of the key libraries that, that we need to focus on from
a, from GO'S capability perspective.
First in, in high power, high performance, ETL.
Goum provides advanced numerical com computations with near native speed.
GIA enables GPU Accelerator machine learning pipelines.
Nats delivers Ultrafast message streaming with 10, 10 million messages per second.
Second is governance spiff inspire, implement implements, zero trust identity
frameworks across distributed systems.
Ori Hydra provides scalable O-O-I-D-C authorizations.
Jagger Pro enables comprehensive data in tracing with microsecond precision.
All of these key components are what comprises of the entire data
F frame data governance framework.
The next is the data storage and retrieval.
Badger DP offers embedded key value storage with SSD performance optimized
performance enterprise Ready Go Client.
Seamlessly integrated with Kafka, Cassandra, MongoDB,
and Elasticsearch ecosystems.
The last piece of the cloud integration as we mentioned earlier, native go
SDKS for A-W-S-G-C-P and Azure deliver optimized cloud for cloud performance.
Terraform provides enable in as a code with declarative
data architecture deployments.
Next.
How do you basically come up with your with Go data strategy?
In a typical, ideal environment, very first thing is we do an assessment.
We basically go conduct a complete comprehensive data architecture audit,
identify critical performance bottlenecks, security vulnerabilities, and governance
gaps, and then we start to create a prioritized roadmap based off the business
impact and implementation complexity.
Then we basically come up with an MVP or a pilot.
We create a set, a high visibility constrained scope initiative for,
the initial go implementation, establish clear KPIs and benchmarks
against existing systems, and then develop internal technical expertise.
We showcasing early wins.
Then once we are able to successfully achieve that MVP,
then we expand and integrate.
So we basically use Go Powered and microservices
across, with prior data flows.
And start to scale it out across, across the other use
cases, within the organization.
And then you basically implement compress and data governance with
automated compliance controls.
And the last maturity level, transform then scale solutions to basically
enterprise by data, fabric architecture, establishing go powered foundation for
advanced AI MAL initiatives achieve 50% reduction in unutilized data.
While improving excess P 2%, 75%, this is where you get your
full near real time capabilities.
your transforming, ETL optimizations.
Then it also eventually feeds into your AI ML initiatives as well.
So overall, as I mentioned, go is a very powerful platform or a powerful
utility, which covers you many tools for you to satisfy in, in, in
building an entirely, robust, cost effective, scalable data architectures,
not just from 1 1 1 perspective.
There are multiple use cases within the industry that can be
solved using those architectures.
So looking forward to more questions.
Thank you for everybody's time.