Transcript
This transcript was autogenerated. To make changes, submit a PR.
Greetings everyone.
Welcome to Con 42, machine Learning 2025 s. I'm Srin.
Was ami, director of Technology Portfolio at SPL Consulting Inc. With
over 23 years of experience in it.
Specialist in enterprise cloud architecture, multi-cloud
integrations, data management.
I also have specialization in financial systems, Oracle Cloud,
ERP, and payment processing.
Do you know that over 67% of multi-cloud integration projects
implementation overrun because of the challenges in the data and the
complexities in the integrations?
So today's sessions, we will cover mastering multi-cloud ERP
integration, your framework for data normalization and synchronization.
So ERP implementations and the systems evolved over the period of time.
From monolithic architecture to highly distributed cloud native implementations.
So organizations are adapted to implement multiple clouds based on the domain
specialization, rather than implementing a single ERP or single cloud ERP.
So example, if you take financials, Oracle Cloud, ERP is based in
the market, whereas Workday is for the human capital management.
Salesforce for customer relationship management and bully under
for supply chain optimization.
So if the enterprise want to use best of the domain cloud ERP system, then
implementing these multi-cloud in the enterprise ecosystem is inevitable.
So this strategic approach delivers a superior domain specific functionality
for the enterprises and the business.
However, it also brings lot of challenges and data models, inconsistent APIs
and varying issues across the board.
Let's us look at this multi-cloud ERP ecosystem.
Oracle Cloud ERP, specialized in financial domain is implemented by almost 41
percentage of the Fortune 500 companies.
Is, so it has various modules, example, general ledger, account receivable,
account payables, fix address, and it also has superior functionalities
of financial consolidations and reporting functionalities.
Workday is a human capital management specialized ERP, and it
has unified data architecture for.
The complete workforce operations for any enterprises, it holds over 38
percentage of market share across the enterprises with over 10,000 employees.
However, Workday is using so based web services with the complex
external schemas for their data needs.
Salesforce is customer relationship management.
ERP system, almost 56 percentage of the multi-cloud ERP ecosystem
holds Salesforce for its CRM needs.
The Salesforce ecosystem uses even driven architecture generating almost
two 12.4 million even notifications daily in large scale implementations.
On the other hand.
Is specialized in supply chain orchestration with over 28
percentage of market penetration among global enterprises.
If you look at these, all these ERP systems, they're superior in functionality
in their respective domain and almost have a very good market share across
the Fortune 500 and other enterprises with the large scale operations.
So the enterprise has adopted these different ERP systems to support
their functionalities and their domain needs, which gives the complexity.
The very first challenge, which we are going to look at
multi-cloud ERP implementation is schema heterogeneity challenges.
We looked at why enterprises.
Or adapting to implement multiple cloud ERP to gain the superior
functionalities available in each domain.
However, the fundamental challenge in multiple cloud ERP integration lies in
reconciling their divergent data models.
Each cloud ERP provider, they designed their data model with
different architecture philosophies to support and gain the.
Superior functionality in each of the domain, though this gives very
good architecture and data model for individual Cloud ERP, integrating
these data models across the cloud.
ERP is always complex and provides challenges.
The substantial barrier.
Is to integrate or exchange this data, which has different
schema, our properties.
If you look at it, the Oracle Cloud ERP, which holds the financial
data, has complex chart of account segments, lot of business rules.
However, these start off accounts and business rules are
exchanged with other cloud ERP to derive accounting information.
Similarly, the workday, which holds the employee data, has 23 related objects
with over 200 plus attributes per record.
When this is exchanged with other cloud ERP in the ecosystem, they
do not need all these attributes.
However, the exchange substantially become complex.
To get any user related information from Workday,
the blue yonder system, which holds the product informations, there are a lot of
master repositories and syncing them with other cloud ERP, bringing lot of latency.
The CRM, which holds the partner data, it has a lot of fragmented attributes.
Which also brings the complexity across the cloud ERPs.
So this schema heterogeneity
always provides some challenges which needs to be solved.
Semantic mapping complexities beyond the structural differences, which we saw the
semantic variations compound normalization challenges in multi-cloud ERP ecosystem.
Even when the systems appear to represent similar business concepts,
subtle differences in terminology, hierarchies, and business rules create
significant integration obstacles.
Let us look at this example of how the business partner represented across
multiple cloud ERP systems, the supplier representation in Oracle, whereas.
In Oracle Cloud, ERP the financial relationship payment terms and tax
information and accounting classification of the suppliers are maintained where the
enterprise do business with any purchasing relationship is called supplier.
However, the same supplier is represented as account in Salesforce,
whereas the primary attributes around these accounts are.
Related to relationship attributes, how we communicate with them, what
is the cases with them, what is their histories and what are the
opportunities we have with them?
It's completely different than what it is being represented
in Oracle, the same supplier, which is mentioned in Oracle now.
If you look at the blue yonder, it's called vendor.
Whereas the primary focus or the attributes for the blue yonder system is.
The logistic performance metrics, delivery capabilities,
and supply chain reliability.
So the semantic reconciliation, almost 43 percentage of development
effort is spent on this.
We have seen schema heterogeneity challenges and
semantic mapping complexities.
However, organization have developed several sophisticated technical
mechanisms to address these challenges.
How these challenges or approaches vary in complexity and architectural
impact with the selection, typically based on specific integration
requirements and technical constraints.
Also, based on the organization's capabilities
there are.
Predominant options available for us to handle these challenges.
The first and widely used option across the enterprises are EDL.
So you extract the data from the source cloud, ERP, transform it to the target,
and finally load the transform our map data in, into the target cloud, ERP.
So this ETL option is widely used and very simple.
In terms of the implementation, however, most of the work are mapping
are being done in the transform layer.
The second option, which we have is schema mapping engines.
So you have sophisticated engine which take care of all your
heterogeneous schema, mappings, and.
Across different systems.
Cloud ERP systems, it reduces the development effort because the engine
provides the mapping for various cloud ERPs, which can be adapted by different
integrations and seamlessly will work.
The canonical data models are evolving as a trend and standardization.
To normalize these complexities, whereas each source ERP system, the cloud ERP
system, which is master in their data,
agree with other cloud ERP systems across the enterprise to standardize the format
or the canonical data model and publish them as a publisher subscriber model.
So that the consumer will have standardized data and attributes
available in the form of the canonical data model to them.
So this takes lot of initial design complexity.
However, in the long run, it reduces the overall, the mapping
or the implementations record by the consumers in the latter part.
Latency inconsistency challenges in the distributed multi-cloud ERP system.
Various data elements are shared across ERP systems, thus bring a significant
timing challenges that can compromise the business process Integrity.
If you look at it approximately around 68% of the organization's report.
Their business impact from timing related data ies across
integrated cloud platforms.
These delays can occur due to various reasons.
Example, there is a network latency or large processing queues between one
cloud ERP to other cloud ERP, or there are validation rules which the data
will not go through to the target cloud.
ERP.
Sometimes there are middleware transformation overhead.
All of these causes two things.
One is either there is a latency, sometimes there is a integrity challenges.
So if you look at it, transaction propagation delays, there are
almost eight to 15 seconds is for the standard transactions.
Up to few minutes for the complex ones across the cloud.
ERP systems, the eventual consistency models.
There are 47% of the data entries operate under even true consistency.
The even best triggers are very complex.
It brings overhead to to the cloud ERP data model.
High volume data streaming also contributes to these latency
and consistency challenges
with latency under data integrity challenges.
There needs a conflict less solution strategy.
So in the multi-cloud ERP, there are a lot of bidirectional data
flows between the cloud ERP systems.
First, we have to detect the conflict on time and also need to provide you proper
resolutions to handle this integrity and the data conflict resolution.
The research indicates that approximately.
Six to 8% records in the bidirectional synchronization scenarios.
Experience these update conflicts.
This postage may increase up to 12 to 15% if the data is getting modified frequently
across the multi cloud ERP systems.
So there are certain strategies available for us to resolve this conflict.
The widely used.
Approach.
Our strategy is time-based.
Resolution.
What it means is always giving the precedence to the most recent update.
If there is a conflict between multiple updates, ignore the one
which we received earlier, go with the latest or recent update.
So this is called time-based time timestamp based approach.
The second one, which we have is domain specific rules.
Say example, when there are conflict error between the data element, the
domain specific rule or the data element wins that conflict and it takes
the priority over the other element.
Say example, if accounting financial system, Oracle Cloud,
ERP, is the master or the supplier.
If Blue Yonder is the master or.
The customer, if Salesforce is the master, the domain specific
system takes the precedence over the other systems and holds that
attribute are the data integrity.
So that is the domain specific conflict resolution approach.
The next one is manual reconciliation.
Say example, you cannot use time-based, or we cannot use domain specific rule.
However, it needs a manual intervention to reconcile the data to understand
which takes precedents and apply it.
There is still a 15% adoption to this strategy.
The last one, which is conflict free, replicated data types is the industry
standard, which we want to go to.
It is 8% adoption.
The way it is implemented will avoid the conflict and it'll be conflict free.
Replicated data type
framework for synchronization.
An effective synchronization framework for the multi-cloud ERP
system must incorporate several architectural components that
collectively address complex.
Requirements for data consistency and re resilience organizations implementing
comprehensive frameworks experience approximately 76% fewer data consistency
incidents compared to those relying on point to point integrations.
If you look at this different architecture or framework, the
Oracle has a B, C, C connector.
Which simplifies the extraction.
Thus, it reduces the extraction complexity by 45%.
If you look at the sales force, which has this change data capture, it reduces
almost 97% in the data volume compared to full synchronization, the even streaming
backbone for bullion, which publishes the data or stream the data into Kafka.
Also almost process 50,000 to hundred K messages per second.
The smart batching strategies, which improves almost one 20%
with the adaptive approaches.
Adapting your right framework and strategy between the multiple cloud
ERP based on their capabilities.
Will improve the data synchronization and reduce the data inconsistency
issues while we are doing data normalization approaches and solution
for heterogeneous schema challenges and semantic mapping complexities, and also
synchronization framework to address data inconsistency and latency issues.
One should not overlook the security issues.
The multi-cloud ERP system brings in, compared to the single cloud
deployment are the on-premise deployments, which are more protected.
The exchange of data between the multi-cloud ERP system, especially the
sensitive data, brings more vulnerability.
So approximately 41 percentage of security in more security incidents
related to the data exchange in the multi-cloud ERP system compared
to the single cloud ERP systems.
So we need to have some security consideration.
Those are like, say cross platform authentication.
So how do we authenticate between multiple cloud ERP systems, the widely used.
Authentication, make comment 2.0, which is almost 82%.
Enterprises adapted these kind of approaches to secure their multiple
cloud ERP systems and its data.
However, there are 72% multi-cloud ERP implementer encounter
regulatory related violations, okay.
During the initial deployment.
So automated compliance monitoring reduces these kind of security incidents.
Almost by 63%.
The data, which is tranent because there are a lot of data exchange
between the multi-cloud ERP system should be encrypted so that tss
misconfigurations are outdated security.
Cheaper suits.
Those are contributing almost 28% of these kind of integration incidents.
Okay, so the organizations which are adopting layered encryption
approaches experience 52% fewer data exposure incidents.
The last one, data drift detection.
So 24% of integration failures attributed to undetected schema changes between
these multiple cloud ERP systems.
So organizations implying automated drift detection.
Identify 83 percentage of potential issues before it impacts the business.
The proposed implementation strategy to address these multi-cloud
ERP data integration issues.
A faced approach to address these normalization and synchronization
challenges provides the necessary structure to manage the complexity
of multi-cloud ERP integration.
So organizations implying structured approaches experience approximately 43%
higher success rate than the organizations those attempting to concurrently
implementing these strategies.
If you look at the strategy.
The first step should be the data profiling and mapping
comprehensive analysis on the data.
And its mapping is very much record.
They should be the initial phase of the project.
It also should understand the quality data, quality issues,
what should be addressed.
Once the data profiling and mapping is done, then enterprise
should move on to make sure to design the canonical model design.
So spending quality time in this phase to understand the actual canonical
model, which will be used by multiple cloud ERPs in the ecosystem will
benefit the enterprise in the long run.
Once the canonical model design is done, now the enterprise should
carefully choose the integration burden across the cloud ERP systems.
Choosing the right integration burden will solve many of the data
latency and inconsistency issues.
Once organization completed this phase, we should move on to.
Synchronization mechanism implementation.
Almost 63 percentage of projects encounter significant technical obstacles.
During this phase.
Incremental deployment achieves more frequent delivery milestones.
Then once all these phases are achieved, the enterprise
continue to monitor and govern.
All these implementations and synchronization processes and frameworks,
so the comprehensive monitoring reduces almost 54 percentage of their
meantime rate to their resolution.
Effective implementations capture 25 to 40 distinct metrics per integration flow.
Data profiling and mapping constitute the essential face-to-face in
multi-cloud ERP integration.
Providing a very comprehensive analysis of the data structures,
relationships, and business rules across the platforms these foundations
ensures the subsequent integration design decisions are based on accurate
understanding of the data landscape.
So there are various profiling tools available in the market which
automatically analyze the sample dataset.
To identify the patterns, relationships, and quality characteristics that
influence the integration design.
Apart from doing this automated schema analysis, we should also focus
on collaborative mapping, effective profiling actions beyond simple structural
analysis to include semantic evaluation, business role, and documentation,
and quality assessment in this phase.
Typically Bo involves both technical specialist and also the domain business
domain expert who provide the crucial context about the data quality assessment.
Also very key in this data profiling and mapping phase.
The next implementation phase is canonical model design phase.
The canonical model design represents a strategic architectural decision
that significantly influence long-term integration sustainability.
This phase focuses on the development of intermediate data models that
bridge the platform specific schemas, providing a consistent reference
point for all integration flows.
If you look at it, the canonical model design phase.
Almost reduces 37 percentage of them maintenance costs for the integrations
across the multiple cloud ERP systems.
In a typical large scale enterprise multi-cloud ERP system implementation,
there will be around 1 22 3 50.
Different canonical definition needs to be done for various data
elements which should be shared across the multiple cloud ERP system.
However, considering these large scale implementations and the effort needed to
implement this canonical model design, there is almost a 25 to 30 percentage
of initial effort increase will be experienced in this implementation phase.
The next and crucial phase in the implementation approach is
integration pattern, selection phase.
Enterprise should carefully select the integration pattern for the various data
elements, which is shared between the multiple cloud ERP systems, based on that
timing constraints, when the data should be available, whether it is real time or
batch based on the volume, whether it's a low volume data exchange or the high
volume data exchange between the multiple cloud ERP and what are the transformation
complexity associated with this.
Data elements, and also if there is a data exchange failure between
the cloud ERP systems, what are the recovery mechanism which is needed?
All these inputs will determine the right integration pattern to be selected
by the enterprise for a particular data element in a large scale enterprise,
multi-cloud ERP implementation, typically there will be five to eight
distinct integration patterns used for various data elements needs.
What are the future directions and conclusions?
So the AI driven mapping tools which are available now in the market, will
use machine learning approaches to automate most complex mapping task and
reduce the implementation time, and also improve the accuracy through patent
organization and suggestion capabilities.
Also industry is evolving the standardized exchange format for
the data elements, which will reduce all the custom transformations
and mapping, which is needed.
The last one is the advanced conflict resolutions.
The Next Generation Conflict Restoration Tools using the machine
learning capabilities will know the context awareness, thus provide.
Yeah, better reconciliation and concrete resolution.
Mechanism.
Key takeaways from this conference.
Note, multi-cloud ERP integration requests, careful orchestration
of data, normalizations, semantic mapping, and synchronous approaches.
We have seen a lot of challenges which needs to be addressed
through these approaches, and also the success depends on them.
Balancing specialized to functionality with the cohesive
data flow across platforms.
The framework, or the implementation approach we have outlined provides a
structured methodology for tackling these complex challenges while maintaining
the security and performance in line.
So begin with the comprehensive data profiling, develop robust
canonical models and select appropriate integration patterns.
For each data flow scenario in a multi-cloud ERP ecosystem.
Thanks everyone for spending time with me to go through this conference note.