Transcript
This transcript was autogenerated. To make changes, submit a PR.
I'm here to present in the platform engineering for FinTech.
We want to specifically discuss about database architecture in FinTech and
how this distributed database starting works, replication and scaling, and
how it helps the FinTech industry.
As we all know, modern FinTech platforms track air platform engineering solutions
that can process massive transaction volumes during peak events in milliseconds
while maintaining the accuracy.
Also, the availability, so as financial services scale globally across different
regions and environments which have different regulatory requirements.
Platform teams must architect distributed database system that
supports retail, blanking trading, digital balance, and payment
processing at unprecedented volumes.
So the agenda of this PPT is split into the six parts.
First we will discuss about the platform engineering foundation how it evolved
around in the financial infrastructure.
How the requirements are captured and scaled.
And what is the role of a database architecture here.
Then we will look at deeply into the database strategies.
We will start with the database strategy strategies.
We have a different starting strategies like horizontal partnering, partitioning.
Geographic or regulatory spreading on operational management, right?
So there are replication models, which can be synchronous or hybrid.
Whichever suits for the financial tech industries and consistency
model, consistency models, strong consistency, eventual consistency
and cashflow, consistency for different financial context.
So then the architectural patterns, how these database, and driven architectures
are used, like even driven architecture, microservices, cqrs, and even sourcing.
Then we look at some case studies and feature directions where this platform
at FinTech goes with respect to database and especially distributed database.
So let's dive in.
As the evolution of financial infrastructure, we know the financial
industry has undergo a significant transformation over the years on how
it approaches the data management and system architecture, right?
So example, traditional banking system built on monolithic
architectures at first have given a way to, given way to a distributor
systems and cloud native platforms.
Recently, platform engineering has emerged a critical discipline and focused
on creative self-service capabilities.
Modern FinTech platforms face multifaceted challenges recurring Highlander
scalability while maintaining the asset properties, which is critical for five.
FinTech industries.
High frequency trade trading system must process market data and
execute trades within microseconds.
Digital banking platforms need to handle millions of simultaneous user at a time.
Payment processes must validate and settle transaction across multiple currencies
and different countries, and which have a different regulatory jurisdiction
while maintaining the perfect accuracy.
So this is our revolution of the financial industry over the years
and the need now we will understand the scale requirements, right?
So accuracy and audit, auditability, financial transaction, demand up,
accuracy, auditability, and permanence.
Zero margin for error is acceptable in calculation and
comprehensive logging is mandated by regulatory and run requirements.
Extreme usage spikes, financial platforms must adaptably handle extreme usage
spikes triggered by market events, product launches, or seasonal activities
like tax season or height trading day.
And due to global events the architecture must ensure.
Consistent performance through robust auto-scaling capabilities.
And there is geographical distribution comes into play as well because
we have to comply with the data resiliency loss across diverse
jurisdiction of different countries.
Coupled with the need for low latencies global access
mandates, multi-regional database deployments for financial services.
With this in mind, the database from monolithic evolved into
database distributed database in the FinTech industries, and it is
critical role, it is playing critical role in serving the consumers.
As well as keeping the regulatory requirements, right?
So we will look at the different strategies involved in this.
First, the shredding strategies.
So under the shredding strategies, the horizontal partnering in partitioning
in financial context, right?
By distributing data across multiple database instances, sh shredding enables
linear scalability while maintaining the low latency QE performance.
The primary challenges lies in maintaining the transactional
integrity across shared boundaries.
So user based sharding is one of the most common in financial platform
because this partitioning based on user data or echo number can be executed
or operate within the single shot.
And all the information can be in single shot of a particular user, however.
Operations spanning multiple user accounts like payment from one account
to another which involves different charge repair, sophisticated distributed
transaction management to maintain performance as well as consistency.
So there is also a geographic and regulatory sharding.
First thing may be a regulation.
Regional compliance system care, right?
Geographic writing enables that strategy enables that regulatory
requirements, which aligns the database distribution with the
regulatory requirements, ensuring.
Customer data remains within specific geographic boundaries, right?
So then it can, it also enables the cross power transaction if you
have a geographic trading sharding payments between regions involved.
Data showed in different geographical chart require complex routing logic
and transaction coordination protocols.
Time based Charlie historical transaction data is critical in FinTech for the
statements, tax purposes, and others.
That can be partitioned by data ranges, enabling efficient archival
and compliance reporting while keeping recent data in high performance charts.
Effective chart management requires automated tools and processes that
handle operational complexity, including monitoring systems that
track shared health performance metrics and capacity utilization.
Short rebalancing represent on of the most complex operational challenges
as user bases grow unevenly.
Our transaction pay 10 changes due to various recent, so the next one is the
replication models for high availability.
So our, the replication models can be a synchronous or hybrid.
We will look at the synchronous replication first.
Synchronous replication for financial accuracy.
Synchronous replication provides the strongest to consistency guarantees,
ensuring all replicas contained.
Identical data at any point in time.
This is essential for critical financial operation where inconsistencies
could result in financial losses or regulatory violation.
Right?
Operation must be confirmed by all replicas before the transaction
is even considered complete.
Ensure any replica can immediately take over without
data loss during primary failure.
Introduces additional latency and complexity due to this, especially when it
is across different geographical location.
Also, multi-master synchronous replication presence, additional
complicate comp complexities.
In while doing the conflict resolution.
So then there is an asynchronous enables financial asynchronous
PLA replication enables financial platforms to achieve global scale while
maintaining acceptable performance.
Operations are committed to the primary database and then
propagate to replica synchronously.
It reduces the right latency significantly and enable geographic
distribution of real operations easily.
Introduce possibility of temporary data.
Inconsistencies between replicas requires careful monitoring of replication,
lag needs, sophisticated conflict direction and resolution mechanism.
If you are using a synchronous replication, which is the downside,
hybrid replication strategies modern financial platform often implement
these kind of hybrid aches combining synchronous and synchronous replication
based on data classification.
So this gives more beneficial of both worlds.
Critical financial data uses synchronous replication within a region and
thus a synchronous across region.
Less critical data may use a synchronous replication.
Exclusively sophisticated data classification and routing
system application must stand in different consistency models
for different data types.
So then there is the consistency models in financial system,
there is strong consistency.
Essential for code ranking system, trading system, and payment processing
where accuracy is paramount.
All nodes see the same data at the same time, require sophisticated coordination
protocol, prioritize consistency over stability, ual consistency, better
performance for analytics, reporting and customer facing application.
Tolerate temporary inconsistencies require conflict resolution.
Mechanism needs monitoring for replication lag.
Casual consistency.
So it's a middle ground for complex financial workflows,
ensures casually related operation.
Maintain order, allows unrelated operation to can be reorder, requires
timestamp and dependency tracking.
So moving onto the next topic, which is architectural pattern
for financial scalability.
First architectural pattern we can take as event driven architecture.
It is essential for building a scalable financial platform that handle complex
multi-step processes while maintaining loose coupling between services.
Even sourcing provides complete audit trials for regulatory complaints.
S careful attention to event ordering and delivery guarantees.
Even streaming platform like Kafka have become central to
many financial architecture.
For this reason, microservices architecture is paramount for any arc,
but it is more paramount to financial.
FinTech org as well.
It enables financial problems to organize complex business logic into manageable
independently deployable services.
Domain driven design helps identify appropriate service boundaries
require sophisticated transaction management across service boundaries.
Service MIS technologies provide essential capabilities for security and monitoring.
Then we have secures command query responsibility, segregation.
Which separates command processing from query processing, optimizing
each for specific requirements, right?
Operation.
Focus on maintaining consistency and business role while read operation.
Optimize for performance and user experience enables specialist three
models for different use cases.
Then there is e sourcing benefit.
It compliments the CQRS by providing reliable mechanism for
propagating state changes even generated by command processing.
Update read models are synchronously, provides natural
audit trails for compliance.
Enables creation of new read models from historical even streams, requires
sophisticated, even processing pipelines to implement this.
Then there is an important topic in the FinTech industry which is always
a talk as security and compliance.
Which is critical for the FinTech industry while maintaining the
customer data and information.
So there is an encryption and data production.
Financial data occurs production both at risk as well as in transit.
Transparent data encryption are for stored data, secure communication
between our system components, hardware security models for key management.
Access control, fine grain mechanism.
Ensure data is only accessible to authorized user, road based and attribute
based access control MFA, multifactor authentication for sensitive systems, road
level and carbon level security policy also being implemented in various places.
Regulatory complaints.
Financial platforms must comply with numerous frameworks
and data Ency requirement.
Private privacy regulations like G gt, GTPR, detailed audit, logging,
logging and reporting capabilities for tax purposes and others.
So we will look at the case studies, real world implementation,
high frequency trading platform.
Major investment bank, implemented distributed architecture, acquiring
microsecond latencies, and perfect consistency, geographic distribution
with co-located databases, instances hybrid approach, which we
discussed in the previous slides.
Combining synchronous application within region and as synchronous
across different region, sub millisecond and performance monitoring.
With automated resource allocation, digital banking platform leading
digital banking platform.
Built globally distributed architecture, supporting millions of
customer sh trying strategy based on geography, location and account type.
Human driven architecture with comprehensive human sourcing, eventual
consistency for user experience and strong consistency for financial operations.
Money transfer stuff, payment processing network, global processor handling peak
volumes with sub-second authorization times combination of graphicy and
merchant based Strat, which is user based Strat real time fraud reduction with
machine learning, disaster recovery, redirecting reprocessing within seconds.
So where the future hits too, right?
With all this infu in financial platform engineering.
Cloud native database technologies, right?
Evolution towards technology, designed for distributed container environment,
serverless database, eliminating operational overhead, multi-cloud
strategies, balancing performance, cost and compliance, integrated
security and compliance capabilities.
Also as talk of the towns NA and ML integration as well,
and quantum security, right?
Transformative capabilities being integrated directly into database system.
Automated performance tuning and anomaly reduction by a real time
fraud reduction and risk assessment.
Quantum resistant cryptographic algorithms is being researched and
then hybrid security approaches during transition period.
So that's all for the presentation today.
Thank you.
Thanks for listening to this.