Conf42 Platform Engineering 2025 - Online

- premiere 5PM GMT

Platform Engineering for FinTech: Database Sharding, Replication & Scale

Video size:

Abstract

Learn how platform engineers architect distributed databases for FinTech at 65K+ TPS scale. Discover sharding, replication, and consistency patterns that power global financial platforms while meeting strict SLAs and regulatory requirements.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
I'm here to present in the platform engineering for FinTech. We want to specifically discuss about database architecture in FinTech and how this distributed database starting works, replication and scaling, and how it helps the FinTech industry. As we all know, modern FinTech platforms track air platform engineering solutions that can process massive transaction volumes during peak events in milliseconds while maintaining the accuracy. Also, the availability, so as financial services scale globally across different regions and environments which have different regulatory requirements. Platform teams must architect distributed database system that supports retail, blanking trading, digital balance, and payment processing at unprecedented volumes. So the agenda of this PPT is split into the six parts. First we will discuss about the platform engineering foundation how it evolved around in the financial infrastructure. How the requirements are captured and scaled. And what is the role of a database architecture here. Then we will look at deeply into the database strategies. We will start with the database strategy strategies. We have a different starting strategies like horizontal partnering, partitioning. Geographic or regulatory spreading on operational management, right? So there are replication models, which can be synchronous or hybrid. Whichever suits for the financial tech industries and consistency model, consistency models, strong consistency, eventual consistency and cashflow, consistency for different financial context. So then the architectural patterns, how these database, and driven architectures are used, like even driven architecture, microservices, cqrs, and even sourcing. Then we look at some case studies and feature directions where this platform at FinTech goes with respect to database and especially distributed database. So let's dive in. As the evolution of financial infrastructure, we know the financial industry has undergo a significant transformation over the years on how it approaches the data management and system architecture, right? So example, traditional banking system built on monolithic architectures at first have given a way to, given way to a distributor systems and cloud native platforms. Recently, platform engineering has emerged a critical discipline and focused on creative self-service capabilities. Modern FinTech platforms face multifaceted challenges recurring Highlander scalability while maintaining the asset properties, which is critical for five. FinTech industries. High frequency trade trading system must process market data and execute trades within microseconds. Digital banking platforms need to handle millions of simultaneous user at a time. Payment processes must validate and settle transaction across multiple currencies and different countries, and which have a different regulatory jurisdiction while maintaining the perfect accuracy. So this is our revolution of the financial industry over the years and the need now we will understand the scale requirements, right? So accuracy and audit, auditability, financial transaction, demand up, accuracy, auditability, and permanence. Zero margin for error is acceptable in calculation and comprehensive logging is mandated by regulatory and run requirements. Extreme usage spikes, financial platforms must adaptably handle extreme usage spikes triggered by market events, product launches, or seasonal activities like tax season or height trading day. And due to global events the architecture must ensure. Consistent performance through robust auto-scaling capabilities. And there is geographical distribution comes into play as well because we have to comply with the data resiliency loss across diverse jurisdiction of different countries. Coupled with the need for low latencies global access mandates, multi-regional database deployments for financial services. With this in mind, the database from monolithic evolved into database distributed database in the FinTech industries, and it is critical role, it is playing critical role in serving the consumers. As well as keeping the regulatory requirements, right? So we will look at the different strategies involved in this. First, the shredding strategies. So under the shredding strategies, the horizontal partnering in partitioning in financial context, right? By distributing data across multiple database instances, sh shredding enables linear scalability while maintaining the low latency QE performance. The primary challenges lies in maintaining the transactional integrity across shared boundaries. So user based sharding is one of the most common in financial platform because this partitioning based on user data or echo number can be executed or operate within the single shot. And all the information can be in single shot of a particular user, however. Operations spanning multiple user accounts like payment from one account to another which involves different charge repair, sophisticated distributed transaction management to maintain performance as well as consistency. So there is also a geographic and regulatory sharding. First thing may be a regulation. Regional compliance system care, right? Geographic writing enables that strategy enables that regulatory requirements, which aligns the database distribution with the regulatory requirements, ensuring. Customer data remains within specific geographic boundaries, right? So then it can, it also enables the cross power transaction if you have a geographic trading sharding payments between regions involved. Data showed in different geographical chart require complex routing logic and transaction coordination protocols. Time based Charlie historical transaction data is critical in FinTech for the statements, tax purposes, and others. That can be partitioned by data ranges, enabling efficient archival and compliance reporting while keeping recent data in high performance charts. Effective chart management requires automated tools and processes that handle operational complexity, including monitoring systems that track shared health performance metrics and capacity utilization. Short rebalancing represent on of the most complex operational challenges as user bases grow unevenly. Our transaction pay 10 changes due to various recent, so the next one is the replication models for high availability. So our, the replication models can be a synchronous or hybrid. We will look at the synchronous replication first. Synchronous replication for financial accuracy. Synchronous replication provides the strongest to consistency guarantees, ensuring all replicas contained. Identical data at any point in time. This is essential for critical financial operation where inconsistencies could result in financial losses or regulatory violation. Right? Operation must be confirmed by all replicas before the transaction is even considered complete. Ensure any replica can immediately take over without data loss during primary failure. Introduces additional latency and complexity due to this, especially when it is across different geographical location. Also, multi-master synchronous replication presence, additional complicate comp complexities. In while doing the conflict resolution. So then there is an asynchronous enables financial asynchronous PLA replication enables financial platforms to achieve global scale while maintaining acceptable performance. Operations are committed to the primary database and then propagate to replica synchronously. It reduces the right latency significantly and enable geographic distribution of real operations easily. Introduce possibility of temporary data. Inconsistencies between replicas requires careful monitoring of replication, lag needs, sophisticated conflict direction and resolution mechanism. If you are using a synchronous replication, which is the downside, hybrid replication strategies modern financial platform often implement these kind of hybrid aches combining synchronous and synchronous replication based on data classification. So this gives more beneficial of both worlds. Critical financial data uses synchronous replication within a region and thus a synchronous across region. Less critical data may use a synchronous replication. Exclusively sophisticated data classification and routing system application must stand in different consistency models for different data types. So then there is the consistency models in financial system, there is strong consistency. Essential for code ranking system, trading system, and payment processing where accuracy is paramount. All nodes see the same data at the same time, require sophisticated coordination protocol, prioritize consistency over stability, ual consistency, better performance for analytics, reporting and customer facing application. Tolerate temporary inconsistencies require conflict resolution. Mechanism needs monitoring for replication lag. Casual consistency. So it's a middle ground for complex financial workflows, ensures casually related operation. Maintain order, allows unrelated operation to can be reorder, requires timestamp and dependency tracking. So moving onto the next topic, which is architectural pattern for financial scalability. First architectural pattern we can take as event driven architecture. It is essential for building a scalable financial platform that handle complex multi-step processes while maintaining loose coupling between services. Even sourcing provides complete audit trials for regulatory complaints. S careful attention to event ordering and delivery guarantees. Even streaming platform like Kafka have become central to many financial architecture. For this reason, microservices architecture is paramount for any arc, but it is more paramount to financial. FinTech org as well. It enables financial problems to organize complex business logic into manageable independently deployable services. Domain driven design helps identify appropriate service boundaries require sophisticated transaction management across service boundaries. Service MIS technologies provide essential capabilities for security and monitoring. Then we have secures command query responsibility, segregation. Which separates command processing from query processing, optimizing each for specific requirements, right? Operation. Focus on maintaining consistency and business role while read operation. Optimize for performance and user experience enables specialist three models for different use cases. Then there is e sourcing benefit. It compliments the CQRS by providing reliable mechanism for propagating state changes even generated by command processing. Update read models are synchronously, provides natural audit trails for compliance. Enables creation of new read models from historical even streams, requires sophisticated, even processing pipelines to implement this. Then there is an important topic in the FinTech industry which is always a talk as security and compliance. Which is critical for the FinTech industry while maintaining the customer data and information. So there is an encryption and data production. Financial data occurs production both at risk as well as in transit. Transparent data encryption are for stored data, secure communication between our system components, hardware security models for key management. Access control, fine grain mechanism. Ensure data is only accessible to authorized user, road based and attribute based access control MFA, multifactor authentication for sensitive systems, road level and carbon level security policy also being implemented in various places. Regulatory complaints. Financial platforms must comply with numerous frameworks and data Ency requirement. Private privacy regulations like G gt, GTPR, detailed audit, logging, logging and reporting capabilities for tax purposes and others. So we will look at the case studies, real world implementation, high frequency trading platform. Major investment bank, implemented distributed architecture, acquiring microsecond latencies, and perfect consistency, geographic distribution with co-located databases, instances hybrid approach, which we discussed in the previous slides. Combining synchronous application within region and as synchronous across different region, sub millisecond and performance monitoring. With automated resource allocation, digital banking platform leading digital banking platform. Built globally distributed architecture, supporting millions of customer sh trying strategy based on geography, location and account type. Human driven architecture with comprehensive human sourcing, eventual consistency for user experience and strong consistency for financial operations. Money transfer stuff, payment processing network, global processor handling peak volumes with sub-second authorization times combination of graphicy and merchant based Strat, which is user based Strat real time fraud reduction with machine learning, disaster recovery, redirecting reprocessing within seconds. So where the future hits too, right? With all this infu in financial platform engineering. Cloud native database technologies, right? Evolution towards technology, designed for distributed container environment, serverless database, eliminating operational overhead, multi-cloud strategies, balancing performance, cost and compliance, integrated security and compliance capabilities. Also as talk of the towns NA and ML integration as well, and quantum security, right? Transformative capabilities being integrated directly into database system. Automated performance tuning and anomaly reduction by a real time fraud reduction and risk assessment. Quantum resistant cryptographic algorithms is being researched and then hybrid security approaches during transition period. So that's all for the presentation today. Thank you. Thanks for listening to this.
...

Karthickram Vailraj

Senior Manager @ Charles Schwab

Karthickram Vailraj's LinkedIn account



Join the community!

Learn for free, join the best tech learning community

Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Access to all content