Conf42 Observability 2025 - Online

- premiere 5PM GMT

Observability in Modern Treasury Trading Platforms: Monitoring High-Frequency Systems at Scale

Video size:

Abstract

Starts with a hook about the massive scale (8.4TB of data)

Creates urgency by mentioning the high financial stakes

Highlights the two most compelling technical elements (ML anomaly detection and distributed tracing)

Includes an impressive metric (78% reduction in resolution time)

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Everyone. In today's session we'll develop into the fascinating world of observability in modern pressure trading platforms. These systems, which handle vast amounts of market data and execute rates in microseconds face immense challenges when it comes to maintaining performance. As the complexity of high frequency trading increases, traditional monitoring solutions are no longer sufficient. Through this discussion, we'll explore how advanced observability tools like distributed tracing, machine learning driven anomaly detection, cloud native observability, pipelines, transforming how we monitor. Let's dive into how observability is revolutionizing high trading and shaping the future of treasury operations. Hello, I'm OU and senior with over 23 years of experience in the investment banking technology domain, specializing in delivering high performance, cost effective technology solutions tailored to meet complex definite needs. Across fixed trading, risk management, regulatory compliance, client reporting, and having worked with industry leading products such as Ion Sunguard, fast intrad Trade Web, Bloomberg, and specialized like Wall Street. Which is a product of Aon. I have involved in design, development and optimization of a trading strategies, leverage advanced quantitative models and market data analytics. My work includes implementing high frequency trading algorithms, market making strategies. Trading strategies that have significantly improved execution, efficiency and profitability. My product knowledge or experience encompasses a range of financial products, including US treasuries. Be it swift messages or ISO related packs or messages. I have extensive experience supporting all of them. Both my career, I have built reputation for managing lining and optimizing advanced technology infrastructures that. Deep dive into how this availability is revolutionizing high sequential trading and shaping the future of how pressure operations will be vulnerability in modern pressuring trading platforms. So drawing from my experience building and ture at major financial institutions. I will share how modern observability practices are revolutionizing operations in systems where reliability directly impacts the financial outcome. Okay, we'll examine how observability solutions monitor systems processing each 0.4 terabytes of market data daily with algorithmic engines. 0.5% of treasury at speeds of 45 microseconds. Unbelievable amount of data that is being. Join me as we dive into the technical components that enable this performance and. Of the modern training more than 8.4 terabytes. We recently implemented database using KD, which used to process about three to 600 million a day. For several dealer to dealer markets, 52.5% of algo trading algorithm, big trading percentage of tertiary trade executed by the algorithms. We also try to integrate smart order router, which process more than 75% in few of the exchanges. Famous like Nasdaq. A broker dealer and and these are the exchanges. Execution is latency. 85 key messages per second. That's the throughput of market data in the peak conditions. The pure volume and velocity of modern pressure trading creates unprecedented observability challenges with microsecond level predict making and execution. Predictional monitoring approaches simply cannot. These systems operate at a scale where even minor performance and degradations can result in significant financial demand. Moving on to the next slide. So attributed RAC in treasury training. So basically this is the empty mean time to resolution or resolution. Implementation of distributed tracing across trading microservices has TR by 78%, enabling faster response and critical trading operations. End-to-end visibility races now connect across previously siloed components, providing transparency from market data. Very key for locking the trades where you place an order and it has to get filled to get the right match place. Analytics identify bottlenecks in the trading pipeline, allowing for targeted optimizations that have improved average execution times by 23% low. Tracing ads less than 2.5 microseconds of overhead. Maintaining critical performance while providing observability, distributed ing has transformed troubleshooting trading platforms, enabling teams to follow transactions across complex, massive microservice architectures without compromising performance. Anomaly detection within learning. So our ML powered anomaly detection systems that achieve 85% accuracy in identifying trading pattern deviations. This capability has prevented millions in potential trading losses. By detecting market dislocations, execution anomalies, compliance issues before they impact trading outcomes. The system continuously learns from both normal operations and detected anomalies, improving detection accuracy over time while maintaining performance under. So as the diagram illustrates realtime pattern recognition identifies market anomalies within microseconds, the historical baseline comparison, compare current normalized patterns, false positive reductions, learning algorithms, minimize alert, automated response actions, triggers, safeguards. So here I wanna touch upon the need for observability for trade surveillance and spoofing in this fast evolving landscape of treasury trading. The need for robust trade surveillance and observability has never been more critical with high frequency systems. Executing thousands of trades per second. The ability to monitor and track every transaction in real time is essential for ensuring market integrity, regulatory compliance, and risk mitigation. Surveillance systems identify potential market. Fraud or compliance violations. While observability tools provide end-to-end visibility into the trading process, allowing firms to detect anomalies, optimize performance, and minimize latency, the integration of machine learning for anomaly detection combined. Market activity and the system health. This powerful combination ensures that trading platforms are not only operationally efficient, but also compliant with evolving regulations in modern treasury trading platforms, mobility growth. Beyond this monitoring because proactive approach that helps organizations detect potential issues before they affect the trading outcomes. Ensuring that systems can continue to operate at a scale with the reliability and security. Moving on to the next slide, cloud native observability pipelines. So today we have ma on web services. Today we have GP we. So our cloud native observability pipelines have transformed monitoring increasing capabilities by almost 30% while reducing operational cost by percent. These pipelines process over 36 million metric data points per minute, enabling real-time visibility across global trading operations. By separating collection from processing and storage, the architecture maintains a resilience during market volatility when observability becomes most critical, depends on all these. So it's like data collection, processing, enrichment, storage and indexing, visualization and alert. So by data collection, the invest metric logs, traces from the system at source, process enrichment like izing data with business metadata and producing the noise storage. Optimizing for both realtime queries and historical analysis. K db the pick database or MongoDB, the NoSQL database are completely efficient and for such observability pipelines. Moving on to the next slide. Smart Router integration, as the. Algorithmic trading where, be it the dealer to dealer or a dealer to fly market, the algorithm Smart order decides how the order would get routed based on the, and it's smartly tax and allocate the is required by the. Round time measurement, continuous monitoring of auto transmission and acknowledgement agency venue responsiveness, measuring performance variations between paying destinations, flip page analysis, realtime comparison of expected results, execution prices. This is very key because sometimes the mass the smart order router may or may not execute, add the desire to fill for that particular instance, which the Desco wants. So the realtime comparison of the expected versus actual execution crisis is very critical, which has to be observed. This integration creates powerful feedback loops that continuously optimize order routing decisions based on real-time performance data. Our implementation has improved overall execution quality by 17%, while reducing diverse selection by monitoring venue specific patterns that might indicate information leakage or toxic flow. Moving on to the next slide. So this is basically the system latency variations that is described here where the heat load is shown by the light orange and baseline latency in the red. The pressure trading systems experience latency variations of up to 20% under three flow conditions, particularly during the market openings. Economic announcements, auctions. Any key IPOs that go online. Observability solutions must adapt to these variations without introducing performance. In, in this, if you notice at 9:00 AM 9:30 AM the peak volatility latency is gone. During it has gone down and during the market close. Quantum computing advances, cryptographic protections. Trading platforms are implementing quantum resistant approaches to monitoring these techniques ensure the sensitively trading data remains secure. Even as cryptographic standards. You all early implementations have demonstrated that quantum resistant approaches can be. Integrated with minimal performance impact by providing future proof security guarantees. Homomorphic encryption, enabling analysis of encrypted metrics without decryption, maintaining privacy while allowing alerting on sensitive trading data. This technology for both compliance with data regulations while. Quantum random number generation leveraging quantum approach. Implementing lattice based cryptographic algorithms to secure monitoring data against potential quantum attacks. These algorithms ensure that today's encrypted observability data remains protected against decryption. This is the post Quantum. On the next slide. Blockchain based audit systems blockchain based audit systems are transforming ti market infrastructure by creating the immutable records of all trading activities. Systems have reconciliation cost by 82% while providing near realtime catch visibility across the parties. Integration of observability data with blockchain records creates powerful new capabilities for regulatory reporting, compliance monitoring, allowing for automated deduction of potential market manipulation or trading goal violations. So realtime settlement visibility plans, packing of lifecycle, trade lifecycle is the key. Back in straight through process rates will impact the reader books, which may or may not be found out until the end of the day. So this is very critical. Reduce reconciliation needs single source of across counter parties. Enhanced compliance, monitoring, compliance, and regulatory are becoming big. All these iso, all the standards are becoming really big. So automated re. And as I mentioned earlier, immutable audit trail, graphically secured transaction. So what are the challenges in observability? Performance impact data volume management, signal versus noise. So observability instrumentation adds overhead that can affect vacancy. Solutions include strategic sampling, brand circuit monitoring. Design specifically for ultra low latency environments, data volume management systems generate terabytes of telemetry data processing. This volume is in real time queries, sophisticated pipelines and storage solutions optimized for time series. Data effective implementations use steered storage approaches. The hot data kept in memory and cold data automatically archive cost effective storage while maintaining query capabilities. KDB, which I explained couple of slides earlier, was one of the key databases which we have implemented in the past that has proven to proven to mitigate such data volume management. Advanced correlation contextualization techniques combined with machine learning help identify fully actionable insights among the noise of normal market fluctuations. Addressing these challenges require a balanced approach that considers both technical and business requirements, the most successful implementations, line availability strategies with specific. So what are. And conclusions, the future of pressure trading observability lies systems that not only monitor, but impact organizations, especially the financial organizations, that capabilities will gain significants and efficiency, compliance, and ultimately profitability. So assessment. Then comes the architecture design, develop observability framework that minimizes performance impact, progressive implementation, deploy instrumentation and phase validating performances at each stage, and continuous optimization refine based on operational insights and evolving creating factors mean time to, me, quality, lower operational, past and enhanced risk as market complex increases becomes not just an operational but competitive necess and, that's pretty much it. We conclude this presentation. Thank you for taking time to listen to this presentation and if you have any questions or any suggestions or any recommendations, do reach out to me on the contact details provided in.
...

Ganesh Marimuthu

@ Anna University



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)